[Classic] The unacceptable state of Classic servers

Forum Avatar
WoW Community Council
#1 - Sept. 1, 2022, 10:18 p.m.
Blizzard Post

Server and faction balance aside, this should never happen in 2022:

Third day of pre-patch and we are still facing a queue 2 hours long.
This kind of unplayability would have had me fired from my job as an online developer.

I cannot comment on the technology Blizzard uses, but I can tell from this situation that it is either old technology that needs replacing or lack of expertise in online development or, worse, leaving it this bad on purpose to save pocket change.
Why does WoW, one of the biggest online game in history, not have scaling servers?
This isn’t a matter of layering, this is literally just either poor management of your hardware or you are purposely not using software technology meant for scalability.

This will just make WotLK launch bad.
And I don’t want this excuse of blaming the players for stacking on one server. The solution isn’t Character Migration, it’s not up to the players to fix your problems.
Even if the entirety of all classic players were all on the same server, your technology should be able to handle this kind of scalability.

I can understand issues for 1 day, I can understand when it’s DoS attacks, I cannot understand 3 days of this from normal user activity.

Forum Avatar
WoW Classic Game Producer
#7 - Sept. 8, 2022, 12:27 a.m.
Blizzard Post

Hi All,

First off, thank you for the comments. I understand this is an incredibly frustrating time if you happen to be playing on one of these heavily impacted realms. I sincerely and personally apologize for the situation on these mega-realms. Please do know that this is the top issue on our minds right now and we are actively working on solutions for the specific realms that are full right now.

Reading forums, reddit threads, Twitter comments, etc., one thing that struck me is the sheer amount of misinformation or misunderstanding floating around, so I am going to try and explain as best I can what we are doing now, where we are going, the state and nature of the problems, and to provide a factual look at the actual state of realms that aren’t full right now. Please bear with me, as this is going to be a very long post.

We have disabled new character creation and incoming paid transfers to the US and EU mega-realms, and they will remain locked indefinitely.

This was a difficult choice that we feel is very heavy-handed. By doing this, we will cut off opportunities for new and returning players to join their friends on these large realms, possibly for many months. However, the situation on these realms is completely untenable, and even if we can eliminate queues in the short term, this is going to continue to be a problem when new content releases as long as mega-realms exist. As a result, we’ve made the choice to effectively close US and EU mega-realms to new and additional incoming players indefinitely. We will also be monitoring our other realms and will take similar steps if they begin to inch into mega-realm territory in the future, and we may do so with little to no warning. We’ve been hesitant to this because we really dislike restricting player movements and potentially breaking up social circles, but that ethos is no longer compatible with the reality we find ourselves in.

Free transfers have been very successful so far, but we need more folks to move.

I want to express a very sincere and heartfelt thanks to those that took the plunge and moved from one of these huge realms to FCM destinations such as Sulfuras-US and Mograine-EU. I’ve seen countless success stories affirming that moving to one of these realms has been a positive experience. These anecdotes were not surprising to us in the slightest, because we know that almost every realm in service right now is a perfectly vibrant and viable place to play. Both of these destination realms are now extremely healthy if not nearly full at certain peak times. If you are considering moving to one of these locations, I advise doing so soon before they too become full, as we will endeavor to end transfers before we allow them to queue, and we may do so suddenly and without warning.

For a bit more specifics around these destination realms’ health, let’s take a look at Sulfuras- US. Prior to last week, this realm had low concurrency at around ½ to 1/3 the size of a 2008 realm. In the past week, this realm has seen almost 40,000 incoming transfers, with hundreds more still occurring per hour. These incoming transfers have made this a very robust and in fact nearly full realm that is now right around 4 times the size of a full 2008 realm.

The fact is, however, this isn’t enough on its own to fix the situation on realms such as Benediction and Faerlina. Now that Sulfuras and Mograine are starting to trend upwards towards filling up themselves, we need to look to other solutions and potential destinations. As a result, we’ve implemented a new FCM solution specifically targeted at Benediction, Faerlina, and Grobbulus in the US (for now; we haven’t forgotten about EU). You can read about this new FCM destination realm, Eranikus; here. As Sulfuras has proven, Free Character Moves can and do result in a positive play experience on the destination realm. We still have far more than enough players stuck in queue on these few mega-realms that we could easily fill multiple other Sulfuras-level realms if players take the next rounds of FCMs that we make available. We again strongly encourage you to take available FCMs especially if you want to jump in and join Sulfuras and Mograine in particular, as we will be ending the transfers to Sulfuras very soon, and we may need to do something similar for Mograine as well in the coming days and weeks.

The Mega-Realms in US and EU are full (also; Layers are not the solution)

We’ll start with the easiest and most clearcut statement to make. The mega-realms in US and EU that are queuing are completely full. This statement is the absolute state of things, and there is no additional capacity we can add to these realms to allow more players on, or to reduce queues.

One common suggestion we get is to “just add more layers”, and it’s very important to understand that layers do not add in any way to capacity. Layers are a Classic-specific solution to alleviate congestion in the game world in densely populated areas. The way they work is that when a certain threshold of players congregates in a small area in the game (say, in Blackrock Mountain) the service will spin up another entire copy of the game world to try to load-balance any new players that log in. This is to prevent a situation where large numbers of players congregating, casting spells, sending server messages and updates to one another causes a severe degradation of the entire service/game world. This functions in some contrast to the “sharding” system that modern World of Warcraft uses which basically does the same thing but spins up additional shards on a per-zone or per-area basis.

Neither of these systems increase realm capacity. Realm capacity is dictated by the total number of connections that the service itself can handle. Every time a player connects to a realm, that connection interacts with numerous services, systems, and adds to the total load on the persistent database that the entire game relies upon to fetch data related to players, spells, quests, creatures, Auctions, etc. When that total number of connections to a realm’s DB and services reaches a certain number, the service will degrade or fail on multiple levels, leading to symptoms like severe Auction House lag or outages, Chat performance degradation, or lag when attempting to loot items. Never in wow’s history had the capacity of realms been as high as they are now, and even with our modern capacity we can still sometimes experience performance degradation when the realms are full and DB load is at its peak.

So put as plainly as possible, we cannot increase capacity any more without inviting additional and likely cascading failures to the service. At present, the best and only way to resolve this issue for the impacted realms, is for people to leave the realm via free transfers. There’s no technology solution to this. There is no hardware solution to this. This situation will not improve when Wrath of the Lich King Classic launches on September 26th, it will only get worse.

A glimpse at other realms, in relative terms

This post is already long, but on the topic of “viable” realm options, I also really wanted to take some time to dispel some outright falsehoods about the viability of realms that are not mega-realms. I’d like to paint a picture of how large some of these realms are relative to a 2008 realm that would have been considered completely full:

US Concurrency relative to full 2008 realm
Sulfuras 4x
Atiesh 2.5x
Windseeker 2x
Ashkandi 2x
Westfall 2x
Everlook 3x
Giantstalker 2.5x
Auberdine 2.5x
Sulfuron 2.5x
Mirage Raceway 2x
Earthshaker 2x
Razorfen 2x

I wish I could share exact numbers with you but suffice it to say: the realms above have a peak concurrent population ranging from several thousand to well over ten thousand players each day.

These realms are not queuing now (many have never queued), have healthy and robust economies, and enjoy hundreds of groups forming for dungeons and raids per day. These realms would have been full-to-bursting based on realm caps that we had in place even as recently as 2014 in modern WoW. The narrative that these huge mega-realms are the only “viable” place to play is just untrue, and we want to do everything we can to drive home the absolute fact that these are great places to play. Any “demographic” data available on third party websites should not be used as a basis for a decision around where to play.

Closing thoughts

Overall, realm health and management has been the most challenging aspect of managing WoW Classic. When these mega-realms started to emerge last year, we were concerned about their impact, but we were very hesitant to take heavy-handed action for fear of breaking up friend groups and restricting player freedom. At this point however, we believe the time has come to end the concept of a mega-realm. We hope that with your help and willingness to consider the current and future transfers we are going to offer, we can maintain a lively, vibrant community across all realms without the need for queues or increasingly heavy-handed actions.

Forum Avatar
WoW Classic Game Producer
#10 - Sept. 8, 2022, 2:31 a.m.
Blizzard Post

I totally understand that concern. I’d posit that these moves and consolidations are less aimed at you, and your very well established social circle, at least not at first. This is aimed more at those who are maybe less attached and are merely following the crowd when deciding where to play, or are returning after a long absence and have found their previous friend groups have moved on.

How many threads have we seen where someone says “Just returning to the game. what server is good for Alliance in NA?” and are told “Benediction”. That’s the group we’d like to take the moves first to help with the situation right now. Long term however, we do want to take much more deliberate steps to reduce the chances of this happening again, and we unfortunately need to be a bit heavy handed going forward to prevent any tailspins that inadvertently create other bigger problems down the road. It’s going to take us some time to realize the full scope of what we need to do to correct this, but for now, our best option is to continue with very tightly monitored and targeted FCMs (including potentially faction-locked FCMs, as Kaivax alluded to in another post here) to alleviate the immediate pressure.

I also just want to reiterate this point though; we’ve raised population caps to accommodate mega-realms to the absolute limit of current technology. I’ve seen a lot of armchair server engineers say things like “just add more hardware”, and I can’t stress it or put it any more plainly, the technology to allow more than we have on realms now does not exist. We’ve added the hardware, we’ve optimized as much as we can optimize, and the current demand on these few realms is just too much. Continuing to push our luck and find ways to reach ever higher realm caps has hit a point where we can’t go any further, and eventually something had to give. That something has given and we are going to be taking different approaches to this problem going forward.

The scenario that led us to the situation you described above was a by-product of past us spinning up tons of new realms rapidly to deal with a level of demand that was orders of magnitude beyond anyone’s wildest expectations when we first launched Classic in 2019. Now that we have consolidated down to a nominal amount of realms, we believe we have the right number of realms to support the number of players we expect and then some, without any realms feeling empty.

But again, I understand completely if you feel a pull towards your realm. The scenarios you describe with multiple raid groups full of folks who may raid in multiple teams or with multiple guilds is a very real situation and that’s hard to visualize breaking away from. If that isn’t you though, and if you aren’t completely tied down to these realms, take the FCMs. Please, take the FCMs. Sulfuras is a success story and we believe Eranikus will be as well.

We’ll be talking more about populations on Eranikus and how things are going as time goes by, so please stay tuned as that situation develops in the coming days.

Forum Avatar
WoW Classic Game Producer
#14 - Sept. 8, 2022, 4:01 a.m.
Blizzard Post

Well I don’t know if I’d say our that’s our “current stance”, holistically speaking. We have seen established guilds transfer, and hope to see more. I was merely saying, I can understand why you’d be hesitant to move, its a big decision with a lot of ripple effects.

If anything we are hoping that some of the less attached players are willing to move in some larger numbers to help alleviate the current and immediate concerns. It’s not a long term fix or strategy on which we can rely on its own.

We have considered it. Nothing to share just now though.

Trying to chase faction balance with transfers is even more difficult than trying to reduce server population, historically. If there’s any perception that someone will be an underdog, they generally won’t make the move. But, there’s definitely a possibility we will try it. I think if we really leaned into that, we’d want to try and pair it with some sort of incentive to move, which is something we are working on designs for, and talked with a few members of the community council about a few weeks ago in a zoom call. Unfortunately it’ll take a bit of time for that to get fully implemented though, should we decide to go that route.

So the fresh realms are a very unique situation and we want to be extremely careful about adding more realms. Adding a 2nd fresh PvP realm is very very likely to do one of two things:

  1. Lead to both realms being unhealthy in the long term.
  2. Essentially guarantee that one of the two realms becomes the “horde” server, and the other becomes the “alliance” server.

Additionally, Fresh realms are also much more likely to experience a bit of a decline once the game actually launches. Many people are leveling in fresh now during pre-patch but we expect some number of those players to return to their normal established servers and guilds once the expansion launches. This might be a situation where its better for us to weather the storm, as it were.

Lastly, the fresh realms’ total “footprint” of players isn’t nearly as large as a server like Faerlina or Benediction. These established older large realms will continue to swell as folks return who may be inactive unless moves occur, which is a different situation than Skyfury is in. While the queues on Skyfury are long, they aren’t as long as the other queuing mega-realms. If everyone in the queue for Benediction went to another realm, that realm would be queuing as well. That queue is more than double the capacity of the realm. If everyone in Skyfury’s queue went to another realm, it would be a fairly busy realm, but not nearly full. Skyfury and the 2nd realm may not survive that division long term.

Fast reactions to immediate problems are part of what got us here, and while our goal is to get everyone into the game as soon as we can, we also want to make sure Fresh realms are a long term success as well. We want to be extra sure that the only option is to open a new fresh realm before we do so.

Forum Avatar
WoW Classic Game Producer
#16 - Sept. 8, 2022, 4:59 a.m.
Blizzard Post

Server merges is definitely a rubicon we’d rather not cross, partially because its a fairly risky process, and partially because once we cross that line we can’t really un-cross it and modern-esque connected realms becomes the expectation. While we are choosing to try and chip away at mega realms and push towards dispersing the population, I’d say that there’s nothing really off the table as potential solutions. In addition to what we’ve already discussed, we are pursuing a lot of options and fallbacks for the future behind the scenes.

This is all sort of academic however. Where we are standing right this moment, there are no realms in service that need merges. Aside from a very very small handful of specific realms of the dozens and dozens we have worldwide, every realm has a very viable population.

This is not the case. The issue we have here is also an issue on certain modern realms such as Illidan and Tichondrius when new expansions are released. They have the same realm caps we have. This is purely a perception thing, we are using the same hardware, and backend infrastructure for both games.

Same! It’s part of the reason I’ve been here for 13 years. :slight_smile: