Official state of skill lag and server optimizations

#1 - Oct. 5, 2013, 1:26 a.m.

I’d like to create a central place for you all to get the current state of where we stand on optimizing the servers and relieving what most of you have come to know as “skill lag”.

Over the past several months we’ve had a small team working on ways to relieve some of the load from the game server while there are large concentrations of players. Most of the work thus far has gone pretty much unnoticed from the player’s perspective, but there have been pretty large steps made towards the issue. This continues to be a large and high priority issue for us so we can continue to make improvements to make this a thing of the past.

The various teams, or as we call internally “Feature Teams”, are now subject to performance reviews both client and server before they can ship. This is one of the many things we’ve put in place to stop the bleeding. Preventing further issues has been the main focus up to this point to ensure we aren’t introducing new issues and constantly cleaning up behind them. Now that we’re to a point where we’re confident that we’ve plugged the holes, larger strides to actually make noticeable optimizations are in the works.

I mentioned sometime last month that I’d like to be more transparent about what steps we’re taking to address the issue. To give you more of an insight of where we stand, we’re starting with taking the bull by the horns. We’ll be going through almost every skill in the game over the coming months and optimizing them one-by-one. This in itself will make an extremely large difference, that when that patch note comes through, expect a very different WvW experience.

I’m going to leave this thread open for discussion, but I urge you to be respectful of one another and keep things civil, otherwise I’ll be forced to lock it.

-Bill

#2 - Oct. 5, 2013, 1:27 a.m.

(reserved)

#24 - Oct. 12, 2013, 12:10 a.m.

Part 1:

To clarify some common questions:

• What does optimization mean in terms of the original post?
  • I’m working primarily on server-side (as mentioned in the title) optimizations. Not to say that this doesn’t have some smaller impact on the client, which I mentioned in posts elsewhere that there will some client-side gains as the changes rollout. Client-side changes are being worked on completely seperate and will release seperately and, sadly, slower. I’m not going to get into the client-side portion of our optimization initiative, because I’m not directly involved with what we have going on there.
• What exactly do these server-side optimizations do?
  • As pointed out above, they will mostly be noticeable with latency in skill execution in heavy battles. I mentioned WvW being a benefactor because that has one of the most consistently dense battles going on. That’s not to say that this won’t be noticeable in other parts of the game; it most definitely will.
  • Though server CPU usage is the biggest factor here, there are other optimizations as part of the changes going on. Bandwidth is another thing that plagues players with slower internet connections. That is something that we’re actively rolling out changes for, with the next set of changes coming at the end of October.
  • Removes the bottleneck in almost all of the game systems. Most of what goes on in the game is processed on the server, including combat, skills and movement. When this bottleneck is relieved, the changes will pretty much speak for themselves. A common example goes back to skill usage where in heavy battles, you press and skill key, but that skill never fires or your auto-attack skill fires instead.

#25 - Oct. 12, 2013, 12:10 a.m.

Part 2:

Moving onto a few specific questions:
Q: My 1st question is : Can you be more specific of what “optimization” means and will it affect the skills as we know it by any negative way ?
A: These changes will go unnoticed functionality-wise. The only changes will be positive in that skill usage will be constantly more responsive.

Q: My Second question is : Wasn’t all these problems caused because you have to broadcast every single condition/buff to everyone which is currently pushing your systems and servers/internet to its limits ?
A: This is a good question. Buffs are a large part of Guild Wars 2 and at first glance you’d definitely think that telling everyone about them would be really expensive. In reality, this is only a hit with server->client traffic, which has been significantly improved with the handful of bandwidth optimizations that have already gone out. As far as server processing, its pretty minuscule (thankfully).

Q: My 3rd question is :If you ever have a solution to this are there any plans to lift the AoE Limit for both WvW/PvE/PvP because thats why this AoE limit exist in 1st place Isn’t skill delay in big battles (WvW or PvE like Taco) a direct result of those?
A: <<< The AoE hit limits exist for more than just performance reasons. We already have a lot of places in the game where AoE limits are extremely high (WvW siege) or uncapped (monster skills). The last thing we want to do is relieve one problem only to introduce it somewhere else.

Q: When you say “optimization”, do you mean making your engine more compatible for multi-core systems? Right now your game’s engine runs primarily on single-thread, which Intel CPU’s seem to thrive on; while at the same time your game’s engine is poorly designed for multi-thread CPU’s, which AMD is best at.
A: As I outlined above, this is mostly server changes that I’m involved with, but there are client optimizations in the works, but we don’t know when those will go out. Optimizing is extremely dangerous and requires a lot of testing. The server side changes are more controlled in the sense that we don’t have to worry about testing on different hardware configurations like with the client. Its unfortunate, but necessary to ensure that the changes don’t end up doing more harm than good.

The engine isn’t as single-threaded as you might think. It will only use as many threads as your machine can support. if you have a 4 core processor, you may only get 2 threads plus an additional 2 worker threads for processing things like animation. Though I we are definitely looking at other ways to improve other game systems to take better advantage of using multiple threads while they’re not in use.

Q: You said that when that patch note comes through, expect a very different WvW experience. should we expect to see these optimization on 15th October? or was that just writer’s freedom of speech?
A: I Apologize for any misunderstanding here. But the answer to this is no, there isn’t really anything of note coming next week. Though I did I have large hand in the skill and combat related changes coming in that release . We definitely do have a few things that’ll help, but most likely won’t be noticeable.

Q: Thank you but what about client side optimizations? We were promised some major engine/client side optimizations to improve our fps in WvW 6~ months ago, has this project been dropped because I haven’t seen anything regarding client performance since then
A: Again, like I mentioned above we are working on client changes, but they are on a much slower release cadence and have to go through a rigorous test cycle before ever being on deck for release. Patrick, the slayer of culling, is a large part of these changes that’re being cooked up.

To respond to the large response by Demosthene.2195:
The idea for having some sort of central location or “issue tracker” as you mentioned, I think there’s definitely something there. I think that central location would make sense to be on the Guild Wars 2 launcher itself, which from what I understand is going through a makeover. I think that’s definitely something for the Community team to investigate.

The rest of your post starting going off-topic. I will say that I want to make sure that people realize that we don’t always have the time to respond to every question, concern and issue that arises, even though we would love to. I constantly say how I try to be more involved and transparent with you guys, but obviously I don’t have the ability to spend time on the forums every day. We love our player-base and appreciate how passionate they are. Your feedback doesn’t go unnoticed, even if it seems like it does.

Also, thanks for the links, that’s really cool to see other developers be so open with their player-base about issues.

#26 - Oct. 12, 2013, 12:11 a.m.

Part 3:

So what am I working on besides ensuring player skills are efficient? Well with our two week release cadence, I work along-side our shiny new Technical Design team and Living World teams to ensure their content is efficient. I’m also constantly exploring other ways to improve the game, which led me to something that I literally just finished wrapping up today. This change will go directly into testing early next week, which will then help us determine how significant it is and when we are comfortable enough to push it to the Live game. I’m really optimistic about this change in particular; seeing massive drops in server load .

In closing, I want to let you guys know what the release plans are for optimizing the game server. To put it simply, the plan is changing almost every day. We have the ability to ship changes whenever they’re ready. And because of our current release cadence, that means that we can literally get these changes out to you guys in a matter of 2 weeks. That being said, optimization is dangerous, especially to a live game. Ideally I will continue to sneak in changes every 6 weeks or so, and when there are changes that players will notice, by all means there will be a release note to call them out.

-Bill

#68 - Nov. 3, 2013, 1:40 a.m.

Hello again!

I disappeared into my Bat-Cave again and finally emerged, and with some good news to share. I’ll start off with what exactly that is.

Since my last updates, we’ve been hard at work prepping some serious server-side optimizations to relieve the bottle-necks during heavy combat. We’ve made some large steps and are almost ready to start rolling out these changes. The first batch of changes have been in testing and we hope to have them start trickling in as soon as the release on Nov. 12th. We’ll know for sure once we’re closer to that date, and so will you.

I can’t stress enough how dangerous it is to optimize a live game. I know how upsetting it can be to be in the thick of a choppy battle, but things are going to be getting better, and soon. We’ve done some temporary things on the back-end to try to ease the influx of players in Wvw, but those changes only go so far.

On to answering some of the common questions.

A lot of you have noticed what seems like an increase in skill-lag in Wvw since the beginning of Season 1. Really, this is just a large influx of players playing Wvw, mostly in servers that usually don’t have queues to get into the map. A lot of the higher tiered servers are pretty used to large battles running into this issue, but obviously that isn’t a valid excuse for it happening. We’ve only increased the focus on relieving this since the season start, and rest-assured, its a top priority.

Some other common suggestions/questions is about a method other games used, which is known as “time-dilation” or “time-scaling”. Well, for starters, this method is extremely risky. We’ve discussed this, among quite a few other alternatives, and it has boiled down to causing more problems than it’ll solve. I know you might ask, “but the current experience is bad enough, how could it be worse?”. Well, to be completely honest, it would just open another can of worms that would end up breaking the game and causing things much worse than a couple seconds of input delay. We opted instead to focus on fixing the issue by simply making the game run better, instead of sweeping it under the rug by watering down the experience. Internally we have the ability to slow down time-scale of the game, and it just feels terrible. Not to mention breaks key mechanics of the game, such as the physics simulation.

Some other assumptions I’m seeing is about our server hardware and inefficient use of communication between them and the game’s database. I’ll go ahead and put these assumptions to rest. Our server hardware was actually purchased new right around the launch of the game, rest assured its not out-dated. And you can sleep at night knowing that our combat doesn’t connect out to the database for information. All of the information it needs is already loaded into memory. The game database is simply for storing persistent character and account information, which is for the most part only accessed when loading in/out of a map or periodic saves which are handled asynchronously. I suggest checking out the links below for more information on our servers.

As far as skills just not executing (noticed some people claiming utilities are more susceptible to this), its mostly just a race-condition as far as processing on the server. You’ll notice that your auto-attack skill seems to process more reliably than other skills. This is mostly due to the fact that we process things like auto-attack timers before player input. Obviously that sounds like a bug (and honestly now that I think about it, I want to look into doing something about it), but the reality is that under normal circumstances, the player input would process before the auto-attack timer triggers. Something you can try to verify this is disabling your auto-attack and see if your other skills become more responsive.

For you tech savvy folks, here’s some good links to help understand our server infrastruture and some other pretty neat things: Link – Another Link!

Well, that’s it for today. I’ll be sure to update you all on our progress and when you can start looking for major improvements. Like I mentioned above, we’re pretty optimistic that this could be as early as our next release. Have a good weekend!

-Bill

#83 - Nov. 16, 2013, 5:23 p.m.

Carzor Stelatis.9435:

BillFreist.1527:

Some other assumptions I’m seeing is about our server hardware and inefficient use of communication between them and the game’s database. I’ll go ahead and put these assumptions to rest. Our server hardware was actually purchased new right around the launch of the game, rest assured its not out-dated.

The hardware can be as new and as powerful as you like. The problem for EU players is that you chose to locate your ‘EU’ servers in Texas, and no matter how powerful the server hardware you’re going to get more lag if the data has to go from (say) France to Texas and back than if it had to go from France to the Netherlands and back.

False. They are located in Germany.

How did you come to this conclusion?

#84 - Nov. 16, 2013, 5:30 p.m.

So during the November 12th release, we rolled out a couple of changes that significantly reduced skill lag. One mostly targetted PvP and WvW, whereas the other was more high creature concentrated PvE areas.

Curious to see if you, as players, have noticed the impact (or haven’t noticed it because the game feels as it should). Of course WvW will still see occasional spikes during large 3-way encounters, but those a significantly shorter and the server can now recover from them almost immediately.

Needless to say, more improvements are well underway.

-Bill

#136 - Jan. 18, 2014, 9:27 p.m.

Quick update:

Happy new year everyone! Now that we’re back from holiday vacations, we’ve got more improvements that’ll be rolling out soon. A handful of client improvements will also be making their way (soon hopefully), that should see a drastic memory usage drop and client frame-rate improvements, especially for lower-end hardware.

I’ll make sure to update once I have a better idea of what release you can start seeing these changes trickle in.

-Bill

#171 - Feb. 4, 2014, 4:59 p.m.

To follow up, the client optimizations that I mentioned before went out with the build today. Let me know if you saw improvements. Feel free to list your specs too!

-Bill

#174 - Feb. 4, 2014, 5:24 p.m.

Thedenofsin.7340:

Were there any optimizations for AMD CPUs?

Theoretically everyone should see an improvement.

#223 - Feb. 7, 2014, 2:15 p.m.

Someone asked for acknowledgment. You got it! Bill is active in this thread as other devs may be as well, but here’s a little added note:

Some of you have noticed improvements, and that’s good. We’re aware that others still are having latency issues, particularly our Mac users. The team is working on the issue/issues.

While there’s no precise timeframe for a roll-out, the Production Director (that’s one of our top guys) stopped by last night to specifically state that this was actively being worked on. See this post. I think that’s relevant and should reassure you these issues are on the to-do list.

#239 - Feb. 28, 2014, 9:39 a.m.

Keep in mind that there are a few very different issues being mixed up here. “Skill Lag” as discussed in the original post here is referring directly to Game Server CPU usage. I want to ensure we are all on the same page, so here’s a little refresher:

• Game Server CPU usage (which is the #1 issue discussed here)
  • Let’s be honest, Guild Wars 2 has a lot of moving parts, and the Game Server is pretty much always in the thick of it.
• Player/Client → GameServer bandwidth
  • Most cases usually come from players having other heavy traffic going on while playing the game, weak WiFi signals, or their ISP is having difficulties.
• GameServer → Player/Client bandwidth
  • This can happen sometimes in our datacenters during peak hours or when hardware is having difficulties, but normally resolves very quickly.
• Client frame-rate
  • CPU and/or GPU bound hardware
  • Extremely heavy combat, dense populated areas, and/or effect spam (which causes a lot of overdraw, which can cripple even the beefiest hardware )

So how about what you actually want to hear… When are more improvements coming?

Like always, I don’t have a date to toss around (yet), but I do have a couple things that are well on their way, and hopefully will make it in the next few releases (fingers crossed). So, what can you expect?

Another round of Game Server optimizations

With most of the previous improvements to the Game Server, I’ve tried to focus on improving combat responsiveness. Its just not a fun experience when you try to execute a skill and either: A: Nothing happens, or B: It just doesn’t execute in a timely manor (which is arguably worse). The changes coming should make a nice impact on content that has large densities of creatures and/or zergs of players.

Though, not all “Skill Lag” scenarios are a product of the bullet points stated above. There are other causes of input delay, and a good example of this would be Ranger Pets. I’m sure plenty of you agree that their responsiveness has been arguably, well, next to none. I recently spent time working on improving the interaction Rangers have with their Pet using the Command Bar. I’d love to hear your feedback on this once it ships. Issues like this are something I definitely plan to continue improving upon (Necromancer Minions also come to mind).

Another similar issue that has been extremely frustrating in combat is trying to hit objects like Engineer Turrets (I’m still having nightmares of Graveling Burrows). Or, rather, turrets getting hit by attacks that seemingly shouldn’t have hit them, resulting in what seems to be “lag”. This is something that myself and a designer spent some time tuning, and the results are pretty self-explanatory. I believe some of these changes are actually coming as soon as the next release, though this will only affect attackable objects that don’t have navigation-blocking collision (turrets, some siege equipment, etc).

And, of course, there’s more non-performance/optimization related improvements well on the way that’ll increase the general experience and responsiveness of combat. An example would be Mesmer Illusions/Clones/Phantasms that unreliably attack their target – Warden, I’ve got my eye on you!

Lastly, client frame-rate and memory usage is still being actively worked on. I’ll have to follow up on when more improvements will ship, but I know there are some exciting things in the works.

-Bill