Is Glicko2 Rating Flawed Right Now?

#1 - Dec. 3, 2013, 10:58 p.m.

Sure Glicko2 is wonderful for accurately rating a chess player, and in other single person games.

And it can work for teams of random players if there is a sufficiently large number of people queuing for teams at all times, as in League of Legends.

But in GW2 at the moment, we have a very small pool of players, with skill ability skewed towards low level.

I just don’t see how Glicko2 can accurately judge your ability to win by assigning you to teams from a small pool of available players, many of whom are less skilled.

One very high MMR may mathematically balance 4 low MMRS…but that does not translate to an equivalent team "win probability " in SoloQ. It’s probably not a linear scale, but rather is a “discrete step function” on skill level. Meaning, sometimes a low MMR player cannot really contribute ANYTHING…but the math assumes they contribute a bit less than 1/5 of the win capability of the team.

Also, class types in a team are important for winning a match, but they are random for matches you are expected to win… to keep or raise your MMR rating.

There may be so much “noise” in the system that the true rating will not emerge for thousands of matches of the same set of people. New players introduce new “noise” in the data, skewing it, and keep the ratings inaccurate much longer still, I suspect.

Thoughts?

Justin ODell

#2 - Dec. 3, 2013, 11:07 p.m.

Once the leaderboards are replaced by the ladders, we’ll be able to experiment with ways to deal with issues like this. Until then we do not feel that it is fair to alter the MMR algorithm. This is one of the reasons we agreed to a reset after the last changes to matchmaking changed things significantly.

Feel free to offer ideas though, as we’ll definitely refer back to this thread if it leads to a good discussion.

Justin ODell

#6 - Dec. 3, 2013, 11:40 p.m.

Laserbolt.6731:
SO if people get into the top 100 and stop playing, they tend to stay there too long. They are not exposed to the same forces that randomly force some losses.

Actually, your ratings deviation increases due to inactivity, so the longer you have been inactive the wider and more random your matches are likely to become. This was masked previously by the flaw in how players were matched.

Justin ODell

#8 - Dec. 3, 2013, 11:46 p.m.

Kharr.5746:
There are a number of problems with assuming that there’s a magical solution to finding everyone’s perfect MMR in SoloQ:

1) Every player has variance in their ability depending on time of day/fatigue/health/how drunk they are.

2) Player skill and knowledge is constantly changing. The skill of inherently good, but new, players will go up each match as they learn new tricks from watching others/dying to them (e.g. terrain use/positioning on maps — you only have to see it once to know that you can provide pressure onto a point from certain areas better than others)

3) Players are not equally skilled at all professions, yet they sometimes swap what they play on-demand, or just for fun. Also, professions aren’t necessarily balanced. This adds even more noise.

Given these problems and the massive amount of day-to-day and match-to-match variance in player ability, the current system does an amazing job balancing the teams in SoloQ.

1) This is basically impossible to predict or deal with. If you’re constantly playing while in a state that your skill is significantly impaired, I’d say a lower MMR is proper.

2) The system is designed to allow MMR to raise or fall over time, just not as rapidly as when your deviation is high (i.e. new player or long periods of inactivity.)

3) I’ve pitched the idea internally of having two MMRs, one account bound and the other per profession. Does any one have an opinion on this?

Justin ODell

#12 - Dec. 4, 2013, 12:05 a.m.

Laserbolt.6731:
Though some pretty dedicated, good players who are struggling down in the “percentages” might disagree about how good the matchmaking can be right now.

In every instance I’ve investigated players are being matched very well as far as MMR spread is concerned. You don’t need to carry a team, you just need to provide enough of an advantage to win. The other team is in the same boat with the same chance at team composition.

What is especially significant, in my opinion, is the frustration of dealing with players that are less serious and less experienced, and I hope using ladders in matchmaking, among other things, will help.