Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you: No adverts like this in the forums anymore. Times and dates in your local timezone. Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance. Email notifications for threads you want to watch closely. Being a part of the oldest wargaming community on the net. If you are already a member then feel free to login now.

Simple question. which is a better indicator of a factions power level in the tournament scene. W/L which can fluctuate based on mirror matches, skill of players, newbies, non-serious armies etc or Top finishes. So if a faction takes lets say...the top 3 places in tournaments is that a better indicator than the fact that the faction as a whole barely broke even with wins/losses overall?

I would have said wins to losses, especially amongst normal average Joe gamers.

Tournament armies aren't a good representation imo because the "top player" crowd tends to be full of people who can win with any army regardless.

Wins compared to losses with both weighted by the faction being played, the round in which the game occurred, and the record of the player piloting the army will always give better results than simply look at the top x places of an event. Something like Goonhammer's Glicko system will also give better results as going 4-1 and making top 8 and going 3-2 and missing out can come down to luck more than skill and a rating system will be able to better account for this than the raw data will.

EDIT: Does anybody want to bet against the idea that a year of Glicko data will be a better predictor of which armies are strong than just looking at GT top-8s over that same span?

W/L should be considered a secondary factor unless it is too low or too high. Once we reach 40% or get past 57-58% , thats an obvious problem (Unless we have a data of 5-6 lists played)


I would not say top 4-8 is very good.

For example, is the top 10 of a 60 man tournament as relevant as the top 50 of a 700 person tournament like adepticon?

In general, the first third of any tournament is were the "competitive" people will end up. Thats the relevant data we should measure.

W/L ratio, simply because you should be ignoring the random flukes that happen. Remember when 6th Edition version of Thousand Sons topped at like number 6 or 8? Or course not, because that was a fluke that should be ignored. In the same way, this accounts for newer factions that less people have built up that could potentially be problematic, like Genestealer Cults were starting to be at the end of 7th.

The poll is currently 9-to-9 but not a single person who's voted for top 4/8 results being a better metric has left a comment yet. Do they have no arguments for their system being better?

For "at-it's-best", use metrics related to "at-it's-best".

Top placements in top tournaments removes, or at least significantly mitigates, a number of biases.

One bias the W/L ratio has is the presence of non-optimal lists. People who show up at tournies to show off their decade-old army they've only changed to fit newer points/rules still count for W/L ratios. But they don't make the top 8 in large fields, so don't bias those numbers.

Another bias is player skill. While some players can win with anything, so can other players. And so those "some players" will spike the W/L of their chosen faction, biasing it away from what we're looking for. But, in the top 8, they'll be facing other high-skill player, driving down the bias.

Discoverability of the codex's strength is another bias. If the codex can be stupidly OP, but most players don't/haven't realized it, the W/L record will reflect something closer to what the average player has realized. But if you're looking for OP/UP codexes, you care more about the strongest ways to play the codex. If only 5 of 50 players at a tourny know how to build a good list with $faction, the top 8 metric will capture that strength much more clearly than W/L.

W/L and Top 8 are better measures for different things, really. How powerful a book can be tacks a lot closer to Top 8 representation than W/L.

Making a claim based on fraction of a percentage would fall under false equivalence.

The whole dataset should be considered.

 skchsan wrote:
Making a claim based on fraction of a percentage would fall under false equivalence.

The whole dataset should be considered.

Most claims are made based on a fraction of a percentage of the population. Most of the work goes into finding the right fraction to use. It's a settled question that a larger fraction is not necessarily more accurate (read up on Gallup's history for detail).

The W/L rate captured by whatever tool isn't a census (complete set) either. It's another subset with a different bias.

Further, what are we measuring? If it's book power, W/L rate, even if you could collect every game ever played, is not an actual answer (but is likely a useful proxy).

I've posted above why I think the Top 8 is the better (but still wildly inaccurate) / more accurate measure.

Bharring wrote:
For "at-it's-best", use metrics related to "at-it's-best".

Top placements in top tournaments removes, or at least significantly mitigates, a number of biases.

One bias the W/L ratio has is the presence of non-optimal lists. People who show up at tournies to show off their decade-old army they've only changed to fit newer points/rules still count for W/L ratios. But they don't make the top 8 in large fields, so don't bias those numbers.

Another bias is player skill. While some players can win with anything, so can other players. And so those "some players" will spike the W/L of their chosen faction, biasing it away from what we're looking for. But, in the top 8, they'll be facing other high-skill player, driving down the bias.

Discoverability of the codex's strength is another bias. If the codex can be stupidly OP, but most players don't/haven't realized it, the W/L record will reflect something closer to what the average player has realized. But if you're looking for OP/UP codexes, you care more about the strongest ways to play the codex. If only 5 of 50 players at a tourny know how to build a good list with $faction, the top 8 metric will capture that strength much more clearly than W/L.

W/L and Top 8 are better measures for different things, really. How powerful a book can be tacks a lot closer to Top 8 representation than W/L.

Top 8s can show spikes but can also fail to show which armies are consistent performers and which just got lucky on a given weekend. Is a faction that took 3 of the top 4 positions at a single tournament and then failed to make top 16 the next four better than the faction that finished top 5 at all 5 events and has yet to claim a top 4 spot?

Bharring wrote:

 skchsan wrote:
Making a claim based on fraction of a percentage would fall under false equivalence.

The whole dataset should be considered.

Most claims are made based on a fraction of a percentage of the population. Most of the work goes into finding the right fraction to use. It's a settled question that a larger fraction is not necessarily more accurate (read up on Gallup's history for detail).

The W/L rate captured by whatever tool isn't a census (complete set) either. It's another subset with a different bias.

Further, what are we measuring? If it's book power, W/L rate, even if you could collect every game ever played, is not an actual answer (but is likely a useful proxy).

I've posted above why I think the Top 8 is the better (but still wildly inaccurate) / more accurate measure.

In my opinion, the faction's "power level" should be determined as a weighted average of a function of win rate and the difference in victory points accrued per attendee.

The former data provide the overview of how much in favor certain factions are, and the latter informs us of how badly the faction beat it's opponents. This helps ensure that the metrics involved are fairly contributing to the overall picture since a win by a hair's width isn't as 'valuable' of a data compared to overwhelming victory.

Both should be taken into consideration but neither, especially W/L, should be relied on solely as an authoritative source. Looking at absolute winrates ignores the varying skill levels and intent behind the army list being run, it also often completely ignores mirror match-ups and how they affect that W/L ratio, especially in a faction that is extremely prevalent and has more instances of mirror match-ups happening. That same prevalence can also be an indicator of power, perceived or real, which of course has an impact on the meta of the game and the tournament as a whole. Harlequins might have an absolute 60% winrate, but if I'm constructing a relatively competitive Necron list all I'm really looking at is how to deal with Space Marines, because outside of a truly gargantuan GT I'm not very likely to come across clowns at all. If the majority of competitive contenders at a tournament are having this same thought and building to counter this army, the fact it is still capable of getting top 3 finishes AND having a positive winrate shows that it's still easily one of the top factions in the game.

Of course on the flipside there also needs to be consistent high placings. A GT win on a faction that doesn't translate to the faction as a whole seeing a resurgence isn't really indicative of anything.

Which leads to...

So if a faction takes lets say...the top 3 places in tournaments is that a better indicator than the fact that the faction as a whole barely broke even with wins/losses overall?

So, Siegler's LVO run with Tau was this and this is more to do with the idea that people didn't think Tau were strong until proven otherwise. It's not like it was a weird outlier; suddenly Tau armies started winning a whole lot more.

We've had instances of GSC or even Tau winning events in 9th, but this has not translated to mass success across the edition so far. Since this thread is about Space Marines we can also easily see that they aren't getting the odd top 3 finish while every other list in the tournament languishes in the bottom half. They are consistently winning tournaments, getting top 4's and getting the strongest representations in top 10's, while also seeing loads of representation in the bottom half too because hey largest faction with the largest spread of player skill and player types using it.

There are so many confounding factors in an analysis like this that any metric would have to be taken with a fairly hefty pinch of salt and an acknowledgement of their potential weaknesses and structural biases. These numbers only really provide a jumping off point for discussion rather than an objective measurement of any truth.

A thorough analysis of something like tournament performance of different factions would require a lot of ancillary and historical data to be collected to try and control for some of these factors, and even then it would be pretty weak data overall by the standards of proper studies.

So if you're going to talk about it, you should talk about both metrics while keeping in mind that both are flawed and need to be supplemented with discussion and argument.

Lists that have gone undefeated in 7+rounds in 128+ player swiss events have been significant data points that I've used to determine what to meta for in the competitive games I've played, namely Magic and Warmachine. There's probably other relevant data that can be mined, but undefeated data points do a lot of work. Winning players typically pick the factions that work against the meta that existed before the tournament, so the actual results of the tournament may shake up that decision for future tournaments. Luckily, 40k metas move rather slowly due to the hobbying aspect of the game, so that effect is subdued.

I haven't gotten to that point in 40k, but I've got a close friend I collaborate with that had a top 100 8th edition ITC rank that discusses meta in much the same terms.

I think they tell us different things, and top table placement can depend a lot on event & standings criteria. W/L loss ratio will tell us how well a faction does overall, but may be gatekept from top tables. We've had armies with very strong W/L ratios that dominate most opponents, but almost never dominate placings because there's something that hardcounters them that they inevitably run into, or where they may win abnormally often but not with significant margins. For example, a list that manages a 70% win rate, but that barely ekes out each win, may not appear in top standings as often as a list that wins 60% of the time but where each win is a total blowout. Different events can impact this in different ways.

If half the lists in the meta use only 1000 pts then those lists will never see a top 4 at a GT, but due to the way most tournaments match players (by their win-ratio rather than completely randomly) all the lists using 1000 pts will still trend closer to 50% than 0% due to them facing each-other. That's why it doesn't really say a lot when a faction has a 45% or 55% win-rate.

Consistently getting a high ratio of top 4s compared to participants using the faction shows that the faction isn't just being scrappy at the bottom tables but is dominating the big dogs at the top tables.

I'm not married to judging factions purely by this measure and this is really just post hoc justification for why I voted for one and not the other. Harlequins or some other faction having a 90% top 4 rate isn't that problematic if their win-rate is 50% assuming it's just because they are skill intensive. Balancing for high vs low level play is hard.

Consistent cross continent high placing of a specific faction show, that the army is not just good, or good because it is piloted by a good player. But that the faction is above other factions. It is like 40+years of domination of ethiopian and kenian runners in long run competitions.

Win rates are imporant too though, they show weakness. If I place high in all events where there is no left handed opponents and have a big drop rate in events, when I run in to them at all levels of a tournament, then this shows my bad sides. Which is also very important.


In general multiple methods should be used at the same time to check how good something is or isn't.

w40k isn't different. If a faction has regular top 1-4 in multiple events all across the globe and has a high win rating then it is good. Same is true for the reverse, if the army sits on 30% win rates, and have not managed to place in any larger event in top 16, and in all bigger events it places in bottom 50%, then it is clearly bad. Only worse thing is when a specific army does not show up in the data at all. If no one plays a faction for a few months, then it clearly in a dire state or even left unfinished by GW.

You can't use just one metric. And even then it's directional.

LLWWW presents a different perspective than WWWLL

Additionally, the quality of wins and losses can vary quite widely. Deathwatch has a 54% WR, but averages 63.34 VP to 63.17 (for January) - so they don't win big, but they win consistently.

Then you have non-skilled players dragging down the faction. What good does WWL do if another player goes LLL and makes the faction 33% WR? Did the first player have better matchups? Better skill? More first turn advantage?

To get a true picture of a faction's power I think you need to consider a lot of different factors in combination with each other. No single metric will tell you the whole story. We saw that with Sm after their 8.5 Codex where the raw win rate was good but nor brilliant until you removed the mirror matches and a truer picture emerged. For me, consistent placing at the top of multiple tournaments is the best single identifier of what's powerful but does not in itself show enough. I'd also say you'd want to look further than the top 4-8. Armies in the 10-16 range would probably be what I'd look at for any large tournament.

LiMunPai wrote:
Lists that have gone undefeated in 7+rounds in 128+ player swiss events have been significant data points that I've used to determine what to meta for in the competitive games I've played, namely Magic and Warmachine. There's probably other relevant data that can be mined, but undefeated data points do a lot of work. Winning players typically pick the factions that work against the meta that existed before the tournament, so the actual results of the tournament may shake up that decision for future tournaments.

But that doesn't tell you anything about a specific faction's power level. It just tells you about a specific player's list building skills to exploit the weak points of an existing meta.

You need a long term and consistent dataset (preferably in multiple venues and circumstances) to even vaguely map 'power level.' It doesn't tell you anything if only Kevin Warboy can sweep a specific yearly event if and only if its marine heavy.

It does tell you a lot about the factions regulary missing from the top 4-8. If they never or sporadically pop up there, and never take spot one, you can be sure that the army lacks the tools to be picked, by the best players to be piloted to number 1 spot.

Slipspace wrote:
removed the mirror matches

I'm going to challenge this concept, because it's pretty bogus.

Why is an Ultramarines Dreadnought list a mirror to a White Scars VanVet list? When CSM get W2 are they going to be considered marine mirror matches, too?

Top placements are usually just strong lists, not strong armies.
There is one build that is strong, while most of the army is trash -> finishes top. Do you consider this to be a strong army?

Personally I don't, so I vote for W/L. This probably covers more different builds of an army and therefore says more about the army.

When I brought top finishes or the scores of notoriously good players to a balance discussion, I was told that those didn't matter, because even if a faction works in the hands of an highly experienced player, the faction can still be bad.

Actually, that was a spot on critic.

Relatively speaking, no one cares about "competitive" play. 99,9% of the players play at casual level. That's where the game should work.

If the game broke at the highest level of play, I wouldn't care. The Adepticon can have the same faction and list in all the spots of the top 10, but if the casual level of gaming is balanced between factions, then the game is fine.

TLDR: Win/losses over top spots.

Spoletta wrote:


If the game broke at the highest level of play, I wouldn't care. The Adepticon can have the same faction and list in all the spots of the top 10, but if the casual level of gaming is balanced between factions, then the game is fine.

TLDR: Win/losses over top spots.

Only if that happens you automaticaly see people start running the same lists. Tournaments started replacing slower MM units with attack bikes, and now recasters are running out of resin to print them for people.

Some stuff is obvious like running max tyrants pre rule of 3, or when chaplain dreads suddenly become gods alongside leviathans. And while it may not take alot to build a working harlequin list, if suddenly someone made a GT winning IG list, and it would win multiple events, people would be copying it very fast.

Reading this thread it seems like most posters should agree with this system of measuring power: https://www.goonhammer.com/the-february-2021-40k-meta-review/

Highlights:

Data Collection:

"Thanks to the wonderful efforts of tournament organizers and app developers around the world, we have access to what is essentially every meaningful piece of data around competitive games of 40k. The data in this month’s study comes primarily from The ITC Battles App, a brilliant app for tracking games both in and out of tournaments. With tournament activity currently slowed in most areas thanks to the pandemic, we were still able to analyze data on nearly 4,000 games of Warhammer 40k played in January."

Correctly Rating Factions That Face Hard Counters:

"One example we’ve seen recently is Chaos Knights, a faction that posts high win rates but does not win tournaments – they score low with regard to 40kstats’ Tournaments in Winning Position (TiWP) metric. This indicates that Chaos Knights will frequently lose early on to better factions but then clean up in the losers bracket."

Factors in What You Win and Lose To:

"Instead another way to consider relative faction strength is to calculate ratings for our factions. Glicko is a rating system similar to ELO (it was designed as an improvement on ELO, in fact), that can be used to gauge performance. As a 40k faction in our sample wins or loses games, they gain or lose points from their score, gaining more points from beating a higher-ranked faction and fewer from winning against a lower-ranked faction (likewise losses to lower-ranked factions have harsher penalties). There are some additional factors to the model, such as how much you want it to react to each new game result, but for the most part, those are the basics. In our model, a score of 1500 is about average."

Top Finishes is a better metric than Win-Loss, but neither are prefect in isolation. As others have said, you need a more nuanced analysis with more data to answer the question as to which factions are good tournament factions.

I don't like Win-Loss as a number because it has space for lots of noise and variance. At the extreme, you can have two factions with a 50% W/L record. One can consistently show in the Top slots because when a well-crafted list is run by a good player it does very well. Otherwise it hangs out at the bottom tables getting sand kicked in its face. The other faction can consistently win around half its games regardless of the quality of the build or payer, but never see the top tables.

In this case, one is a better tournament faction while the other is a well balanced faction.

 Bosskelot wrote:
So, Siegler's LVO run with Tau was this and this is more to do with the idea that people didn't think Tau were strong until proven otherwise. It's not like it was a weird outlier; suddenly Tau armies started winning a whole lot more.

We've had instances of GSC or even Tau winning events in 9th, but this has not translated to mass success across the edition so far. Since this thread is about Space Marines we can also easily see that they aren't getting the odd top 3 finish while every other list in the tournament languishes in the bottom half. They are consistently winning tournaments, getting top 4's and getting the strongest representations in top 10's, while also seeing loads of representation in the bottom half too because hey largest faction with the largest spread of player skill and player types using it.

Siegler's Tau I think is interesting, because I feel in the world of pure theoryhammer, a lot of people were starting to go "there really should be something with Tau, they seem to have the numbers to go the distance." Which prompted a lot of doubt and scorn from existing Tau players who felt their faction was terri-bad and needed a hard reset.

But then it happened, and suddenly we had "Tau are overpowered, nerf pls".

The problem with both systems is that winning tournaments and rates over lots of games are largely a function of "what do the cool kids think works at the moment". It might be that there is a really powerful Tau or GSC combo that exists waiting to be discovered - but unfortunately all the good players are playing Marines/Sisters/Daemons/Harlequins/DG/Custodes/Necrons etc etc so never find it. This might perhaps be especially exacerbated right now, given the number of tournaments globally is way down. The problem is that theoryhammer usually doesn't lie - and its hard to see what those armies can bring that would expect to hold up.

I like to use 7th.
I don't think its hugely contested that Eldar were the best faction. But I don't think their win rate, or "1st place spots in tournaments" was as high as you might have expected. Its just that a bit like SM now, they tended to make up around 30%-35% of all lists. At least in my FLGS, every other table would have Eldar on it (sometimes on both sides). This meant that if you went to a 5-6 game tournament you were highly likely to play Eldar at least twice, maybe 3 times. So if you wanted to get through, your list - and *you* - had to be able to deal with them. And if you went to tournaments regularly you got a lot of practice.

I think the top 4 (or 10, or whatever) is better than win ratio, because player skill differentials is the biggest determinant of winning a game of 40k. For the game it might be a problem if a "bad" Space Marine player usually crushes a "bad" Tau player - but to a degree, its hard to balance on that basis, because you don't know "how bad" they were in that game. Were there lots of bad decisions - or did the player just happen to fall into the good ones without really understanding they were good?

As you get two good players who are theoretically making optimal decisions more often, the statistical imbalances between factions will become more significant in explaining different outcomes. If you hard crunched the probabilities to give normal distributions you'd expect faction X to come out ahead of faction Y more often etc. This points to a fundamental imbalance rather than "why did you move there, why did you shoot that, why did you attack in this order, why didn't your list include this synergy" being the issue.

 Daedalus81 wrote:

Slipspace wrote:
removed the mirror matches

I'm going to challenge this concept, because it's pretty bogus.

Why is an Ultramarines Dreadnought list a mirror to a White Scars VanVet list? When CSM get W2 are they going to be considered marine mirror matches, too?

That depends on how the data is collected. The point is, if the data is calculated counting all SM armies from Codex: Space Marines as a single faction and it doesn't account for mirror matches then the win rate of that self-defined faction will be inaccurate, especially if it's both popular and powerful, which it was at the end of 8th. There have been endless arguments here and elsewhere about whether to break up SM into the various chapters or treat them like all other armies and only track the results per Codex. I was simply highlighting one way in which that method can lead to inaccurate results. It's demonstrably true, as you can see from the debates that happened after the 8.5 SM Codex was released.

 Canadian 5th wrote:

Bharring wrote:
For "at-it's-best", use metrics related to "at-it's-best".

Top placements in top tournaments removes, or at least significantly mitigates, a number of biases.

One bias the W/L ratio has is the presence of non-optimal lists. People who show up at tournies to show off their decade-old army they've only changed to fit newer points/rules still count for W/L ratios. But they don't make the top 8 in large fields, so don't bias those numbers.

Another bias is player skill. While some players can win with anything, so can other players. And so those "some players" will spike the W/L of their chosen faction, biasing it away from what we're looking for. But, in the top 8, they'll be facing other high-skill player, driving down the bias.

Discoverability of the codex's strength is another bias. If the codex can be stupidly OP, but most players don't/haven't realized it, the W/L record will reflect something closer to what the average player has realized. But if you're looking for OP/UP codexes, you care more about the strongest ways to play the codex. If only 5 of 50 players at a tourny know how to build a good list with $faction, the top 8 metric will capture that strength much more clearly than W/L.

W/L and Top 8 are better measures for different things, really. How powerful a book can be tacks a lot closer to Top 8 representation than W/L.

Top 8s can show spikes but can also fail to show which armies are consistent performers and which just got lucky on a given weekend. Is a faction that took 3 of the top 4 positions at a single tournament and then failed to make top 16 the next four better than the faction that finished top 5 at all 5 events and has yet to claim a top 4 spot?

Funny enough you don't look top 4-8 of one tournament...


Automatically Appended Next Post:

HMint wrote:
Top placements are usually just strong lists, not strong armies.
There is one build that is strong, while most of the army is trash -> finishes top. Do you consider this to be a strong army?

Personally I don't, so I vote for W/L. This probably covers more different builds of an army and therefore says more about the army.

Power of book is optimal build. Not lists made with trash units.