Switch Theme:

Statistical Analysis of the 2018 Tournament Season - Primary Faction Win Rates  [RSS] Share on facebook Share on Twitter Submit to Reddit
»
Author Message
Advert


Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you:
  • No adverts like this in the forums anymore.
  • Times and dates in your local timezone.
  • Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance.
  • Email notifications for threads you want to watch closely.
  • Being a part of the oldest wargaming community on the net.
If you are already a member then feel free to login now.




Made in us
Rampaging Carnifex





South Florida

First off, I want to thank Peter for all his work curating the database at 40kstats.com

I wanted to share some of the analyses i've started of the data covering the entire 2018 tournament season (you can find the data here). Peter and the folks at Chapter Tactics have mainly talked about means and trends in these data, but have not (to my knowledge) discussed any formal statistical analyses.

My first slice at these data is looking at primary faction win rates. Obviously, list building is much more complex than this, and looking at just primary faction is a limitation of these analyses.

You can see the technical methods details for these analyses here:
Spoiler:
For my first analyses of these data, I ran a a multi-level mixed effects model on the individual player tournament-summary data (Excel tab "Data" from the downloadable 2018 master spreadsheet ). The outcome variable was win percentage. I excluded "dark mechanicus", "renegade knights", renegade guard", "corsairs", "adeptus titanicus" and "inquisition" from the analyses. Observations were nested within players, and I analyzed the fixed effects of Primary Faction. I allowed each Faction to have their own residual variance.

My SAS code is here so you can see the model specification:

proc mixed data=tournament2018 method=ml covtest ;
class Player Tournament Faction ;
model Win_Percentage = Faction /solution cl chisq ddfm=bw outp=pred;
random intercept / subject=Player;
lsmeans Faction/ pdiff cl ;
repeated / group=Faction subject=Player;
run;
ods pdf close;





Here you can see the model estimated means of win percentages by faction, plotted alongside the observed means from the raw data. As you can see, the model recreates the observed means closely - except the observed means are likely overestimated, probably because the simple means aren't accounting for particular players driving the means up or down in contrast to the model. Also plotted on all the means are the confidence intervals - this gives us an indication of the precision of our estimate. Our best guess about the true mean out there in the world falls within that confidence interval. This is useful because the means themselves might be misleading alone. Genestealer Cults, for instance, have a high win percentage, but the 95% CI is actually quite large, so we can't be as confident about those results as we might be about a faction like Knights. After all, we have less data for Genestealer Cults than other factions.

There was a highly significant effect of Faction, F (23, 290) = 10.52, p < .001, indicating that some factions had significantly higher win percentages than others.


Here you can see the results of each individual mean comparison between factions. I corrected for multiple comparisons by using false-discovery rate (FDR). This table summarizes the significant differences (accounting for FDR) between each factions' win rates. You can read the table by finding your faction on the vertical axis, and comparing it to a faction on the horizontal axis. A negative value means your faction (on the y-axis) performs significantly worse generally than the faction (on the horizontal axis) and a positive value means your faction performs significantly better than that faction. The values themselves are the model-estimated difference in win-rate. Do note, this is not the how well your faction performs against the different factions. This is a comparison on mean win rates overall. The results are generally in agreement with what you might expect from eye-balling the means. However, there were also a lot of non-significant differences between factions - meaning we don't have evidence either way to know which faction is better than which. Non-significant comparisons are left as blank squares in the matrix.

Finally, I conducted a k-means clustering of the model estimated means of win rate by primary faction to see whether we can summarize these results based on statistical evidence for "rankings" of different factions. A 3 cluster (tier) solution was the best solution for the data. That is, adding additional "tiers" didn't explain a whole lot more variance over lower cluster solutions. The 4 cluster solution, for instance, just explains a little more variance than the 3 cluster solution, and splits Drukhari and Ynnari into their own top-tier ranking.


I interpret the clustering results as pretty hopeful, honestly. It means that armies generally fall into just 3 ranks. Poor, average, and good armies. Furthermore, an examination of the model estimated means shows that the vast majority of armies has a 95% CI that overlaps with a 50% win rate - which is the ideal win rate for an army if there is nice balance between factions.

Let me know if anyone has any questions, or ideas for additional analyses. This is really just a first pass at these data - as it's a really rich dataset.



This message was edited 2 times. Last update was at 2019/02/28 16:26:23


   
Made in us
Journeyman Inquisitor with Visions of the Warp






Dimmamar

I'm glad to have some statistical backing regarding how garbage my Grey Knights are. And now I can show my TSons friends that no, they are not even CLOSE to the same boat as me!

40k Resources

LVO 2017 - Best GK Player

The Grimdark Future 8500 1000 1500 4500

"[We have] an inheritance which is beyond the reach of change and decay." 1 Peter 1.4
"With the Emperor there is no variation or shadow due to change." James 1.17
“Fear the Emperor; do not associate with those who are given to change.” Proverbs 24.21 
   
Made in us
Rampaging Carnifex





South Florida

Yes! Virtually every faction performs better on average than Grey Knights at tournaments.

   
Made in us
Missionary On A Mission




Tacoma, WA, USA

At the risk of complicating your calculations, did you adjust the data to exclude Mirror Matches (games where both players have the same Primary Faction)? Those will get an equal win/loss rate and can warp the numbers if there are enough in the sample.
   
Made in us
Rampaging Carnifex





South Florida

No, this is an analysis of a players overall win rate over the course of tournament. It could very well have happened that mirror matches popped up, and this will have affected the overall win rates. Something like what you are suggesting could be done.

   
Made in us
Missionary On A Mission




Tacoma, WA, USA

I figured as much. I imagine it would be a lot of work to pry those out of the numbers since they warp them to an extent when trying to do a faction to faction comparison. Not important on the player to player comparison.
   
Made in us
Longtime Dakkanaut





Primary faction is 50% of the army's points value.

For any army capable of allying, the data for mixes armies versus mono faction is convolutes together.

It would be more interested to see data for only 100% mono faction added. Would allow to see which factions are/have Ally options that skew their results.

Currently that information is buried in the results you posted.
   
Made in us
Stealthy Warhound Titan Princeps






blaktoof wrote:
Primary faction is 50% of the army's points value.

For any army capable of allying, the data for mixes armies versus mono faction is convolutes together.

It would be more interested to see data for only 100% mono faction added. Would allow to see which factions are/have Ally options that skew their results.

Currently that information is buried in the results you posted.


Very few people who have the option to take allies play mono-faction in tournaments. Sure, it happens, but its certainly not the majority.
   
Made in us
Longtime Dakkanaut





Of course it's not the majority, and that is the point.

Let's take drukhari for example, they are basically in second place in the graph above but that is not reflective of how powerful they are, but rather how powerful the Ally system is. If you can see the difference between the mono faction and primary faction result you can judge how much of that is based on some synergy combination outside of the primary faction.

Much like knights, mono knights doesn't do as hot as knights+AM

It's even more significant when 3 factions are involved since the primary faction is under 50% many of the times, so it's more representative of the Ally issue than the primary faction being powerful.

This message was edited 1 time. Last update was at 2019/03/04 04:03:28


 
   
Made in fi
Decrepit Dakkanaut





Problem is thus because not many do that there's not enough statistics for it...

https://middleagedstrategybattlegamers.home.blog/2019/10/20/tneva82-tournament-report/ <- lotr painting blog

12 factions for Lord of The Rings
11772 pts(along with lots of unpainted unsorted stuff)
5265 pts
5150 pts
~3200 pts Knights

 
   
Made in us
Horrific Hive Tyrant




Tampa, FL

Now see how many of those were soup. I'm getting a bit tired of this "Look how diverse everything else!" ITC propaganda when it's soup almost all the way down with the same components cropping up. That's not actually diversity.

- Wayne
Formerly WayneTheGame 
   
Made in us
Been Around the Block





Will the stats change for GK post CA2018?
   
Made in nl
Longtime Dakkanaut





 _SeeD_ wrote:
Will the stats change for GK post CA2018?
GK's are fundamentally flawed. More expensive marines that die like any other marine is just not good.
   
Made in us
Been Around the Block





 Ordana wrote:
 _SeeD_ wrote:
Will the stats change for GK post CA2018?
GK's are fundamentally flawed. More expensive marines that die like any other marine is just not good.

That's what I'm afraid of too. I'm hoping for a reduction in CP costs within the next year. Maybe I should post a cool picture of GK on the warhammer40k subreddit every day until they change it
   
 
Forum Index » Tournament Discussions
Go to: