Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you: No adverts like this in the forums anymore. Times and dates in your local timezone. Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance. Email notifications for threads you want to watch closely. Being a part of the oldest wargaming community on the net. If you are already a member then feel free to login now.

 Iron_Captain wrote:

But you are an uninformed bystander. Unless you worked for GW or otherwise have inside knowledge of how GW playtesting works, you know as much about it as the rest of us or any random person on the streets. Nothing. Just because we played the game and collect the miniatures doesn't mean we suddenly have knowledge about how GW works. That would be like saying you know how a car works just because you can drive one.
And even if we did have that knowledge, I agree with hollow one that the discussion would still be pointless. A thread and a discussion like this only exist because some people want to vent their frustrations. That is the only use this whole discussion is ever going to have.

Playtesting and balancing isn't some higher science, anyone with decent experience of wargaming and an adequate handle on maths can playtest and balance rules.The key to it is being as thorough as possible (testing as many variations of a given situation with as wide a group of players as you can, as many times as you can, then repeat for the next variant of that situation, record your results, note undesirable results, and alter one factor at a time until balance is achieved (next to impossible, but you have to try). Read Rick Priestley's book on game design if you're interested, he goes into detail about it far better than me.

They do this.

Why else they would have buffed my Shadowsword thrice from Index to Codex? Okay, quadruble times if you also count Doctrines and Stratagems. No, wait, what was the word for five times...?

Testing a game like 40k is like testing software, thousands to tens of thousands of individual components that can interact with each other in multiple ways gives a near limitless number of interactions to check. Let me share with you one of my favorite quotes on software testing:

Edsger W. Dijkstra wrote:Program testing can be used to show the presence of bugs, but never to show their absence!

Given the impossible scope of testing a system like 40k to completeness, Any kind of worthwhile testing has to make assumptions/generalizations about the interactions involved, to reduce the number of specific items to be checked, in testing nomenclature this is called equivalence partitioning. Some of those assumptions will be wrong, and there is a common vector for errors to sneak in even with a seemingly reasonable amount of testing. For instance if we assume a plasma gun is the same no matter who wields it, because it's strength, AP, and damage are fixed values, we run into issue like scion command squads.

Another thing to consider is testing has diminishing returns, you might find 20 bugs in the first hundred hours of testing, and 2 in the second hundred hours. Thus testing is a balancing act between finding bugs and responsible use of time. You could spend hundreds of hours testing something and still not find all of the problem interactions. Which dovetails nicely into picking what to test, unless your equivalence partitions are absurdly large, you will not have time to test them all to the highest level of effort required. So you pick based on importance of the components and probability of the interaction occurrencing. A rule for space marines that doesn't work is more important than a rule for harlequins that doesn't work. A balance issue with plasma weapons is much more likely to come up than an issue with a grav flux bombard. So you take the areas you deem important and or likely, and give those a disproportionate amount of your testing.Thus a unit that was rarely used in prior editions, like conscripts, might have gotten a fraction of the attention a unit like intercessors got, which could lead to all sorts of issues.

The final issue is that GW inherited a rules base that was created by decades of accretion, which included many, many bad decisions. Some issues they were able to drop, like the vehicles rules or d weapons, but other issues had to be worked around as they were part of a brand identity, like plasma weapons sometimes blowing up. In AoS they got to start fresh, which is a whole lot easier than trying to drag a shoddy set of rules into a functional state.

I'm not making excuses for GW, I'm just recognizing from my own experience with similar situations that the task they had in front of them was very hard. With that said there are a few things I think GW could have done better, there insistence on secrecy kept the pool of testers very small, and as most of them were volunteers, this meant the total hours testing the new rules set was probably below what I would have felt comfortable with if I were their test manager. I'd be surprised if there were more than 40 external testers, and they were likely involved in the processes near the end of development after many decisions had already been made. The other problem is that they were likely volunteers, which means you get a lot of work out of them at the front end but not so much at the back end. Rules design is an iterative process, and when you are quickly iterating thru changes with a mostly volunteer staff the final rules get less testing hours than the early rules. GW did this because they are selling the rules as a product, and didn't have the budget to hire the number of testers required for such an expansive rules set. If I were the test manager, I would have argued that we farm out pieces of the testing publicly, things like scenarios with fixed unit compositions, with a mix of old and new rules and observe the feedback. We could get a massive amount of eyes on our issues, and it could be a hype thing, much better than the one page blurbs we got, which were seemingly written months prior to the release of the article, based on old rules that were not going to make the final cut.

hollow one wrote:

I think my post was pretty clear, my suggestion was to have less pointless hate in these forums. If you think that is not valuable to you that's fine. Unfortunately for me it seems that most people agree with you.

There isn't hate.

Pointing completely valid complaints and obvious flaws in a game and a company isn't hate. If you consider that hate, then your posts are nothing short of mindless white knighting, but of course you'd hate to be accused of that, so I'd probably pump the brakes on calling criticism 'hate'.

If you have something useful to add, then by all means, make it. Telling us we're somehow wrong, hateful, ignorant, or otherwise incapable of forming intelligent ideas on how to fix the game is not adding anything useful, and runs completely contrary to your first post in this thread.

So you decide; do you want to be what you espoused in your first post in this thread and add something useful, or do you just want to slag on people for having a different opinion?

"Hundreds of hours".

1) sit down
2) open the draft of the codex
3) write down, like, 20 lists and see which units I keep including and which one I keep excluding

Is something that does not take "hundreds of hours". Is something people do just after they bought the new codex, often with a feeling of just being scammed.

 Kaiyanwang wrote:

3) write down, like, 20 lists and see which units I keep including and which one I keep excluding

There’s a fundamental flaw here. When I do that, I write 20 lists with Terminators and exactly zero with Guilliman, Assbacks or Stormravens. Does that imply that Terminators are overpowered and the others are underpowered? No, because I love Terminators and have no great love for the others.

Your process might work if you were a hardcore power gamer that writes lists for efficiency rather than aesthetic or theme or just what units you think are cool. I don’t believe GW has any staff that think like the former, and instead all of them think like the latter. This is probably a fair assumption, since we’ve heard that the playtesters are just rules/fluff writers who playtest between other duties, and I don’t think people with a power gamer mindset would be effective Codex writers in general. The feedback from the ITC guys play testing was probably the most valuable from a tournament balance perspective, but they may well have had less influence than GW’s in-house staff play testing.

It's been stated several times that GW doesn't playtest with a WAAC/power-gamer/hyper-competitive, "break the game" approach in mind.

So what if it doesn't? 'Competitive' players are (generously) estimated to make up perhaps as much as 15% of the TOTAL player base. It's been (wisely, in my opinion) pointed out that a number of factors likely affect the playtest process, including deadlines, diminishing returns, and operating under the various limitations that a volunteer staff brings with it. Why would GW chain itself to putting out a product that 15% (generous, again) of their customers will find to be 'tightly-written and thoroughly unbreakable' when 85% or more of their customers want pretty much exactly what they've gotten: a game that's fun to play?

DCannon4Life wrote:
It's been stated several times that GW doesn't playtest with a WAAC/power-gamer/hyper-competitive, "break the game" approach in mind.

So what if it doesn't? 'Competitive' players are (generously) estimated to make up perhaps as much as 15% of the TOTAL player base. It's been (wisely, in my opinion) pointed out that a number of factors likely affect the playtest process, including deadlines, diminishing returns, and operating under the various limitations that a volunteer staff brings with it. Why would GW chain itself to putting out a product that 15% (generous, again) of their customers will find to be 'tightly-written and thoroughly unbreakable' when 85% or more of their customers want pretty much exactly what they've gotten: a game that's fun to play?

Because if you design the game to be balanced for the competitive crowd, you wind up with a better game for the casual crowd.

That's what they always used to do. If you modify the target then -1 is good and +1 is bad.

No, it used to be when you had a negative to hit. You just needed to roll a 4+ rather than a 3+ for example. Everything else stayed the same. A 1 was just a 1 and a 6 was just a 6.

Now its changed, for example the 2 you rolled is modified to a 1, and that 6 is modified to a 5.

It's different and greatly affects how dice rolls are made.

Because if you design the game to be balanced for the competitive crowd, you wind up with a better game for the casual crowd.

Exactly. You don't balance a game for the casual crowd. They don't care either way and will take whatever they want regardless of how it performs on the battlefield. You balance the game against WAAC players. If the game is balanced for them, it automatically balances for everyone else.

 Brutus_Apex wrote:

That's what they always used to do. If you modify the target then -1 is good and +1 is bad.

No, it used to be when you had a negative to hit. You just needed to roll a 4+ rather than a 3+ for example. Everything else stayed the same. A 1 was just a 1 and a 6 was just a 6.

Now its changed, for example the 2 you rolled is modified to a 1, and that 6 is modified to a 5.

It's different and greatly affects how dice rolls are made.

Because if you design the game to be balanced for the competitive crowd, you wind up with a better game for the casual crowd.

Exactly. You don't balance a game for the casual crowd. They don't care either way and will take whatever they want regardless of how it performs on the battlefield. You balance the game against WAAC players. If the game is balanced for them, it automatically balances for everyone else.

Balancing the game for WAAC players is an impossible feat for a small team, the only way to pursue this is how they are doing it right now. Throw out there something that is reasonably working, wait for someone to inevitably break stuff and fix it. Logic wants that each times someone manages to break stuff, it will have a smaller impact on the game. We are getting 3 global fixes per year now! We used to have between one every 2 years or even 7 years depending on your faction.

 Brutus_Apex wrote:

That's what they always used to do. If you modify the target then -1 is good and +1 is bad.

No, it used to be when you had a negative to hit. You just needed to roll a 4+ rather than a 3+ for example. Everything else stayed the same. A 1 was just a 1 and a 6 was just a 6.

Now its changed, for example the 2 you rolled is modified to a 1, and that 6 is modified to a 5.

It's different and greatly affects how dice rolls are made.

And thats good. If you're -1 to hit something then its actually still hard to hit as opposed to a minor inconvenience for models with rerolls. They way they've done the modifiers is actually pretty clever because they're harder to mitigate.

Yea I think it's pretty clear that GW don't playtest 40k enough to catch all the outliers and weirdly costed units. A cursory glance at almost any index tells us this. Or likely the people responsible for play testing have their favourite armies and play them more than others.

Either way it looks like (hopefully) GW are now listening to their Players and are even involving us in the testing (look at the beta rules).

Anything as bad as the Dakkajet-DeffKopta comparison needs to be reported to GW so they can sort it. I'm sure it was unintentional but they can't fix what they don't know about.

Spoletta wrote:

 Brutus_Apex wrote:

That's what they always used to do. If you modify the target then -1 is good and +1 is bad.

No, it used to be when you had a negative to hit. You just needed to roll a 4+ rather than a 3+ for example. Everything else stayed the same. A 1 was just a 1 and a 6 was just a 6.

Now its changed, for example the 2 you rolled is modified to a 1, and that 6 is modified to a 5.

It's different and greatly affects how dice rolls are made.

Because if you design the game to be balanced for the competitive crowd, you wind up with a better game for the casual crowd.

Exactly. You don't balance a game for the casual crowd. They don't care either way and will take whatever they want regardless of how it performs on the battlefield. You balance the game against WAAC players. If the game is balanced for them, it automatically balances for everyone else.

Balancing the game for WAAC players is an impossible feat for a small team, the only way to pursue this is how they are doing it right now. Throw out there something that is reasonably working, wait for someone to inevitably break stuff and fix it. Logic wants that each times someone manages to break stuff, it will have a smaller impact on the game. We are getting 3 global fixes per year now! We used to have between one every 2 years or even 7 years depending on your faction.

Correct, but often it seems they aren't even attempting to get things close. You always will have to adjust things later that you miss, but GW tends to miss a lot that they shouldn't necessarily miss. Unfortunately GW seems to have (or had in the past, although I haven't seen any real indicator that they have changed this mindset) the mentality that everyone plays like they do, so not only do they miss the blatant WAAC-type of listbuilding (which is somewhat understandable) they also tend to miss a lot that even decent players can find and break, and even if not their balance in factions are all over the place so you still end up with the person liking Unit X which is super strong just crushing the person who likes Unit Y which is very weak, without either of them trying to break the game.

Balancing the game for WAAC players is an impossible feat for a small team, the only way to pursue this is how they are doing it right now. Throw out there something that is reasonably working, wait for someone to inevitably break stuff and fix it. Logic wants that each times someone manages to break stuff, it will have a smaller impact on the game. We are getting 3 global fixes per year now! We used to have between one every 2 years or even 7 years depending on your faction.

It probably is hard, but that doesn't mean they shouldn't try to. I'm not asking for perfect balance, but something close i believe is feasible.

I will give credit where credit is due. They are attempting to fix things in a short time period, and thats a good thing. Obviously things will get missed and need to be fixed after release accordingly.

However, I don't feel like many of the issues that are currently affecting the game would even exist if the team had put more thought into creating a better system. Specifically things like the psychic phase, characters not joining units, morale, cover, terrain. etc.

They seem to think that it's a good idea to make a rules pamphlet and make it more complex later with fixes and FAQ's added after the fact. I think this is a terrible strategy. They should have just made a complex, in depth rules set to begin with and thought through all the permutations before they released it, not after.

I personally don't give a gak if little Timmy can't figure out 50+ pages of rules. Thats his problem. I grew up with it, they can too.

And thats good. If you're -1 to hit something then its actually still hard to hit as opposed to a minor inconvenience for models with rerolls. They way they've done the modifiers is actually pretty clever because they're harder to mitigate.

I disagree, because now we have situations where fighting at night somehow makes plasma weapons more dangerous, or because you are wielding a power fist makes you are somehow not as mad at the imperium. Not to mention how counter intuitive it is and slows the game down.

I think too many of you are over stating the case for GW playtesting and specifically seem to have fallen into the trap that the game is too complex to ever be playtested enough to satisfy everyone.

Here is the thing. I agree. This game is FAR to big to be playtested by a handful of testers and to fix every flaw. It just isn't possible.

HOWEVER! and I really can't over state this, GW clearly didn't even do the bare minimum because if they had they would have noticed ENORMOUS holes in their game without doing anything more then glancing at he proposed points costs.

Nobody, and I mean NOBODY, even remotely familiar with the ork codex thought that Deff Koptas were even remotely priced correctly. The same can be said for most units in the index. Warbikers gained 1 wound, lost their cover save and went up in price 50% and GW is still trying to figure out why nobody plays them. How about our beloved Stompa. Only 1 person in the entire universe thought that a 1,000pt Stompa was a good price, and that 1 person was Reese from FLG.

What about weapons/gear? Who wouldn't gladly pay 5pts on 1 model to give it a 6+ FNP save? Ohh and did I mention it doesn't stack and you can only have 1 per squad of Nobz?

What about the PK. It went from a 25pt CC weapon that could theoretically kill any vehicle in the game in a single turn (Barring super heavies) to causing -1 to hit for the bearer and at the most doing D3 damage....but it still was priced the same. I mean hell, even at its reduced price (about 1/2 the cost) it is still iffy and not an auto include in most armies, hell I still take BCs instead.

Ohh and my personal favorite, the combi Rokkit for Orkz. Talk about a MASSIVE waste of points. 12pts for a regular rokkit and its like 17 for a combi-rokkit which hits on 6s if you fire both (I'm not 100% on the price because it was so stupidly Over priced I have never used one)

So to conclude that rant, yes the game is to complex to be properly playtested in a short amount of time, however, the giant glaring errors should have been easily noticed and corrected

 An Actual Englishman wrote:
Yea I think it's pretty clear that GW don't playtest 40k enough to catch all the outliers and weirdly costed units. A cursory glance at almost any index tells us this. Or likely the people responsible for play testing have their favourite armies and play them more than others.

Either way it looks like (hopefully) GW are now listening to their Players and are even involving us in the testing (look at the beta rules).

Anything as bad as the Dakkajet-DeffKopta comparison needs to be reported to GW so they can sort it. I'm sure it was unintentional but they can't fix what they don't know about.

GW may be listening since they got kicked in the bottom line and came round to the fact that the world had turned but the results of that are worrying, conscript nerfed and then ending up at the same cost as guardsmen, the smite beta rule which seems worse every time I think about it etc.

Ix_Tab wrote:

 An Actual Englishman wrote:
Yea I think it's pretty clear that GW don't playtest 40k enough to catch all the outliers and weirdly costed units. A cursory glance at almost any index tells us this. Or likely the people responsible for play testing have their favourite armies and play them more than others.

Either way it looks like (hopefully) GW are now listening to their Players and are even involving us in the testing (look at the beta rules).

Anything as bad as the Dakkajet-DeffKopta comparison needs to be reported to GW so they can sort it. I'm sure it was unintentional but they can't fix what they don't know about.

GW may be listening since they got kicked in the bottom line and came round to the fact that the world had turned but the results of that are worrying, conscript nerfed and then ending up at the same cost as guardsmen, the smite beta rule which seems worse every time I think about it etc.

Did they get kicked in the bottom line? I might be wrong, but all reports I heard were that business was still booming this edition.

At the start I'm sure sales were great but as they never have things in stock that's going to hurt them, then you have the increasing dissatisfaction with each codex release putting people off.

I imagine one good half year report followed by one bad.

Well we'll see. I feel like the feeling outside of places like this is not generally so negative though. At least in my experience.

Obviously the stock issues will hurt, but that's circumstances beyond their control.

Here is my own findings on playtesting and balancing the game I'm developing. Not to blow my own horn or anything, it sheds some light on the main issues I encountered, and the nigh-on impossibility of getting points values to be anything more than reasonably accurate. Most of you are probably aware of this to some degree already, it'll be informative to those of you who aren't.

Points Values, Balance and Relativity

While we have done our utmost to balance the entries for wargear, units and armies, please note that these (painstakingly calculated!) values are prone to a considerable number of factors that can’t be accounted for in the design room. However it can be accounted for by you the player. That’s right –it’s up to YOU! Allow us to explain:
Points values are calculated based on the general usefulness and effectiveness of units in the game as a whole. This does not mean that two opposing units of equal points value will perform equally against one another. While superior tactics and strategy will greatly outweigh such disadvantages, there are a few notable factors which can skew the balance of power: Faction bias, unit specialisation and situational value.

Faction Bias
Not all armies are equally good at every aspect of warfare. While there are units of every role available for each faction, some factions simply excel in certain areas, and suffer in others. For example, Mayaraan armies have access to the fastest units, best skirmishers and very accurate shooters, but most of their units aren’t very durable. So while they will excel at mobile strikes and evasive tactics, in a static shoot out with a Republic gun line they will be at a disadvantage. That is not to say that a Mayaraan player should never engage in trench warfare, but to be aware that they will come up against other factions that can do that job, better.

Specialisation vs Generalisation
The above is a particularly issue with units that are designed and/or equipped for a certain role. E.g. fielding support weapons squads loaded up with laser cannons will most likely defeat an opponent fielding a large number of tanks. But such slow firing weapons will perform poorly against hordes of light infantry. Thoughtful deployment and manoeuvring will reduce the chances of this happening, but even so specialised units will occasionally find themselves vulnerable or lacking a suitable target. On the other hand, support weapons team with a mix of anti-tank and anti-infantry weapons will be more flexible, but will not excel at destroying either target. So while specialist units and unit builds have the potential to be more effective, generalist units are the more stable and flexible choice. When choosing a force, players should be mindful of the benefits of forming a balanced army.

Situational Value
On a related note to specialisation, some units have intrinsic abilities (or paid for perks and equipment) that significantly increase their effectiveness in specific situations. E.g. skirmishers with urban fighter and camo gear will prove very effective for their cost if they are in cover in ruins or buildings, but if those same units clashed in an open ground, the skirmishers, unable to avail of either of the abilities they have paid for, would be quite underwhelming.
Once again, placing and moving units so they can utilise their abilities is the mark of any good commander. When creating an army list, thought should be given to selection of such perks and equipment. Particularly with regard to what enemies you anticipate and what terrain you will be fighting on.

NOTE: It is good practice for players to come to some sort of consensus on terrain choice and placement. Deciding in advance to fight on a certain battlefield type (e.g. forests, open fields and a few deep bodies of water) will give both players a chance to assemble a suitable force.

You may want to build nicely stylised battlefields to fight over. Whether it’s a half-ruined town with narrow winding streets, or open meadows with only a few sparse stands of trees for cover, remember to make it usable to both armies. Are the streets of that town to narrow for vehicles? If so, then make some of them wider. Is your opponent complaining about lack of cover for his light infantry? –then place more cover. If in doubt, taking turns to place terrain or randomising it’s placement are good methods for avoiding arguments over advantageous setups.

Of course in some scenarios, such as Last Stand, and Ambush, it is intentional that the terrain is in favour of one player!

hobojebus wrote:
At the start I'm sure sales were great but as they never have things in stock that's going to hurt them, then you have the increasing dissatisfaction with each codex release putting people off.

I imagine one good half year report followed by one bad.

I doubt very much that selling everything they can make is going to result in a bad half year report. Selling as much as you can might not be as good as selling everything you could have sold but that doesn't make it bad.

They arnt out of stock because of sales but because of technical problems in the factory.

I would like to see pictures of the table setups and the play testing games as they unfold. What are the criteria for army selection when play testing? Game time limits? Is computer simulation implemented?
I still don't think they play test with massive spam list in mind. A 2000 pt blue horror spam list? You can't expect them to play test for that kind of stuff.

Some people can’t handle change.

 Primark G wrote:
Some people can’t handle change.

What does this have to do with anything?

The post that listed problems because of things that have changed like characters and flyers.

 Primark G wrote:
The post that listed problems because of things that have changed like characters and flyers.

It really helps if you quote the post you are referring to. Otherwise it can get pretty confusing.

 Kaiyanwang wrote:
"Hundreds of hours".

1) sit down
2) open the draft of the codex
3) write down, like, 20 lists and see which units I keep including and which one I keep excluding

Is something that does not take "hundreds of hours". Is something people do just after they bought the new codex, often with a feeling of just being scammed.

Good thing you're not on the playtesting team. I think this is the only testing methodology I've ever seen that would be worse than simply not testing at all.




Automatically Appended Next Post:

Ix_Tab wrote:

 An Actual Englishman wrote:
Yea I think it's pretty clear that GW don't playtest 40k enough to catch all the outliers and weirdly costed units. A cursory glance at almost any index tells us this. Or likely the people responsible for play testing have their favourite armies and play them more than others.

Either way it looks like (hopefully) GW are now listening to their Players and are even involving us in the testing (look at the beta rules).

Anything as bad as the Dakkajet-DeffKopta comparison needs to be reported to GW so they can sort it. I'm sure it was unintentional but they can't fix what they don't know about.

GW may be listening since they got kicked in the bottom line and came round to the fact that the world had turned but the results of that are worrying, conscript nerfed and then ending up at the same cost as guardsmen, the smite beta rule which seems worse every time I think about it etc.

And classic Dakka financial reporting. The thread is finally complete.

Given the speed GW is trying to get out all the old codexes I don't think it was particularly practical for GW to test every scenario. However there are glaring issues. It is unclear why these exist but I would postulate that it might be due to popularity in the testing regime itself. It is possible that more successful armies get tested more frequently (both from sales and personal perspective) which might leave less popular armies to fall back on the base formulaic approach. Additionally only a few playtests of certain armies/units leaves them open to looking a lot better/worse than they are in reality because of statistical freak results (i.e. a few games where you roll really high).

I think a better approach would have been for GW to release draft pdfs of all the armies (except those with new models, which in the end is most of the current books being released) so that the gaming populace could have identified the worst issues (like a beta). That would have released GW to focus on the new books as they balance them against the increasing database of existing armies.

Not obvious how breaking down the codex is a bad approach to balancing. If you have two platforms for say carrying a lascannon but one is dramatically more expensive for no obvious benefit its probably a poor design. Most extended codexes suffer this problem - Marines and Orks are full of entries that do the same thing, whether melee or shooting.

With that said if the difference was marginal it might justify aesthetic choice. Often though you have units which are 25%-50% better and so its a no brainer.