Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you: No adverts like this in the forums anymore. Times and dates in your local timezone. Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance. Email notifications for threads you want to watch closely. Being a part of the oldest wargaming community on the net. If you are already a member then feel free to login now.

BrianDavion wrote:
true but following D&D 3.5 WOTC relased a new edition that was balanced, and everyone agreed 4th edition sucked.

But at least it doesn't suck because of balance issues.

BrianDavion wrote:

WayneTheGame wrote:
Is it possible? Yes. But GW doesn't care, either because they're really that stupid or more likely because they aren't taking into account that you can do something, because they'd never think of it.

It reminds me very much of 3.5 D&D, which while it was (supposedly...) playtested, it was common knowledge and verified by third-party playtesters that WotC had specific ideas of how the game should be played, so willfully ignored things that were broken because they'd never in a million years consider doing it and therefore incapable of realizing that it was broken and needed to be fixed. For example, in base D&D it was possible for a high-level Druid to completely break the game by using Wildshape to turn into something ridiculous like a Dire Bear or a T-Rex or whatnot (I forget the specifics). But no designer would ever consider that, so it was never addressed (of course D&D being D&D, there was at least the DM who could reel in abuse). Or, Wizards/Sorcerers commonly used utility spells to render creatures out of a fight (what was commonly known as "save or suck" spells), but WotC played Sorcerers as blasters, taking offensive spells that generally were lackluster compared to utility. Then you had the Character Optimization group (aka the powergamers) who would purposely find those broken combos, sometimes just for the lulz but often to point out "This is broken and needs to be fixed, here's why" and to showcase the lack of general thought given by the designers. All in all though that's not a huge deal in a tabletop RPG because you have a GM to arbitrate things and nip abusive combos in the bud. Not so in a wargame in the vast majority of circumstances.

I think GW is in the same boat - you'll never see half the abusive combos come up in their games because that's not how they play, so they can't fathom that anyone would play like that at all. The fact that 40k (as of 6th edition anyways) is clearly meant to be played casually with a gaming club or known group lends itself more to that fact, as what kind of spanker would play something deliberately broken against their mates, knowing it would ruin their fun, just to win?

true but following D&D 3.5 WOTC relased a new edition that was balanced, and everyone agreed 4th edition sucked.

4e sucked because they took away flexibility and shoehorned in a specific way to play (which 40k does too) that made it feel super bland (I liked 4e, but it was really bland). And instead we have Pathfinder which largely fixed a lot of the issues with 3.5 but didn't go to the other extreme.

It's a lot easier than the 9999999999 hours estimate, for several reasons:

1) Playtesting doesn't necessarily mean full games between 2000 point armies. You can test the opening turns, set up a scenario halfway through the game and see how an army's endgame position looks, etc. The goal isn't just to compile a win-loss record, it's to see how things function. If something is significantly overpowered then it should be pretty obvious, and you can just end the game and send the unit/rule back for redesign.

2) You don't have to test every possible combination. 40k has a lot of armies that are very similar, so you can combine all marine armies into a single opponent to test against, group similar armies into a single archetype, etc. After all, what matters for initial balance testing is how an army deals with gunline opponents, the precise differences between Tau gunlines and IG gunlines aren't all that important.

3) You don't have to do it all at once. GW only faces the potential of 9999999999 hours of playtesting because they've been so incompetent in the past and created a situation where they need to redesign and rebalance the entire game just to get it up to an adequate level (forget making an excellent game, that's probably years of work away from where GW is now). If GW hadn't failed so spectacularly at maintaining the game then all they would need to do is playtest each new release as it happens.

4) You can hire more than ten playtesters. Hire a hundred playtesters and cut that down to four weeks of playtesting. Or, since a new edition is a major event that should involve years of development time, you can spend the same amount of weeks on playtesting while doing a much better job of it.

Peregrine hit the nail on the head. Proper playtesting isn't just playing a game, it's playing something specific and setting up conditions. So you might set up a hypothetical in-progress game to test some rule, not play out a full game and if nobody actually does that, not care, which is how GW seems to playtest.

For instance to playtest the daemon spam nonsense, you'd set up that kind of situation to see what happens.

MMOs seem to manage this, and they are even more complicated than 40K

Computer simulation could speed it up a lot. Run models through high power servers. Add basic programing it could basic play test just about maybe.

Leave advanced stuff to humans it just handles the basics however and saves a lot off time,

I've playtested for other games.

To determine point values, you take you new unit and play it against the same amount of points of similar units that you already trust the point value of. Have a bunch of testters do this do this 2-3 times against each comparative foe and compare the results.

When the win/loss ration is about 50/50 you've hit the right point value.

 scottmmmm wrote:
MMOs seem to manage this, and they are even more complicated than 40K

Every MMO out there has numerous bugs on a regular basis and nothing in them is ever balanced though

Zodiark wrote:

 scottmmmm wrote:
MMOs seem to manage this, and they are even more complicated than 40K

Every MMO out there has numerous bugs on a regular basis and nothing in them is ever balanced though

MMOs are arguably a different beast. You strive for some balance but not a lot because a lot of it comes down to player skill in PVP, and PVE has no equivalent in 40k. 40k still has almost worthless levels of balance.

Also, MMO developers listen to their customers, and explain their reasoning when they believe something. And MMO developers never play some bullgak like "forge the narrative" to handwave away imbalances; gross imbalances get fixed, and everything is close enough that skill matters the most. You'd never see a WoW developer for example say that it's part of the narrative that a Rogue can defeat a Warrior in PVP, so the Warriors should stop complaining.

yet that argument works for 40k.

Yeah, so why doesn't everyone stop comparing 40k to anything other than another miniatures wargame?

Makes sense to me.

WM/H, Flames of War...far superior rules. Companies actively listen to their customers. Including competitive players.

I know Spartan Games runs a beta test team.

I don't know much about Dystopian Wars, but Firestorm Armada has seen some vast improvements since their first edition, both in terms of rules clarity/depth, and balance between factions.

Further, they actually use the concept of a living ruleset. Changes, tweaks, and additions are ongoing.

All of these things take effort, but they're well worth it.

The surest way to playtest (though playtesting internally while caring about balance does help) is to crowdsource it to your playerbase. As a designer you can be pretty sure that there's going to be some flaw in your game, somewhere, probably a lot actually, and it's the huge numbers of eyes of your players that will locate them.

Talk about "competitive players" is fine, and it's good to have those types of players testing your stuff, but they aren't the be all and end all. Balance doesn't "trickle down" - some things are great in the hands of or against experienced players, but not weaker players, and that's true all the way through the skill spectrum. You need to have a variety of players testing your game. Fortunately, crowdsourcing is great for that too.

The other problem is when it comes to actual balance rather than clarity of rules, just testing before you release isn't enough. It will likely turn out that models you thought were going to work don't, because for whatever reason their upsides just aren't useful or their downsides are worse than you thought they would be. I'll point to Warmachine for an example of this, because there are certain models that could have been good in a different world, but just aren't in this one, like the Dire Troll Blitzer, a model that can fire numerous shots that can't hit anything it can kill and can't kill anything it can hit. There's nothing to say this couldn't have been a fine model - it could work in a game, just it doesn't work in Warmachine. When stuff like that happens, you need to be able to correct it after the fact. It won't necessarily be apparent until the metagame has had time to develop.

) You can hire more than ten playtesters. Hire a hundred playtesters and cut that down to four weeks of playtesting. Or, since a new edition is a major event that should involve years of development time, you can spend the same amount of weeks on playtesting while doing a much better job of it.

why don't they do open playtests . It took how long for people to notice that eldar can get a++2 hit and run abomination in 6th ?5 min tops after reading the ally section . Same with buffmanders joining centurion stars etc.
FW seems to do something like open playtesting that with their experimental rules. They show rules , get feed back about it being crap or OP , and then when the model gets real rules in a book it is either still bad or nerfed or stays the same if it is eldar.

Makumba wrote:

) You can hire more than ten playtesters. Hire a hundred playtesters and cut that down to four weeks of playtesting. Or, since a new edition is a major event that should involve years of development time, you can spend the same amount of weeks on playtesting while doing a much better job of it.

why don't they do open playtests . It took how long for people to notice that eldar can get a++2 hit and run abomination in 6th ?5 min tops after reading the ally section . Same with buffmanders joining centurion stars etc.
FW seems to do something like open playtesting that with their experimental rules. They show rules , get feed back about it being crap or OP , and then when the model gets real rules in a book it is either still bad or nerfed or stays the same if it is eldar.

Open playtests go against GW's mentality and thought process, keep that in mind. They don't *need* to playtest because they're the best in the business. They don't need to playtest because nothing can be broken or OP, and if it is then it's the fault of the players for not properly discussing the game beforehand.

Since no one has mentioned it yet, perhaps we should talk about the realistic feasibility of large scale playtesting. We know that they *do* playtest - we've seen early playtest versions of very broken things leak that were cleaned up later. But people feel that isn't enough (as evidenced by this thread and hundreds others like it). There are two problems here.

The first is that suggesting "hiring competitive players" not only assumes that they don't do this already, but also gives an overabundance of credit to the current competitive player scene. While the giant braintrust (and I mean that without irony) that is the competitive gaming circuit does eventually find all of the holes in the game, they don't do so within a matter of weeks. It takes months, sometimes up to six months, before the competitive gamers have worked out a codex's full potential. The competitive scene always focuses on breaking the big things that end up pretty well balanced while ignoring the smaller ones that crop up and become a problem in the scene later. Right now, even in this very thread, people are mentioning summoning demon engines as the biggest problem in the game, ignoring that it's not consistently winning. Same thing happened with Superheavies back in December. The Revenant Titan made everyone wet themselves, the competitive community banned it, called it unbalanced. Then people actually playing against it, adapted, and found it wasn't scary after all. Those RT armies began to consistently lose. And 2++ rerollable demons didn't show up until months after the codex was out. Before then Flying Circus was all the rage and it took a while for anyone to notice the real problem.

So how long exactly do we expect them to playtest the rules publicly, using all of this brain power? Three months? Six months? FW does that with its weirder models and the result is that fewer people buy the models. Players want to wait to see if they get nerfed after the playtest and competitive players refuse to play against them because they aren't balanced or not part of the meta they're training for.

Which leads to the second problem, which is that of GWs business model. The largest games in the world are so because they either A) consistently shift the balance of power in the meta (codex creep) B) phase out older units/cards and/or C) change the rules between editions radically enough that you have to buy all new things to stay current. GW needs to move models; it's the point of their business. Creating a truly balanced game means allowing customers to buy enough models to play, never needing to buy more. Some companies do this. The companies that do don't stay around long. The sad truth is that by creating a perpetual state of shifting imbalance, competitive players keep buying new models.

So GW could publicly playtest, but the process would be longer than what is being suggested and the end result would be fewer models sold. This is, at least, the thinking that gets us here.

They fail at one of the most basic rules of design. Always show your work to someone who had no part in making it before you publish. Thats it.

GW doesnt do that, so they will never achieve the perfect product (or get close).

So when they make the rules they have made it for themselves and only themselves. The way the book is written is in a way that assumes we know how certain things interact (which to them makes perfect sense I assume). However if they just showed the rules to someone who did not have any role in design then they would (in theory, if they where dedicated enough to look into it) spot a lot of discrepancies within the text.

Before I publish anything professional I get someone else to look through it and check everything. Even if they have nothing to do with it. Its the most basic way of creating something decent.

In terms of playtesting, I think it improves nothing unless the play testers have nothing to do with the creation of the game itself. Who cares if the designers playtest it and think its great, it only matters if someone random gets selected for it and can figure it out and give it an honest review.

It would solve a lot of problems if they did that. But instead they themselves (I assume) playtest the rules and dont have any fresh perspective or uninformed opinions on their work.

 Swastakowey wrote:
It would solve a lot of problems if they did that. But instead they themselves (I assume) playtest the rules and dont have any fresh perspective or uninformed opinions on their work.

Pretty much. They play in a specific way, so it's likely they'd never consider playing something before like Eldar + Dark Eldar with Baron whatshisface to give the 2++ rerollable without a "narrative" behind it (i.e. almost never). So they err in the case of allowing a variety of playstyles or themed armies, but never bother to look into what it allows because they can't imagine doing it. In most cases if you look at how they create armies, they're usually very generic, with little or no tricks to it. So things like CSM armies with mostly CSM squads in Rhinos, maybe one Heldrake, no Plague Marines because it doesn't fit the army, etc. and they never realize that CSM are weak because they're playing against something like footslogging Guardians, Howling Banshees and Striking Scorpions because their opponent likes how the Banshees look and dislikes Wave Serpents (or doesn't have any) so he never fields them. Thus they think that Banshees are great and Wave Serpents aren't anything to write home about.

That's how they think and operate, so it's no wonder that things are broken.

 milkboy wrote:
I see quite a few opinions about GW not play testing enough before releasing a product so that led me to thinking. Is it actually possible to play test this game? So perhaps a little math hammering can help with this conundrum. And so I proceeded to try to do Maths again.

We have now 17 armies being sold on the Gw site. If we include data slates, chapters, FW lists (dark harvest etc) that's probably 25? It's just an approximate number.

So to test a new edition, we need every army to play against every army. That'll be a total of 300. The formula being games = (25 x (24+11))/2

Thats not playtesting, its just random words you've strung together. Theres no need to play every army against every other army, let alone all the little sub-armies in FW.

I think we are in the infancy for these type of things.

You look at the companies starting to record statistics, give it a year or so and GW will start looking at these stats and gleaning from them which armies are currently OP.

I find it frustrating that people completely ignore the balances that GW bring to the game in new releases. It also frustrates me when people say "GW don't do this" but have absolutely no clue how the company actually operates.. and no "but I heard it from this one guy" does not equate to proof and frankly neither does "Jervis said this one line once 4 years ago"

In the OPs example you would need to spend roughly $180000 pounds for purely play testing after release. I'm not sure if you can't find other more productive ways to maintain some balance.

bodazoka wrote:
I think we are in the infancy for these type of things.

You look at the companies starting to record statistics, give it a year or so and GW will start looking at these stats and gleaning from them which armies are currently OP.

I find it frustrating that people completely ignore the balances that GW bring to the game in new releases. It also frustrates me when people say "GW don't do this" but have absolutely no clue how the company actually operates.. and no "but I heard it from this one guy" does not equate to proof and frankly neither does "Jervis said this one line once 4 years ago"

In the OPs example you would need to spend roughly $180000 pounds for purely play testing after release. I'm not sure if you can't find other more productive ways to maintain some balance.

In theory each release would cost you less in terms of playtesting as a lot of the playtesting results would still be there from the last edition. After the initial Testing phase is complete, each phase of playtesting should get easier. So if it had been done properly from day one, then the only changes to each edition will be changes to make the game exciting or include new stuff rather than to tweak it.

The way I see it is there are at least three different aspects of the game that we would like to see fixed by playtesting (or fixed by grots, or Eldrad--Just fix them!):

Rules that are confusing, contradictory, or otherwise illogical:

This doesn't actually need playtesting, but playtesting is one way to discover it. This is an area where GW really, desperately needs to bring in outside help. Not because they are incompetent nincompoops, but because when you are very, very familiar with something, you tend to gloss over it and fill in details yourself. Remember GW forgetting to tell you that you could use the highest leadership in the unit? That's this kind of error. GW needs outside readers to find this type of problem, and ideally, you need some playtesting with that (because it's most important that the rule isn't confusing IN PLAY.

Rules that don't reflect the background.

Again, this doesn't actually require playtesting, if you have experienced players reading over the rules, but you still might discover some stuff that you hadn't though about. For example, whether or not dreadnoughts are effective, they certainly aren't effective in the way that the 40K universe depicts them (as lords of war, striding unharmed through the battlefield). They are sometimes viewed as effective throwaway units, or effective artillery platforms, or effective using a specific CSM option, but not like the background portrays them. This is a really important thing that GW seems to keep missing. Your rules for a unit need to match what that unit is supposed to be doing, if your intellectual property is so valuable. People shouldn't be fielding ChaosTerminators as 3 man deep-striking melta bombs, but they are, because terminators aren't particularly good at what they are 'supposed' to be doing.


Units that are over (or under) costed

Obviously, this needs to be fixed, but, unlike the first two problems, it doesn't need advance playtesting.

I know that GW is totally paranoid about leaks and rumors and advance information, but this would help them so much if they would think about this. The first two types of playtesting/editing need to be done, but they can be done on a small scale with a select group of playtesters, so you could still quash rumors if you really needed to (you could also just do open playtests, but let's not stretch credibility that much). In any case, though, you need to get both of the first two right BEFORE you print the book. Poorly written rules and poorly implemented rules HAVE to get fixed in the books. It's too much trouble and drama to fix them in FAQs and updates. It's unrealistic for people to keep FAQs around to find out how their units actually work, or how the rules actually play.

On the other hand, incorrectly costed units can get fixed during the release cycle. If something turns out to be way more effective than expected, or way more prevalent in tournaments (or gets used not at all), then just adjust the cost. It would be a really short, simple document to download from Black Library if the FAQ and Updates didn't have all these rules clarifications and changes, but just UPDATED POINTS COSTS. Can you imagine how easy that would be? Just find your army, and print out the page with the updated points. Once you've made your army list you don't even need the FAQ!

If GW could be brought around to this style of thinking, they could spend their pre-release testing on the areas that urgently needed it, and then interact with their players and tournaments to see what needs adjusted on points. Think of how much more balanced 40K would be if the obviously great choices had their points increased, and the obviously bad stuff went down in cost. Even if you didn't take bad stuff, because it was 'bad', you'd be paying the premium for taking the good units.

It would also mean that you could adjust the 'meta' on the fly, not release by release. If some new unit becomes the new hotness, and everyone spams it, then increase the points cost! No need to nerf it in the next codex, if it is working as intended. If it's too useful, then make it cost more points! You wouldn't have broken units dominating play through an entire edition (or until the next release of their codex). You could fix these things promptly.

 Swastakowey wrote:
In theory each release would cost you less in terms of playtesting as a lot of the playtesting results would still be there from the last edition. After the initial Testing phase is complete, each phase of playtesting should get easier. So if it had been done properly from day one, then the only changes to each edition will be changes to make the game exciting or include new stuff rather than to tweak it.

I put up exhibit A) 7th edition.

7th edition completely changes the way the game plays and is arguably not that big of a change to the actual rules, I doubt you could get much out of the previous set of data last edition. If you wanted to play test properly you would absolutely have to complete the process again, which as the OP suggests is basically paying around 5-6 guys a wage each year to just play the game.

As I mentioned sites like TOF which collate some stats would be well worth supporting (at least after a few years) from GW as they can basically do all that work for them for much much cheaper. So give it a few years and im sure they will be looking at that sort of site (if they still exist).

bodazoka wrote:

 Swastakowey wrote:
In theory each release would cost you less in terms of playtesting as a lot of the playtesting results would still be there from the last edition. After the initial Testing phase is complete, each phase of playtesting should get easier. So if it had been done properly from day one, then the only changes to each edition will be changes to make the game exciting or include new stuff rather than to tweak it.

I put up exhibit A) 7th edition.

7th edition completely changes the way the game plays and is arguably not that big of a change to the actual rules, I doubt you could get much out of the previous set of data last edition. If you wanted to play test properly you would absolutely have to complete the process again, which as the OP suggests is basically paying around 5-6 guys a wage each year to just play the game.

As I mentioned sites like TOF which collate some stats would be well worth supporting (at least after a few years) from GW as they can basically do all that work for them for much much cheaper. So give it a few years and im sure they will be looking at that sort of site (if they still exist).

The basics have not changed. Many units and guns are not effected. So MOST of the rules do not need to be extensively tested again. Only the changes and then how those changes can effect the current rules. No need to test the movement phase every edition. It wont be hard to test how the psychic phase effects the game, rather then test the whole game again.

It would also help, if each edition was made to fix the game and change things up, rather than just change things up. It will get to the point where the game is near perfect and each edition is a change up (which only the changes need to be studied rather than the whole game). Hence why I said if they did it properly from Day 1 then the issue wouldnt be so big. Because the problems will only get smaller from day 1.

Zodiark wrote:

 Blacksails wrote:

 PrinceRaven wrote:
Which game?

I think Starcraft, based on your reference that was quoted.

Correct. Starcraft is by far nowhere near a balanced game.

Zodiark wrote:

 PrinceRaven wrote:
Well of course Starcraft isn't really balanced any more, not only have they released a new game but after it got to the point where the game was so balanced the matches were decided more on micro and button clicking than tactics and strategy they shook it up and introduced a bit more imbalance into the game.

It was never balanced from its original inception tbh. But adding new things definitely did not help.

Wow you guys have never actually played Starcraft have you. First thing to know is, what you read about Starcraft from people whining that their race is underpowered is nearly always false. People's skills are whats lacking not the power of the race.

Your opponent can be a league or two below you in playskill, if you don't use proper tactics and strategy and build say a bunch of Colossi vs his army of Corrupters and Broodlords, you are going to lose. The game being balanced does not mean that every unit is just as useful for every situation. Being dependant on micro macro and all things encompassed by "button-clicking" is merely a product of the nature of the game. If you are at the same or even a similar skill level as your opponent, the game has always very much been decided by tactics and strategy. If your opponent is well below you in terms of macro, well, it doesn't really matter what you build as you are just going to steamroll him regardless.

That isn't to say there is not balance issues among certain units. I personally think they are minor, because at the end of the day almost every race as a whole is very balanced and functioning very well. Saying SC2 is badly balanced just leads me to believe that you will never be happy with the 40k ruleset no matter what happens, which is disheartening as there is a lot of critical issues here that have broken the game, for some of us it's not just a whine-fest offshoot of our disappointement at GW's ridiculous prices.

Its a good thing we're no longer comparing 40k to Starcraft, eh?

Forgive me I must have been somehow mistakenly led into believing the events on the prior page happened somewhat recently

Comparing Video Games and Table Top games is a waste of time in my opinion.

Maybe play testers don't play full games? Maybe they just use situations and circumstance

 Swastakowey wrote:
Comparing Video Games and Table Top games is a waste of time in my opinion.

Yeah.

Its the same reason you can't compare how cheap 40k is to luxury yachting, or how expensive it is to watching grass grow.