Regarding that huge list of rule sources, I clearly remember when
GW conveyed the message that they wanted to stop putting new rules in
WD, so that people didn't have to carry around a sheaf of supplemental rules.
I don't think the
existence of all these options is a problem. I think the problem is that there is no clear, easy way to indicate which rules options you are using in a game, or a tournament, and there is an unspoken assumption that every rule must be okay every game.
As an illustration, imagine if
GW had a new 'rule' called Warzone (like the location books). The Warzone would specify what type of battle you were playing, by 'flipping switches' for things like Superheavies, Escalation, Flyers, Allies, Apocalypse.
For instance, you might have a Kill Team Warzone, which could cut on a limited force org (and points value) and cut off fliers, Superheavies, allies, etc.
You could have a Cityfight Warzone, which might cut off fliers (since the game is assumed to be in the dense urban terrain of a ruined city. The Cityfight Warzone might also make it clear that there should be a lot of dense area terrain which might make it difficult or impossible to place Superheavies easily, and that objectives might be in buildings or ruins which might make it hard for Monstrous Creatures to claim or contest them.
Instead, when you show up, you might have an opponent with a Knight army, or tons of fliers, or all monstrous creatures. You can't reasonably set up a lot of perfectly legal boards (underhives, boarding actions, Zone Mortalis, dense urban ruins) if your opponent might be totally unable to play on them. The 'unrestricted' existence of those options means that any sort of 'pickup' game (including tournaments)
has to be fought over a relatively bare board, which really favors certain unit types and models. It also means that a player can't reasonably collect a certain type of army (say, a cityfight themed largely infantry force) without being forced to assume that he's gonna get curbstomped for showing up to games without anti-aircraft and anti-armor capabilities.
Now, if you had
well-explained rules for Warzones, you could tell new players:
"Start playing with the Kill Team Warzone. This will give you small games where you can experiment with new units and new rules, and the game will not bog down with complexity that you haven't learned yet. When you feel you have mastered the Kill Team level of play, or you wish to add new and different models to your collections, you might try out these different types of Warzones (if, for example, you really want to add a flyer to your growing army)."
Those options wouldn't look so expensive, because they would clearly be
options, and not just 'stuff that will hose you if you don't buy some too".
Just a clarification: I'm not claiming that Superheavies, or Flyers, or Monstrous Creatures, or whatever, are
bad. I'm just saying that 'everything, all the time", shouldn't be the default assumption of
40K. The
40K universe is just as much about hive battles, and cityfights, and jungle combat, and boarding actions, as it is about war across an open battlefield.