Yes, I don't think most people will do this- for one, it's not fun to spend such a large chunk of points just on magic
defense, not offense (although, of course some offense is basically built in for "free" with the Winds of Magic). You're spending enough points, that you could take losses from magic and buy no wizards and still come out ahead instead. Also, with some (all? I need to check) of these items letting spells go off, you're not really even defending!
Personally, I don't see people taking more than one "scroll caddy", and that won't be their only purpose now- they also can serve as a back-up caster in case your big one blows himself up