...that model loses any invulnerable saves granted by wargear immediately and for the rest of the game...
It's even plural in the actual rule, so yeah in the case you mentioned they'd lose the
SS 3++ and the Termie 5++, but it doesn't destroy the wargear so you still have 2+ armor and still can't get the 2
CCW bonus.
In addition to the Grimoire not being wargear, it doesn't grant an invul, it just gives a penalty or a bonus to your roll.