You get the rough texture when you're spraying from too far away and/or it's too warm. The primer dries mid-air and leaves a rough surface.
I would probably not spray during summer heat in the sun for said reason. Never had issues in "normal" room temperature and humidity. But I live in Finland so it's never too warm basically.
Also when spraying put your minis in one row. Do not place minis behind your "target" mini. The dried up spray will hit them and create that same surface too.