Charisma is classically a 'dump stat' for a lot of D&D games and variants unless you're playing one of a few specialized classes.
CHA of 5 probably translates as people actively don't want to talk to you. It could work for a narcissist, though. Sounds more like your character's social skill would be intimidation, not diplomacy.

Intimidation being strength-based in a lot of D&Ds, though.
It might hurt you with the 'summoning' focus, though. Not a Pathfinder expert, but if there's any rules requiring negotiating with demons, I could see Charisma playing a part. You may want to check this before committing.