Castles are awesome, castles are my thing. I just spent £500 on plastic castle pieces.
That being said I cannot recommend a single good book to read. There are so many, however castle design was an art and tradition rather than science, until the Renaissance period, so you really don't need anything more than a childs book on the subject. I would recommend a book designed for children with clear graphics, this will not necessarily dumb down your education on castles.
This is the one I remember
https://www.goodreads.com/book/show/847018.Castle
I had Castle, City and Pyramid by the same author, and while the building a pyramid book is based on conjecture all three books were well researched and very clearly illustrated so that you can map the construction clearly in your own mind and get a 3d image of the construction via your imagination. This was frankly what a medieval engineer had to work with rather than paper plans most of the time.
The fortification constructed in Castle is a fictitious building based on common 13th century architecture and is set as a castle build by King Edward I in Wales, which is a good choice of backdrop as the Welsh castles were very advanced for their time and based on earlier French castle building methods.
Now there was one definitive contemporary book on siege warfare though, though it covers a later black powder period. Sebastian
Le Prestre
De Vauban's seminal work on the mechanics of siege warfare called
A Manual of Siegecraft and Fortification is a heavy but good read, based on reason and mathematics and new ideas on ballistics, concentration of force and battlefield probability.