Not in Bicester but the Oxford Wargames Society meet in Wolvercote village hall.
http://www.oxfordwargamessociety.org.uk/
If you don't drive just get the S5 and get off at 5 mile drive. Walk down 5 mile drive, round Wolvercot roundabout then down the road to Wolvercote. It is on the corner of the last turning on the left before the bridge (I think). Never been myself, but I know a few people who are members and by all accounts its a good club.
Alternatively I think there is one in Brackley, but I could be wrong.
I don't think there is a club in Bicester.