Buildings do vary in size a fair bit which is a saving grace, that 1:76 house, could be a decent 'large' town house. will also work nicely if you have raised roads, also because the infantry are both heroic and on bases it can work.
May look a bit odd for sheds and shacks, but for houses etc, well keep in mind the model of one house isn't really one house (e.g. a pistol will fire a bit further than the length of a house) so its a bit abstract anyway.
As an example, the 'linka' buildings - 1:76 scale with some
BF models
They are a bit on the large side, but perfectly usable.
Look at the price, get one or two and try them, make stuff around them fit the scale and it should work nicely