The truth is that, as we get older we have to be more concerned with how we are viewed by others. This is a social world and we are social creatures. The older you get the more important social appearance means. Which seems counter intuitive since as we get older we actually start to care less. And that’s good, because you really shouldn't care what people think but you should be aware of it. As an adult, you'll wind up with a career, applying for loans, making investments and ect ect. You reach a point where you know that people are judging you based on your appearance and as such you'll find you have to act differently to make things easier.
I come from a goth/punk background. In high school, standing out and looking different is all part of the childhood experience. But, as an adult, I don't feel I can walk around nearly as hard core as I used to because, as an adult, I require more respect to accomplish things. This certainly doesn't stop me from dressing that way anyway, but it does put me into a position to be considerate of where I am. That is why I think I've gone in the psychobilly/rockabilly route, simply because its slightly more conservative without me having to adopt khaki slacks and polo’s (shudder).
The same thing applies to our hobby. I don't walk around yawping off the roof tops, "I play with little mens!" Our hobby is not mainstream (thank God) and so doesn't have the social acceptance that a lot of nerd culture does now. I'm personally a little P-offed that being a nerd is cool now, when I was in high school it was not. Now it’s all, " I <3 NERD!" Sure, society loves the socially sanitized nerd. The "Big Bang Theory (I do like this show anyway

) Nerd. Comics, video games and even D&D have become more "acceptable" socially now. But, building plastic models and playing with them? It’s no more understood that collecting action figures. But, if you are really worried about it, then all we have to do is get the hipsters to like Warhammer and next thing you know you'll be able to buy shirts at Hot Topic with little cartoon space marines on them wearing big glasses and a cute little quote that tells the world how much you love nerds.
To conclude this rant, you do what you want, who gives a fluff what others think. But, be aware of how people might perceive you and ask yourself if it matters. I can be more open at work about my nerd culture because I'm a software engineer and most of us are nerds anyway. But, if you work in say, a hospital, or some
BS were impressions might matter then I wouldn't go there. If you're at a club or whatever trying to pick a chick and you don't get the vibe she's going to be into your little mens then you might not want to bring it up. But, if you're at Renfair, and you see a guy walking around with the Aquila on his shirt and you scream, "Death to the false emperor" (which I have

) then go nuts and nerd out likes its going out of fashion. Just be conscientious, make sure that you're not hogging the conversation to yourself. That, I feel, is the real issue when we nerd out. It’s not the subject matter usually, it’s the lack of social edicate. If you start a ramble and don’t stop to catch your breath because this random stranger needs to know about your massive comic collection, then you might be pushing it a bit.
Man that was longer that I thought it would be. Sorry,