There's a difference between anime fans and stupid people. The difference is usually obvious: one likes anime, the other won't shut up.
Anime fan: Talks for a while about some random thing, but when you mention anime can talk to you about it for a while. *notices when you're bored of the topic*. Generally fun to be around because they know what they're talking about.
Stupid person: Starts a conversation about some anime you've never heard about, *doesn't shut up*, follows you around and talks *at* you. Not fun to be around because they can't take a hint (i.e. a hammer to the face).
That, from my observations, is the difference.