I think the only time it's an issue is when someone else is set on ruining your fun (or if you're set on spoiling theirs). I think it's also pretty commonly perceived that just because you've moved on and aren't spending as much time with one fandom or another that you 'hate it' or 'deserted it', but perhaps you're disappointed, you've said everything you have to say, or you're tired of the cold shoulder.

Sometimes we grow apart, and that's OK. You might see that 'high school friend' once and a while and say hi and catch up, but that's enough for another 6 months or so (and some people it's enough for forever! Take that from someone who went to their 10 year reunion.) Changing: it's all part of growing up.