I think it's also pretty commonly perceived that just because you've moved on and aren't spending as much time with one fandom or another that you 'hate it' or 'deserted it',

There definitely is that assumption. I do think sometimes it's correct, but - well, even then you can look to your real life examples. Sometimes it is malicious, and you need new friends who don't ditch you because they're not into that fandom anymore; other times, you need to give them a break, because while they're in new fandoms it doesn't mean they've deserted you. They just found something fresh.