It's more the former than the latter. With the portrayal of women, the crazier reaction to Ginny and Hermione having children ("They're turning into barefooted baby producing machines!") comes to mind. I actually have never encountered people calling each other misogynist because they dislike a female character. Once in a while I can see some people accusing slashers of being misogynist or something to that effect.

But regarding my example, Zoe of Firefly is quite far from being a victim of sexism and yet people still call Whedon misogynist for creating a woman character who is a capable subordinate of a male character. Somehow the fact that Zoe used to serve under Mal in the military and therefore she's calling him 'sir' by that habit completely escapes them.