I had that exact same problem with Lost! It was to the point where I started to really dislike the show, and I never even watched it.

Plus, the people who gabbed about it made it sound really juvenile because all they talked about was how hot Locke and Charlie and other people were. It wasn't until I made a new friend who also watches the show that I realized it's actually quite deep and intense.