Am I wrong in not accepting every invitation to go out?

I don't think that you are doing anything wrong, but that they are doing everything right by continuing to invite you. I mean, you may feel a little strange if they weren't invinting you...possibly hurt. Maybe just try it once, you are never too old to enjoy yourself, maybe if it's close to where you live you could show up for a few minutes...