It's really a personal decision on how "opsec" you are, I think. I'm the same way as you are. I convinced a friend of mine that gardening is the bee's knees, and now she has a bigger (and more productive) garden than I do. I told my mom about it, and now her trees are all fruit producers.
Nobody has to believe that you're afraid of the coming global superstorm, or aliens, or the government, or whatever the tin-foil topic of the day is. If you don't act like a crazy person, they'll be like "oh, Kate? She's really outdoorsy. She's always bringing us the most delicious veggies from her garden, and she's out hiking all the time. It sounds like she has so much fun."
I'm picky about who I allow to come to my house, but if I had a homestead, I'd probably be inviting everybody at TSP over on a regular basis, and all of the moderators here would have a key to the house and knowledge of exactly how to get to the food and ammo, should they need it. Most everybody here knows my full name, and my mailing address, and my husband's name. So even that level of opsec is a personal choice. It's all in what you feel comfortable with, and how well you feel you could defend yourself, should you need to do so.
All of this is my opinion, of course.