Well, I forgot I can just turn that activity stream off in settings… so that is what I am doing for now.
In case anyone else comes up against this. I was using the additional privacy plugin that allows you to make a site totally private. It works great. But that might be part of the problem. I don’t think so though because I uninstalled it and set the blog to not be searchable, and it still was showing up in the feeds.
Perhaps it was only showing up to me? Or could other users see it as well?