Yes, Agreed. I have found posts about this going back 11 months and no one seems to see this as a serious detractor for Buddypress. A group centered plugin that is not actually focused on its members.
Anyone can see ALL public group activity whether they belong to the group or not. Simultaneously that same person cannot see any Private or Hidden Group activity even if they belong to the groups.
Activity should be for groups the person is a member of. Otherwise buddypress is just confusing.
Please let me know if there is a patch since this is really a bug.