You would need to setup some (maybe) pretty complex membeships, but it would be possible to solve with PMP. You can add users into memberships – and these memberships can have access to specific groups only by default.
Not sure if this helps and solves your problem, but only solution I can think of when it comes to restrictions and BuddyPress Features.