BuddyPress.org

Invite Anyone does have email functionality, but one of it’s key features is exactly what you are looking for. As the admin of a group (hidden or otherwise), you will see an invite members button that allows you to comb through and checkmark a list of users, or start typing their username in an autocomplete textfield to send invitations. Users receive a notification when they have been invited.

I use this with Friends turned off (I use BP-Followers instead)