BuddyPress.org

What you could do is check if there is a thread_id for the new message, if it’s not set it’s a new message and you can remove the recipients so that sending fails.

function venutius_check_recipients( $message_info ) {
	$recipients = $message_info->recipients;

	$u = 0;

	foreach ( $recipients as $key => $recipient ) {

		// if site admin, skip check
		if( $GLOBALS['bp']->loggedin_user->is_site_admin == 1 ) {
			continue;
		}

		// make sure sender is not trying to send to themselves
		
			if ( $recipient->user_id == bp_loggedin_user_id() ) {
				unset( $message_info->recipients[$key] );
				continue;
			}

		/*
		 * Check if the the message has a thread_id, if not it's new and not allowed.
		 *
		 * If we get a match, remove person from recipient list. If there are no
		 * recipients, BP_Messages_Message:send() will bail out of sending.
		 *
		*/
		if ( ! isset( $message_info-thread_id ) ) {
			unset( $message_info->recipients[$key] );
			$u++;
		}
	}

	/*
	 * If there are multiple recipients and if one of the recipients is not a
	 * allowed, remove everyone from the recipient's list.
	 *
	 */
	if ( count( $recipients ) > 1 && $u > 0 ) {
		unset( $message_info->recipients );
	}
}

add_action( 'messages_message_before_save', 'venutius_check_recipients' );

You coud put this in your child themes functions.php or if you want it to apply to any theme, add it to your bp-custom.php in the plugins directory.