bp_core_sort_subnav_items() is added as an action to wp_head so you could create your own version in your theme’s functions.php file. You would then have to remove the existing action something like this:
remove_action('wp_head', 'bp_core_sort_subnav_items', 1);
add_action('wp_head', 'my_custom_sort_subnav_items', 1);
In my_custom_sort_subnav_items() you could do whatever you wanted. Right now I think everything is sorted by the key name (the call to ksort).
I guess the question is, how do you want them ordered? You might have to hardcode the order if it’s not something easily sorted. Or figure out how to modify the key names.