BuddyPress.org

You can’t use that function for this contextual menu.

Use the action hook coming after bp_user_displayed_nav function: bp_member_option_nav to simply add html and your link.

Try this (add to bp-custom.php or child theme functions.php)

function lh_custom_setup_nav() {
  if ( bp_is_active( 'xprofile' ) )
?>
<li><a href="<?php bp_members_directory_permalink(); ?>"><?php printf( __( 'All Members', 'buddypress' ) ); ?></a></li>
<?php  
}
add_action( 'bp_member_options_nav', 'lh_custom_setup_nav' );

Apply CSS to li and a tags accordingly to your theme if necessary.

Reference file: bp-templates/bp-legacy/buddypress/members/single/home.php