You can’t use that function for this contextual menu.
Use the action hook coming after bp_user_displayed_nav function: bp_member_option_nav
to simply add html and your link.
Try this (add to bp-custom.php or child theme functions.php)
function lh_custom_setup_nav() {
if ( bp_is_active( 'xprofile' ) )
?>
<li><a href="<?php bp_members_directory_permalink(); ?>"><?php printf( __( 'All Members', 'buddypress' ) ); ?></a></li>
<?php
}
add_action( 'bp_member_options_nav', 'lh_custom_setup_nav' );
Apply CSS to li and a tags accordingly to your theme if necessary.
Reference file: bp-templates/bp-legacy/buddypress/members/single/home.php