@bojanski,
Here is the code you are looking for, you’ll find it under /wp-content/themes/yourtheme/activity/index.php around line 22. This is presuming you have created a child theme based on the default BP theme.
<li class="selected" id="activity-all"><a href="<?php echo bp_loggedin_user_domain() . BP_ACTIVITY_SLUG . '/' ?>" title="<?php _e( 'The public activity for everyone on this site.', 'buddypress' ) ?>"><?php printf( __( 'All Members (%s)', 'buddypress' ), bp_get_total_site_member_count() ) ?></a></li>
Remove that code and you should be good to go.