BuddyPress.org

No, you’re not doing anything wrong – this appears to be intended behavior. Check out bp_activity_get_permalink() in bp-activity.php. The primary_link is used for the View href only when the activity type is new_blog_post, new_blog_comment, new_forum_topic, or new_forum_post.

Would you consider opening an enhancement ticket on trac.buddypress.org for this issue? It seems to me that if a primary_link is provided by a custom component like yours, that it should be used in bp_activity_get_permalink().