Check out how feeds are done in BuddyPress itself.
There are two things you need to do:
1) Create an action listener to tell BuddyPress that the URL you’re on is the feed
2) Next, tell BuddyPress what template file to use for the feed and create an activity loop to generate the feed.
For example, to see how BuddyPress generates the sitewide activity feed:
1) BP creates an action listener for the sitewide activity feed here:
http://buddypress.trac.wordpress.org/browser/tags/1.5.3.1/bp-activity/bp-activity-actions.php#L359
2) BP uses the following template file referenced in step 1 to generate the sitewide activity feed:
http://buddypress.trac.wordpress.org/browser/tags/1.5.3.1/bp-activity/feeds/bp-activity-sitewide-feed.php#L1
See how that file structures the RSS feed? It uses an activity loop (bp_has_activities()) with specific arguments to sitewide activity.
—
To create an unreplied RSS feed, follow the steps above as a guide.
Instead of an activity loop in the RSS feed, you’ll probably need to use a forum topics loop with ‘type=unreplied’ as the argument.
Hope that gives you a few pointers.