BuddyPress.org

I would recommend installing WPMU in the root directory, and modifying home.php in your buddypress-home theme to best match your existing site while still incorporating most of the typical, familiar BuddyPress links.

To answer your questions, WPMU gets installed in the root directory, and BuddyPress is installed in the wp-content/mu-plugins/ directory. (Check codex.buddypress.org for install help.)

The BuddyPress homepage is the home.php file I mentioned above. The files in your root directory are just functionary place holders to help tell WPMU/BuddyPress what content to get from the database and then to tell them which theme files to display that data with.