BuddyPress.org

bp-custom.php loads up before any BuddyPress components do, and functions.php loads after all plugins and WordPress core code does.

So it isn’t a matter of “if,” it’s a matter of “why.”

Basically…

Use bp-custom.php to override core BuddyPress settings like slugs or constant values, or to load specific code BEFORE BuddyPress loads for whatever reason you might need to.

Use functions.php if you need to add actions or filters to BuddyPress things. It’s good practice to make sure that actions and filters exist before attaching code to them. If you add a filter to a function that doesn’t exist yet, or you add it before/after the code is loaded/executed, then you’re ahead/behind the action/filter curve, so to speak.

Different files for different purposes.