please read the documentation !
BuddyPress Codex
The question of ‘how to create a custom theme’ is a fairly open ended one. My advice would be start with creating child themes of the parent theme. As Chouf1 already said the codex is a great place to start. You may also want to look at how others have done things and learn by example for instance here: https://buddypress.org/extend/themes/. If I was going to advise someone on how to learn I’d say you have to first off do a child, add a widget area.. learn about what CSS does what maybe using something like firebug to do inline changes and learn. If you’re looking to do a total fresh take non default theme then you want to follow the more methodical process of creating image > XHTML / CSS > theme when learning. By having your end goal from the start you can focus the things you will need to learn and apply.