BuddyPress is not Photoshop. The cropper is only cropping, ie from too big to admitted default size.
Size is dimension (150x150px), not quality (dpi).
Default avatar are registered in full-size of 150×150 and 50×50 for thumbnails. Both CSS adjusted depending where the avatar is used.
Adjusting a size from to big to smaller is generally better rendered as going from too small to bigger. But once a picture is cropped, his size fits to what is setup for default_size by WP/BP or you.
Picture format: jpg or png ? In brief; jpg remain static, png is more elastic. But nothing to do with BuddyPress img handling displaying (avatar, covers).
To fit img quality – from mobile to retina – you need an original picture of at least 145 dpi, what ever the size and the picture format. Actually for best quality, 300 dpi is a common usage.
But who knows what your users upload for their avatar ? A 3mo jpeg in 72 dpi 2400 x 1850 or a 30ko jpeg in 150 dpi and 800x600px ? Or an exact and optimized (default) 150X150, jpeg/145dpi of 9 ko ?
Frankly the only thing you can do in regard of “quality” pictures for BuddyPress, is to use the default size for avatars, eventually adjusted by a few more pixel because of your theme and available place.
I won’t struggle much with code but inform clearly my user to use a picture of 250×250 max for custom avatar and to respect – so far as possible – the size indicated for a cover img, for which you can recommand a dpi of 150. But without any waranty they obey your recommandations.
To adjust max cropping size, you can add this to your bp-custom.php file:
/* AVATAR MAX CROP SIZING */
define ( 'BP_AVATAR_THUMB_WIDTH', 60 );
define ( 'BP_AVATAR_THUMB_HEIGHT', 60 );
define( 'BP_AVATAR_FULL_WIDTH', 175 );
define( 'BP_AVATAR_FULL_HEIGHT', 175 );
