…when there are possibly more users hammering on the shared box that we’re hosted on.
There are just too many variables to address when you’re hosting a WPMU install on a shared box. It could be your install, it could be the server’s resources are being hogged by a few other domains on that server, it could be, it could be…
This BP thread may be helpful: https://buddypress.org/forums/topic/buddypress-and-shared-hosting
In case you have not seen this, here’s an article in the BP Codex about improving performance.