Thanks for all the responses! The improving performance guide cut down the load times a bit, but ultimately it ended up being a memory problem. I upgraded from 128 MB to 192 MB and rebooted the VPS, now my initial wait times are down to around 1 second.
