In the entity-command docs it says “This package is included with WP-CLI itself, no additional installation necessary.”
It is however, not installed.
When I check the wp_cli composer.json, wp-cli/entity-command is listed under require-dev. That could perhaps explain why entity-command (and therefore CommandWithDBObject) is not installed in a non-dev environment.
I suppose including entity-command manually would solve my issue, and I should probably ask the wp_cli people about what is going on as well.