You will need to modify your server configuration. Specifically PHP and a general rule.
This can be done with both php.ini and .htaccess.
An example php.ini file would be
upload_max_filesize = 4M
post_max_size = 4M
memory_limit = 4M
and for .htaccess
php_value upload_max_filesize 4M
php_value post_max_size 4M
php_value memory_limit 4M
Both files should be placed typically be in public_html of your web server.
Be sure to name the files as I have told you above, otherwise they will be ignored.
Lastly if you have a multi site install of WordPress you might be able to go into network and set the value there. This will only work if the 4mb is less than what your default or current server config is.
If all of that isn’t working you should next speak with your web host about how to do this.
Thanks again @studiocrafted! I’ll give this a try. Will the restrict users on the frontend only? Or even for admins?
It’s a rule for your web server, so yes. If you set it, it applies to all uploads.