You can forbid uploads of a specific size (or for other reasons) in the wp_handle_upload_prefilter hook that is used in the wp_handle_upload() function.
It get’s the file array passed, it’s a single item in the PHP standard superglobal $_FILES one that is documented in the PHP Manual: POST method uploads.
Just create a function and add it to the filter. Inside your hook, check for the filesize and set $file['error'] to your error message like “Files larger than X bytes are prevented from uploads”.
add_filter('wp_handle_upload_prefilter', function($file) {
$size = $file['size'];
if ($size > 500 * 1024) {
$file['error'] = '"Files larger than X bytes are prevented from uploads.';
}
return $file;
});
This method does not technically permit the upload which means, your server is still receiving the upload data of the file from the users browser. It’s just thrown away afterwards.
To prevent uploads of certain size at the server level, you need to configure your server accordingly for that which depends on what you use.