Which WordPress file controls the featured image upload maximum width and height of 3000px

It’s set at the PHP level. In php.ini there’s a line like

upload_max_filesize = 64M

which you may need to add or change. The file is at the root of your site, and might have to be added if it isn’t there already.
It’s also possible to limit the size via the .htaccess file (which the web server looks at, and again is at the root of your site).

Also, this is a blanket limit, so if you wanted the limits to vary by role you’d probably need a plugin.