You’re looking at the problem the wrong way. The error you’re seeing isn’t an error coming from WordPress, it’s a PHP error.
Somehow, somewhere, something is limiting the memory limit to 96M, and it ain’t WordPress that’s doing it.
Here’s the thing: WordPress can’t actually limit the memory on most servers. I know that it attempts to increase the limit using ini_set and such, but that’s more of a just-in-case sort of thing. Sometimes it can set the limit higher. Most of the time, PHP simply ignores that line.
You’re also misunderstanding how memory works. This is made clear by the line “WordPress throws the error also on images as small as 300-400K and under”. The size of the compressed image has little or nothing to do with the size the uncompressed image takes in memory. That is all about the resolution. Megapixels matter. One pixel takes 4 bytes. 8 million pixels (8 megapixels) takes 32 megabytes of memory to represent, and the size of the resulting compressed JPG is irrelevant. WordPress needs to hold a couple copies of the image in memory too, the original one that it’s working from, and another that it is creating the modified versions of. So yeah, lots and lots of memory is needed.
But again, WordPress can’t set the memory limit in most cases. So something is limiting it to 96M. Which will severely limit the maximum size of the images WP can work with. Until you find where that 96M limitation is coming from, looking at the WordPress core code and such is kinda a waste of your time here.