WordPress isn’t natively friendly to such granular size management. Also it could take a considerable toll on humans to manage it, if manual.
The common approach is to switch image sizes generation from the moment of upload to the moment of size being requested. So the sizes which are never actually used are never generated accordingly.
If I am not mistaken WPThumb is one of popular implementations.