Doing this in vanilla WordPress using hooks requires code that hasn’t been released yet, specifically:
That Trac ticket adds new filters which are available in WordPress 6.1.
Note that if your server cannot process webp images for resizing then webp will not work for you even with these filters. You will need to contact your server admins to install the needed packages ( which package is needed depends on how your server is setup and the OS used ).
Is there a reason not to use imagewebp for this?
If you create copies of all your images in webp there will be nothing to tell WordPress that they exist, or to use them. WP also will not return webp URLs when returning the URLs of images, because it’s unaware.
Fixing this would mean making WordPress aware of multiple mime types for an image, which is exactly what the Trac changeset that’s being released in v6.1 is trying to do.
The closest you could get is trying to determine from the server side if the browser can handle webp, but this has caching consequences, as is extremely unreliable.