Most likely the server is not handling byte range requests to support seek.
Test if you get a 206 response from e.g:
curl https://example.com/test.ogg -i -H "Range: bytes=0-1023"
Try (temporary) setting the preload="auto"
to your shortcodes to fully download the audio on page loads.
See more here:
How to test:
Server ogg media config info:
Bunny:
https://bunny.net/academy/http/what-are-http-range-and-byte-range-requests/
Audio shortcode:
https://wordpress.org/documentation/article/audio-shortcode/
and varius discussions here
https://stackoverflow.com/questions/9563887/setting-html5-audio-position#9565178