Best way to modify a plugin with no hooks and no pluggable functions?

It seems like the plugin you are using are quite important to you and/or your client, and the plugin itself is most probably regulary updated with proper support.

I would definitely contact the author and discuss possible changes to the plugin. If a change is a constructive change and would bring some degree of usefulness to the plugin, I see no reason why the author would not implement your changes in the near future releases/updates.

Primarily, if your change is to out of scope of the current plugin, you can suggest and ask the author to just add a proper filter where needed in future releases/updates.

But yes, I would start at the plugin author for such kind of support before looking into other solutions