How to write update query in WordPress to expire transients

Transients are not guaranteed to use database at all. They will use object cache if it is enabled.

And since there is no bulk delete in object cache API, effectively there is no reliable way to clear out transients across all possible environments.

There are some performance reasons to clear them out of database (which core is now doing on upgrades), but your program logic should stay away from it.

As for why specifically it fails in your case it is hard to say. You would need to determine what storage is being used and resulting raw SQL query to test.