When formatting an oEmbed iframe, how (and where) does WordPress determine the height value?

It doesn’t, that’s not how oEmbed works.

WordPress doesn’t generate or format the HTML for an OEmbed, the OEmbed provider is responsible for that. It’s not even responsible for the creation of the iframe tag. The most WordPress can do is suggest a maximum height and width, but it’s nothing more than a suggestion.

So if you embed a youtube video, WP will go to youtube and ask it for the embed code.

As for where WP gets that suggested max height and width, it gets it from the defaults function/filter:

https://developer.wordpress.org/reference/functions/wp_embed_defaults/

Remember, this is sent to the oembed provider, but is not used to generate the HTML, modify the HTML, validate the response, or for enforcement.

If you use an embed block or shortcode then the height/width used there will override those defaults e.g. will pass 900 and 600 to Youtube when it asks for the embed.

Note that as an OEmbed provider, Youtube can choose to ignore this. WordPress doesn’t interfere with what Youtube returns to enforce this, and can’t as an OEmbed may return a script tag. Some embeds have dynamic heights too that change based on user interaction or the device.

There is the edge case of when you yourself register an OEmbed provider that has a local source. In this case you are the source of the height as the code is entirely local and written by yourself. If this is your situation and you cannot find where the height is set, it may be because your code does not set the height.

Most of the time, the browser will render the iframe, then the iframe has JS that sends a message to the parent frame containing the height of the content, and JS in the embed code receives it and resizes the iframe to match

E.g. this OEmbed response from a toot on mstdn.social:

<iframe src="https://mstdn.social/@alessa_ed/109599742827119029/embed" class="mastodon-embed" style="max-width: 100%; border: 0" width="400" allowfullscreen="allowfullscreen"></iframe>
<script src="https://mstdn.social/embed.js" async="async"></script>