Why is unfiltered html allowed in custom fields for author / contributor roles?

Custom fields are pretty much what it says in the name: custom. Most themes don’t display them in any real way, and they’re there for plugins and themes to do what is needed with them.

Internally, there’s only two real things we’re talking about: Posts and postmeta. The posts are the main part of the post content, obviously, and everybody displays that stuff. The postmeta info, on the other hand, is mostly used for various internal things. It’s not displayed anywhere on the front end of the site, normally. The “Custom Fields” box on the post edit screen allows a (very) rudimentary method to access that data, but most people don’t really use it for anything.

Because postmeta is arbitrary in nature, filtering it by default wouldn’t make any sense. Because it’s not displayed anywhere on the site anyway, filtering it wouldn’t add any security.

If you’re writing a plugin and adding a meta box for it, then you need to do the filtering yourself. You can use the various wp_kses functions to do that easily.