How to turn this customized core function into a filter/action in functions.php?

Seems like you asked this Q on SO as well, I posted following answer over there:

The meta_form function is not pluggable, and there are no hooks
available, so, as hacking the core is not really recommended, you’ll
need another approach. The following is a jQuery solution. Put the
code in the functions.php file of your theme, or in a plugin:

add_action( 'admin_footer', 'so17239871_mod_metakeyselect' );
function so17239871_mod_metakeyselect()
{
    global $pagenow, $post_type;

    if( ! ( 'post.php' == $pagenow && 'video_photo' == $post_type ) )
        return;
?>
<script type="text/javascript">
jQuery(document).ready(function(){
    jQuery("#metakeyselect > option").hide();
    jQuery("#metakeyselect > option[value^='tqmcf_']").show();
});
</script>
<?php
}