The quickest way to answer this question is to follow the code. See get_post_custom
in Codex. Scroll down to Source Code section:
get_post_custom() is located in wp-includes/post.php
There we’ll see that get_post_custom
calls get_post_meta
, which then calls get_metadata
, which points us to:
get_metadata() is located in wp-includes/meta.php
there we’ll finally see the available filter:
$check = apply_filters( "get_{$meta_type}_metadata", null, $object_id, $meta_key, $single );
in this case, post
is the $meta_type
, so our filter is get_post_metadata