The add_post_metadata and update_post_metadata filters are meant to allow filtering whether a specific meta data should be saved (added/updated) or not, only. The default value (of $check) is a null, and if you return anything other than exactly null (i.e. !== null), the meta data will not be saved (into the database).
However, the add_metadata() function (used by add_post_meta()) and the update_metadata() function (used by update_post_meta()), both use sanitize_meta() for sanitizing the meta value (although sanitization are done via filters and not the function itself), and the Codex says:
Applies filters that can be hooked to perform specific sanitization
procedures for the particular metadata type and key. Does not sanitize
anything on its own. Custom filters must be hooked in to do the
work. The filter hook tag has the form
“sanitize_{$meta_type}_meta_{$meta_key}“.
So in your case, the $meta_type would be post, and $meta_key would be mycustomkey, and you could filter the value like so:
add_filter( 'sanitize_post_meta_mycustomkey', function( $value ){
return 'new_value';
} );
Check also this example.
Drawback
Unfortunately, the sanitize_meta() doesn’t pass the “object” (e.g. post or user) ID to the sanitization filters (i.e. functions hooked to sanitize_{$meta_type}_meta_{$meta_key}).
There’s a way to overcome that, but it will depend on your code/context, so I can’t explain the “how”.
UPDATE
Actually, there’s a tricky (or hacky?) solution, which works fine with the add_post_metadata and update_post_metadata filters. But I’m not including the code here (yet), and you can find it on Pastebin.