How to validate the file name of the Media File Uploads?

The filter is sanitize_file_name. You get the $filename as parameter.

Sample code:

add_filter( 'sanitize_file_name', 'wpse_77892_filter_filename' );

function wpse_77892_filter_filename( $filename )
{
    return str_replace( '%', '-', $filename );
}

See my plugin Germanix URL for an extended example.