How to customize file path for 404 redirection in php?

This should work for you, using plugins_url():

...
// Assumes that your templates are in a subdirectory called 'templates' in your plugin. 
// Adjust accordingly.
$templates_dir = plugins_url( 'templates', __FILE__ );
$page_404      = $templates_dir . '/404-' . $post_type . '.php' );
...

Update: template_include

If you want to have WordPress loaded up, you might be better off to use the template_include filter.

add_filter( 'template_include', array( 'MyClass', '404_template' );
class MyClass {
    function 404_template( $template ) {
        if ( is_404() ) {
            global $post_type;
            $my_template = plugins_url( 'templates/404-' . $post_type . '.php' , __FILE__ );
            if ( file_exists( $my_template ) ) {
                $template = $my_template;
            }
        }
        return $template;
    }
}