How do I create a single.php for a specific category?

I think this does what you want:

function ag_templates( $template="" ){
    global $post;
    $meta = get_post_meta( $post->ID, '_category_permalink', true );
    if( 53 == $meta ){
        $template = locate_template(
            array( "single-category-53.php", $template ),
            false
        );
    }
    return $template;
}
add_filter( 'single_template', 'ag_templates' );