Save All Post Permalink From A Specific Category into a .txt file

You can use the function in_category() to check if the post is in some specific category.

And better check if the url already exists so you won’t have multiple records with the same url.

function custom_update_txt_file( $post_id, $post ) {
    // Check if post is in category you can check by term_id or slug or array
    if(in_category( 'uncategorized', $post_id )) {
        $post_url = get_permalink( $post_id ) . "\r\n";
        $fp = fopen( ABSPATH . '/post-links.txt', 'a+' );

        $url_exists = false;

        // Read each line to check if the url already exists.
        while (($line = fgets($fp)) !== false) {
            $url_exists = ($line == $post_url) ? true : false;
        }

        if(!$url_exists) {
            fwrite( $fp, $post_url );
        }

        fclose( $fp );
    }
}
add_action( 'publish_post', 'custom_update_txt_file', 10, 2 );

Another thing if you want to run this only one time when the post status is changed to publish and not everytime the post is update with the status publish.

Then better use the transition_post_status hook. You can check with it if the post status is changed and then do something.