Media/attachment urls don’t redirect with fake URL structure

Milo got me on the right track. If anyone else happens to have this issue the code I used is:

/**
* Redirect broken attachment links to 404
*/

function ww_404_broken_links(){
  global $wp_query;

  if(is_attachment()){
    $attachment_path = get_attached_file( get_the_ID() );

    //If the file doesn't exist on the server show 404
    if(!file_exists($attachment_path)){
        $wp_query->set_404();
        status_header( 404 );
        get_template_part( 404 );
        exit();
    }
  }
}

add_action('template_redirect', 'ww_404_broken_links');

I also added an htaccess file in the old files directory with just:

RewriteEngine off

So both the attachment posts and the direct links to the files 404 if the file is missing.