Filter by post category in Media Library

You can use pre_get_posts to filter the query.

So you can

1. retrieve a category from query vars
2. the retrieve the post with that category
3. set the media query to include only post having that posts as parent

To give an UI you can use restrict_manage_posts hook to output a category dropdown.

add_action('pre_get_posts', 'my_filter_media_by_cat');
add_action( 'restrict_manage_posts', 'my_add_media_cat_dropdown' );

function my_filter_media_by_cat( $q ) {
  $scr = get_current_screen();
  $cat = filter_input(INPUT_GET, 'postcat', FILTER_SANITIZE_STRING );   
  if ( ! $q->is_main_query() || ! is_admin() || (int)$cat <= 0 || $scr->base !== 'upload' )
      return;
  // get the posts
  $posts = get_posts( 'nopaging=1&category=' . $cat );
  // get post ids
  $pids = ( ! empty( $posts ) ) ? wp_list_pluck($posts, 'ID') : false;
  if ( ! empty($pids) ) {
    $pidstxt = implode($pids, ',');
    global $wpdb;
    // get the ids of media having retrieved posts as parent
    $mids = $wpdb->get_col("SELECT ID FROM $wpdb->posts WHERE post_parent IN ($pidstxt)");
    if ( ! empty($mids) ) {
      // force media query to retrieve only media having retrieved posts as parent
      $q->set( 'post__in', $mids );
    } else {
      // force media query to return no posts
      $q->set( 'p', -1 ); // let query found nothing
    }
  }
}

function my_add_media_cat_dropdown() {
  $scr = get_current_screen();
  if ( $scr->base !== 'upload' ) return;
  $cat = filter_input(INPUT_GET, 'postcat', FILTER_SANITIZE_STRING );   
  $selected = (int)$cat > 0 ? $cat : '-1';  
  $args = array(
      'show_option_none'   => 'All Post Categories',
      'name'               => 'postcat',
      'selected'           => $selected
  ); 
  wp_dropdown_categories( $args );
}

Negative of this code is that you have to rune 3 queries to filter the media…

Result on UI: