What filter should I use to insert a button inside on Media>Add New

Edited after OP insert edited question

---

Once you are creating a new upload system, I thing that you want to add button/link to different part of WP admin:

• A submenu on “Media” menu
• A submenu on the “New” menu in the Admin Bar
• A button inside the Media > Add New page (like in OP screenshot)
• A button near page title in the “Media Library” page

First 2 are pretty easy, thatnk to, respectively, add_media_page and $wp_admin_bar->add_node functions.

The third, is a bit hard to obtain, because thta button is hardcoded in core, it can be added via javascript, but in core there is an hook, 'post-plupload-upload-ui', that allow to output something after the dragging area, I think it can be good as well for your scope (see screenshot at bottom of this answer).

The fourth is the more hard to obtain, because there is no hook for that, however, just before the output of deafult “Add New” button, in the code there’s this line:

echo esc_html( $title );

where $title is __('Media Library').

The value returned by esc_html can be changed using the esc_html filter, so you can use this filter to output the markup of your custom buttom. However you should pay attention to run the filter only on proper page, only for the wanted string and remove it after first run: otherwise any string passed to esc_html will be affected.

Note that is a tricky way, so in future versions of WP it can’t work anymore, but with current version (3.8) and previous (3.1 and newer) it works.

I created a class, that handle all the 4 tasks. The class is posted here as a standalone plugin, but probably it will be usefull to be integrated in your plugin.
Of course you have to customize the first 3 functions.

---

<?php
/**
 * Plugin Name: Custom Upload UI
 */
class CustomUploadUI {

  static function getLabel() {
    // change here the label of your custom upload button
    return 'Custom Add New Media';
  }

  static function getUrl() {
    // change here the url of your custom upload button
    return add_query_arg( array('page'=>'my-custom-upload'), admin_url('upload.php') );
  }

  function render() {
    // this is the function that render your custom upload system
    if ( ! current_user_can( 'upload_files' ) ) {
      echo '<h2>Sorry, you are not allowed to upload files.</h2>';
      return;
    }
  ?>
    <div class="wrap">
    <h2>Custom Upload System</h2>
    <p>Hi, I'm a custom upload system</p>
    <p class="submit"><input name="submit" onClick="alert('Foo!');return false;" id="submit" class="button button-primary" value="Upload Something" type="submit"></p>
    </div>
  <?php
  }

  function __construct() {
    add_action('load-upload.php', array($this, 'indexButton'));
    add_action('admin_menu', array($this, 'submenu') );
    add_action( 'wp_before_admin_bar_render', array( $this, "adminBar" ) );
    add_action('post-plupload-upload-ui', array($this, 'mediaButton'));
  }

  function submenu() {
    add_media_page( self::getLabel(), self::getLabel(), 'upload_files', 'my-custom-upload', array($this, 'render') ); 
  }

  function adminBar() {
    if ( ! current_user_can( 'upload_files' ) || ! is_admin_bar_showing() ) return;
    global $wp_admin_bar;
    $wp_admin_bar->add_node( array(
      'parent' => 'new-content',
      'id' => 'custom-upload-link',
      'title' => self::getLabel(),
      'href' => self::getUrl()
    ) );
  }


  function mediaButton() {
    if ( current_user_can( 'upload_files' ) ) {
      echo '<div><p align="center">';
      echo '<input id="custom-browse-button" type="button" value="' . self::getLabel() . '" class="button" />';
      echo '</p></div>';
      $this->mediaButtonScript();
    }
  }

  function mediaButtonScript() {
    if ( ! current_user_can( 'upload_files' ) ) return;
  ?>
    <script>
    jQuery(document).on('click', '#custom-browse-button', function(e) {
      e.preventDefault();
      window.location = '<?php echo self::getUrl(); ?>';
    });
    </script>
  <?php
  }

  function indexButton() {
    if ( ! current_user_can( 'upload_files' ) ) return;
    add_filter( 'esc_html', array(__CLASS__, 'h2Button'), 999, 2 );
  }

  static function h2Button( $safe_text, $text ) {
    if ( ! current_user_can( 'upload_files' ) ) return $safe_text;
    if ( $text === __('Media Library') && did_action( 'all_admin_notices' ) ) {
      remove_filter( 'esc_html', array(__CLASS__, 'h2Button'), 999, 2 );
      $format=" <a href="https://wordpress.stackexchange.com/questions/126969/%s" class="add-new-h2">%s</a>";
      $mybutton = sprintf($format, esc_url(self::getUrl()), esc_html(self::getLabel()) );
      $safe_text .= $mybutton;
    }
    return $safe_text;
  }

}

$ui = new CustomUploadUI;

---

Screenshots

In the Media page

---

In the Media > Add New Page

---

New Submenu of Media Menu

---

New Submenu of “+ New” Menu in Admin Bar