Is there a simple way to just insert a link to an image (without inserting an image)?

I found a solution based on the code of this page:
https://core.trac.wordpress.org/ticket/22180

All attachment files have a post status of ‘inherit’. So first you need to add “inherit” as one of the possible post status to search for. You can use the wp_link_query_args filter to do that.

function my_wp_link_query_args( $query ) {

     if (is_admin()){
          $query['post_status'] = array('publish','inherit');
     }

     return $query;

}

add_filter('wp_link_query_args', 'my_wp_link_query_args'); 

By default the url you would get would be the attachment url, and not the file url. So if you need the file url you can use the filter wp_link_query to filter the results.

function my_modify_link_query_results( $results, $query ) {

  foreach ($results as &$result ) {
    if ('Media' === $result['info']) {
      $result['permalink'] = wp_get_attachment_url($result['ID']);
    }
  }

  return $results;

}

add_filter('wp_link_query', 'my_modify_link_query_results', 10, 2);

The foreach loops through all results, find the ones that have a type of Media, and replaces the attachment URL with the file url.