How to allow to user non logged in WP system upload in media library?

First, you do not want to edit core WP files, because your changes will disappear when there’s an update. That’s why you are encouraged to create child themes.

Second, you may want to peruse this Codex page on AJAX. Note that it shows how to handle both kinds of users (logged in and not logged in):

add_action( 'wp_ajax_my_action', 'my_action' );
add_action( 'wp_ajax_nopriv_my_action', 'my_action' );

The my_action function would go in your theme files (when I was working on mine, I created my own plugin to handle AJAX).

Hope this starts you down the right path.