Virtually all the core files and scripts that WordPress uses for displaying the media library are within the wp-admin folder, plus it uses various admin-only hooks, so using this on the front-end is pretty much out of the question.
It’s not difficult to allow users to upload to the media library from the front end (either writing your own system or using a plugin like Gravity Forms which has this functionality built-in), but providing all the media management functionality is going to be a very big task indeed.