WordPress 3.5+ upload tool filter

The library parameter is actually responsible of what you see in the library frame and not what you can upload. It accepts : image,audio,video,file or any other mime type for example to show only pdf’s :

library: {
    type: 'application/pdf'
},

Now to actually limit the upload to a file type you need to add a parameter to the uploader and catch that parameter using wp_handle_upload_prefilter filter hook.

To add a parameter use:

file_frame.uploader.uploader.param( 'allowed_Type', 'pdf');

and to filter the file type use

add_filter('wp_handle_upload_prefilter', 'Validate_upload_file_type');
function Validate_upload_file_type($file) {
    if (isset($_POST['allowed_Type']) && !empty($_POST['allowed_Type'])){
        //this allows you to set multiple types seperated by a pipe "|"
        $allowed = explode("|", $_POST['allowed_Type']);

        $ext =  substr(strrchr($file['name'],'.'),1);
        //first check if the user uploaded the right type
        if (!in_array($ext, (array)$allowed)){
            $file['error'] = __("Sorry, you cannot upload this file type for this field.");
            return $file;
        }
        //check if the type is allowed at all by WordPress
        foreach (get_allowed_mime_types() as $key => $value) {
            if (strpos($key, $ext) || $key == $ext)
                return $file;
        }
        $file['error'] = __("Sorry, you cannot upload this file type for this field.");
    }
    return $file;
}