I think the proper filter would be mime_types found here.
function wpse294198_mime_types( $mimes ) {
$mimes['xls|xlsx'] = 'application/vnd.ms-excel';
return $mimes;
}
add_filter( 'mime_types', 'wpse294198_mime_types' );
You can use command line tool file
(linux|macOS) to see the mime type, e.g. file --mime-type -b somefile.xls