I wouldn’t recommending any core files.
Instead, you can filter the title and make your changes, like so:
add_filter( 'document_title_parts', function( $title ) ) {
// $title is an array of parts
if ( strlen( $title['title'] ) > 10 ) {
$title['title'] = substr( $title['title'], 0, 5 ); // Show only first 5 characters
}
return $title;
}, 10, 1 );