Theme support title tag – how to replace the default WP separator (with a filter)?

You use the document_title_separator filter.

So in your case:

<?php
function theme_prefix_filter_document_title_separator() {
    return '|';
}
add_filter( 'document_title_separator', 'theme_prefix_filter_document_title_separator' );
?>