How to define a custom Font family for the Site title of a wordpress site?

You have to clarify which custom font, as your question is too broad. Is the custom font available through Google Fonts or which other source? If it is accessible through Google Fonts then use one of the Google Fonts plugins you can find in your WP backend, this will apply the Google Fonts script all over your website.

After that it depends on the theme itself. The best thing would be for you to find out which class your site title uses (go to your main page, check your source code in browser), and then customize your main page. In the customizer you will find a ‘Custom CSS’ part, in which you can then customize your font family, font size etc.