What is best practice for writing/using custom PHP classes in a theme?

Ultimately it’s about maintainability and readability. Functions.php is good for adding just that – functions. Small snippets of code that modify or add to the theme. When you start getting into classes or building some kind of framework, you need to start separating your concerns whether that be moving them into a plugin or simply including your class files into the theme and keeping a well organized directory.

And there’s no performance to contend here – it’s all about what makes development the easiest and most maintainable for you.