So my question is, should I assume that this is a theme-based
functionality, where some themes provide support for files with the
template-{slug}.phpnaming?
Correct.
Or do all WordPress applications allow for such files? If so, where
can I find instructions on how to usetemplate-{slug}.phpfiles? Or is
there another way these days to create a true “template” where
multiple pages can use its layout?
I don’t really understand what you mean by “all WordPress applications”. In WordPress, templates are provided by the theme. So if you want a custom template that can be used for pages, it needs to be part of the theme. The documentation for how page templates work, including how custom selectable templates work, is here.
If you’re using a child theme, you’d just create the template you want, probably basing it off an existing template, and include it in the child theme. One thing that you’ll notice from the documentation, that I’ve linked above, is that page-{slug}.php has a specific meaning that allows the template to be used automatically. template-{slug}.php does not have any such meaning. It’s merely a convention. What makes a template selectable is the presence of the Template Name comment at the beginning of the file:
<?php
/**
* Template Name: Full Width Page
*/