but it looks like that’s not possible?
Correct, it would not make sense to be able to do so as patterns are just predefined snippets of block HTML. They aren’t entities/blocks/objects themselves.
Block templates are placeholder blocks, a suggestion for the user that can either be locked in place so they can’t be changed, or just the starting child blocks.
A pattern is like copy pasting some content you saw on a site of examples.
A block template is the what you get before the user has inserted data for a post types content, or a blocks inner blocks. Think of it like that initial paragraph block that tells you to start writing here.
Variants are when your block has an attribute and you display the block multiple times but each registered variant has a different value for the attribute that can have a different name/icon, e.g. all of the core embeds are a single block that changes based on what it’s embedding.
Variations in newer versions of the block editor can be shown as a selection for the user when the block is first added. This is how the query block asks the user how they want to display content. These variants use the innerBlocks: [ field to change the content of the blocks once the user has selected a variant. If your goal is to allow the user to select a predefined set of blocks for inner blocks, then this is the closest you will find