Should I use docker in wordpress production?

My question is, is it a good idea to use docker for a big woocommerce website (more than 100k products)? Or will this incur a noticable performance cost compared to running wordpress straight on the VPS (without containerization)?

It won’t make much difference, nothing you will notice. It’s not even in the top 50 things you should be concerned with for performance.

running wordpress straight on the VPS (without containerization)?

A VPS doesn’t run on bare metal, the V stands for Virtual, so it’ll either be a Virtual Machine, or a container of some sort.

At 100k+ items of content, I would be more concerned with avoiding problematic queries and setting up object cache, perhaps some varnish style equivalent.