Can I have a php site on subdirectory of WordPress?

Technically, it’s possible. And it works.

You could have a PHP site on a sub-directory like example.com/shop.

But, a better practice is to host that PHP site on a sub-domain like shop.example.com. It helps in better maintenance and provides a better structure for organizing the site.

On a Nginx server, your config should be something like:

server {

    listen 80;
    listen [::]:80;

    server_name shop.example.com;

    root /var/www/html/shop;
    index index.php index.html index.htm;

    location ~ \.php$ {
                include fastcgi.conf;
                include fastcgi_params;
                fastcgi_pass unix:/run/php/php7.1-fpm.sock;

        }
    }

If you still insist on having your shop at example.com/shop, no extra configuration is necessary. Your pre-installed WordPress setup is all. It would work seamlessly. But, keep in mind that shop files must be kept inside a folder called shop in your home directory.