Is it possible to install WordPress within WordPress installation?

Yes, you can do this. There won’t be a 404 error because of the default rewrite rules that WordPress has (if you use ‘nice URLs’, if don’t, there is no default .htaccess at all).

Because of the exceptions (see below) if there is a ‘real’ directory or file in the directory of your WordPress installation, WP won’t treat it as a URL for a page or post.
So this way you can install one or more new WP instances into a ‘parent’ WP directory.

You can check the rewrite rules in the .htaccess file.

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

These are the exceptions:

If string is not a valid file:

!-f

If string is not a valid directory:

!-d