WordPress is not designed to have a replace-able database layer. Computer Science has some different concepts in it’s chest to deal with different databases in the same application on different level of abstraction, WordPress does not tend to make use of any of such. To say it in more simple words: Not at all.
But wordpress offers the ability to replace it’s own code with some other code, for example the whole database class. That is not always (read: normally) enough to replace the whole database layer but helps tinkering. The WPDB class replacing HyperDB for example does not add a new database layer in full, but handles multiple MySQL servers instead of one, which is a well known example (this is considered bad practice to do with PHP, there are better ways, but it’s a good example what can be done).
Next to replacing the database class, WordPress has the ability that even more code can be replaced and you can completely change it, e.g. creating an own distribution that takes care of everything regarding the database layer on it’s own.
If you want to see a real life example on how to replace the whole database layer, so to make wordpress use a totally different type of database server (here: MSSQL instead of the hardcoded MySQL), you find a full distribution and a patch here:
WordPress on SQL Server Distro & Patch
It’s free software you can study how such a database layer can be implemented, the website provides more information about that topic as well.
Tip: Don’t fight WordPress. Stick to MySQL and that’s it. Otherwise look for something else better designed to fit your needs. Just take the right tool for the job.