WordPress MU
Originally, WordPress MU was a “fork” of the traditional WordPress system. The goal was to use WordPress to power a network of discrete sites rather than a single one. In the end, this proved to be a powerful, in-demand feature and the MU fork was rolled back into WordPress core to become what we know today as Multi-site with 3.0.
Lyceum
Lyceum was another fork that meant to extend WordPress from a single site to a multi-site system, specifically for enterprise systems. It was abandoned (“shut down”) about the same time as announcements regarding the WordPress MU merge came out. The site is still live and you can still download an old version, but the fork is no longer maintained.
Others
I’ve heard of versions of WordPress that do nothing different except for replace the database structure with something other than MySQL. I’ve run across one that uses Microsoft SQL Server, and another that uses a flat-file system similar to a wiki. Each was developed to fit a specific server architecture and interface with a specific other platform (The SQL Server system had to run on SQL Server. The flat-file system had to integrate with another CMS that could only read flat-files).
But aside from MU and Lyceum, there haven’t been any major forks that I’ve heard of that have lasted longer than a month or two. WordPress has an incredibly ambitious development cycle, and most “forks” have come about because someone wanted to add a specific feature but still stay relatively in-sync with WordPress while the core team added other new features. MU, for example, was always a few months behind WordPress, but that didn’t seem to bother anyone.
Like you mentioned, WordPress is itself a fork from another project. So forking is possible, and can be successful. It just doesn’t happen too often.