Twitter and other high-volume sites probably do this one layer ahead of the servers. Probably with a load balancer that can detect the server load over all servers and if the load is too high (when machines stop replying) they redirect traffic to a server that returns only the “fail whale” page.
This answer on Stack Overflow suggests to look at sys_getloadavg() and display an error when it is too high (and “too high” should be defined by trial and error). I would do this in the main index.php file that “boots” WordPress, to get a quick exit and not put more load on the server.