How does WordPress search work behind the scenes?

EDIT – The current version of WordPress supports relevance, so this answer is no longer accurate.

There’s no concept of rank or relevance, it’s just a simple LIKE query on the post title and content:

($wpdb->posts.post_title LIKE '{$n}{$term}{$n}') OR ($wpdb->posts.post_content LIKE '{$n}{$term}{$n}')

You can use the posts_search filter to modify or completely replace the search query with your own. From /wp-includes/query.php:

// Allow plugins to contextually add/remove/modify the search section of the database query
$search = apply_filters_ref_array('posts_search', array( $search, &$this ) );

Also see the available query filters on the WP_Query Codex page.