What’s the algorithm to verify user password?

First: Be aware that this is pluggable in WP so first of all check that you’re site is using the default implementation.

Then have a look at wp_hash_password and wp_check_password in pluggable.php and the PasswordHash class. There you should find what you are looking for.