How and why can a hook call itself without causing recursion?

This:

apply_filters( 'image_downsize', false, $id, $size )

doesn’t call the image_downsize function, it applies any filters hooked to the image_downsize tag, which would be a different function.

The only way that would cause recursion is if the filter you hook in turn called that function:

add_filter( 'image_downsize', 'wpd_downsize', 20, 3 );
function wpd_downsize( $return, $id, $size ){
    // collapse the universe
    return image_downsize( $id, $size );
}