How to add a custom filter in functions.php

To be honest, I had to actually see the source code to get the correct answer for you.

The problem with this particular filter is that the $filter part is dynamic and thus you have to know what particular frequency you want to modify. There’s a numerous of options:

homepage,

blogpage,

$post_type . '_archive',

$post_type . '_single',

$c->taxonomy . '_term',

author_archive.

So once you figure out what particular filter you want to modify (I have chosen a homepage one), you put the following to your functions.php

add_filter( 'wpseo_sitemap_homepage_change_freq', 'my_custom_homepage_freq', 10, 2 );

function my_custom_homepage_freq( $default, $url ) {
    return 'hourly';
}

Changing author_archive is also easy one, as you replace the $filter part in the filter name to author_archive.

In case of $post_type . '_single' (or the other one with $post_type in name), you have to replace the $filter part with the post_type name. Eg for pages only:

add_filter( 'wpseo_sitemap_page_single_change_freq', 'my_custom_page_freq', 10, 2 );

function my_custom_page_freq( $default, $url ) {
     return 'hourly';
}

To reveal you the part of figuring out what can stand for $filter in your original question, I had to check out the ./inc/class-sitemaps.php file (can be seen here in Subversion: http://plugins.svn.wordpress.org/wordpress-seo/trunk/inc/class-sitemaps.php ) where on the line 578 is the function called filter_frequency and it constructs the filter. That function filter_frequency is being called few times from within the file and contains the $filter part as it’s first argument. Eg:

$this->filter_frequency( 'homepage', 'daily', $this->home_url )