update_option_$option action not working as expected

Looks like update_option_{$option} will only fire on updated options, ie, not brand-new ones.

Reading the code for update_option(), I see this:

/** This filter is documented in wp-includes/option.php */
if ( apply_filters( "default_option_{$option}", false, $option, false ) === $old_value ) {
    // Default setting for new options is 'yes'.
    if ( null === $autoload ) {
        $autoload = 'yes';
    }

    return add_option( $option, $value, '', $autoload );
}

… before the update_option_{$option} hook. Since your $old_value is unset, it’s false, and so, since false === false, you’re hitting the return add_option() code.

add_option() has its own, related action hook: add_option_{$option}. So you should be able to hook into that, too.

Your updated code:

function my_register_settings() {
     register_setting('my_settings_group', 'my_custom_option');
}
add_action('admin_init', 'my_register_settings');

function do_after_update($old, $new) {
     // Do the stuff here
}
add_action('update_option_my_custom_option','do_after_update', 10, 2);

function do_after_add( $new ) {
    // Do the stuff here
}
add_action( 'add_option_my_custom_option', 'do_after_add' );