Customize Walker_nav_menu to show posts if item is category

You could either extend the Walker_Nav_Menu walker class (i.e., the start_el function is sufficient, as already mentioned).
Or you could hook in when the original output (i.e., the category) has been created. That would be the walker_nav_menu_start_el hook.

For instance like so:

function wpdev_139801_start_el( $item_output, $item ) {
    if ( isset( $item->object ) && $item->object === 'category' ) {
        // Set up your query here,
        // then concatenate the output to $item_output
        // You could do this using wp_get_archives()
    }

    return $item_output;
} // function wpdev_139801_start_el

add_filter( 'walker_nav_menu_start_el', 'wpdev_139801_start_el', 10, 2 );