Giving specific category posts its own permalink structure returns 404

First off, the add_rewrite_rule() syntax is add_rewrite_rule( 'RegEx pattern', 'WordPress query string/vars', 'position/priority - top or bottom' ).

Now I could see you’re trying to have the following permalink structure:

Structure: example.com/foodguide/<post ID>/<post slug>
Example:   example.com/foodguide/1/sample-food-guide

But your rewrite rule (the second one in your code) does not have the proper RegEx subpattern for the post ID part in the permalink; hence, the permalink leads to a 404 error page.

And to fix the issue, you can add \d+/ to the rewrite rule:

add_rewrite_rule(
    'foodguide/\d+/([^/]+)(?:/([0-9]+))?/?$', // <- here, add the \d+/
    'index.php?category_name=foodguide&name=$matches[1]&page=$matches[2]',
    'top'
);

Alternatively, in the RegEx pattern, you can “capture” the post ID (i.e. use (\d+)) and not the post slug, and then in the WordPress query, use the p (i.e. ID) parameter instead of name (i.e. slug):

add_rewrite_rule(
    'foodguide/(\d+)/[^/]+(?:/([0-9]+))?/?$', // capture the \d+
    // $matches[1] is now the post ID, so use &p= instead of &name="index.php?category_name=foodguide&p=$matches[1]&page=$matches[2]",
    'top'
);

PS: Don’t forget to flush the rewrite rules — just visit the permalink settings page.

UPDATE

If you want the post permalink to end with a .html, then:

1. In custom_permalink(), make the following change:

   // Change this:
   $permalink = trailingslashit( home_url("https://wordpress.stackexchange.com/". $cat_name . "https://wordpress.stackexchange.com/" . $post->ID . "https://wordpress.stackexchange.com/" . $post->post_name ."https://wordpress.stackexchange.com/" ) );
   
   // To this:
   $permalink = untrailingslashit( home_url( "https://wordpress.stackexchange.com/". $cat_name . "https://wordpress.stackexchange.com/" . $post->ID . "https://wordpress.stackexchange.com/" . $post->post_name . '.html' ) );
   

2. In custom_rewrite_rules(), change the second add_rewrite_rule() to this:

   add_rewrite_rule(
       '^foodguide/(\d+)/[^/]+\.html$',
       'index.php?category_name=foodguide&p=$matches[1]',
       'top'
   );
   

3. After the function custom_rewrite_rules() { ... }, add this:

   function cancel_canonical_redirect( $wp ) {
       if ( '^foodguide/(\d+)/[^/]+\.html$' === $wp->matched_rule ) {
           remove_action( 'template_redirect', 'redirect_canonical' );
       }
   }
   add_action( 'parse_request', 'cancel_canonical_redirect' );
   

   That would prevent “adding” the ending slash (/) to the permalink URL.

And once again, don’t forget to flush the permalinks.

Also, I’ve not added support for paginated requests (i.e. posts having the <!--nextpage--> tag or the Page Break block in Gutenberg); and if you need that, let me know.

UPDATE 2

For paginated requests with this permalink/URL structure:

example.com/foodguide/<post ID>/<post slug>/page-<page number>.html

1. In custom_rewrite_rules(), change the second add_rewrite_rule() to this:

   add_rewrite_rule(
       '^foodguide/(\d+)/[^/]+(?:/page-(\d+)|)\.html$',
       'index.php?category_name=foodguide&p=$matches[1]&page=$matches[2]',
       'top'
   );
   

2. Change the cancel_canonical_redirect to (* the RegEx pattern needs to match the one used in the above add_rewrite_rule()):

   function cancel_canonical_redirect( $wp ) {
       if ( '^foodguide/(\d+)/[^/]+(?:/page-(\d+)|)\.html$' === $wp->matched_rule ) {
           remove_action( 'template_redirect', 'redirect_canonical' );
       }
   }
   add_action( 'parse_request', 'cancel_canonical_redirect' );
   

3. After the above code, add this:

   function custom_wp_link_pages_link( $link, $i ) {
       if ( in_category( 'foodguide' ) ) {
           $name = get_post_field( 'post_name' );
           $link = str_replace(
               "{$name}.html/$i/",     // replace example.com/foodguide/<post ID>/<post slug>.html/<page number>/
               "$name/page-{$i}.html", // with example.com/foodguide/<post ID>/<post slug>/page-<page number>.html
               $link
           );
       }
       return $link;
   }
   add_filter( 'wp_link_pages_link', 'custom_wp_link_pages_link', 10, 2 );
   

And as always, be sure to flush the permalinks.