Lets take these functions and paraphrase them into english, then put them in the order they’re ran. I’m going to deliberately avoid using PHP in this answer, but a basic understanding of PHP will be necessary:
When the `post_type_link` filter runs: // the add_action calls
run: `wpa_course_post_link` as it has priority 3 // function wpa_course_post_link(....
if the object is a post
and that post has a course term
search for %course% in the URL string and replace it
Then, run `custom_post_permalink` as it has priority 10: // function custom_post_permalink(...
Ignore the URL we got given and construct a brand new one by returning the post_type + "https://wordpress.stackexchange.com/" + posts name + ".html"
So custom_post_permalink doesn’t add .html to the end of your URLs as you thought it did, it creates a whole new URL. If you changed the priority from 3 to 11, your .html URLs will work, but your course URLs will not as %course% was replaced.
Luckily the wpa_course_post_link function provides a much better template for how to do this. Insted of grabbing the course terms and doing a string replace, just add .html to the end of the $post_link string and return it
So instead if we write it out as pseudocode in plain english, we might get this:
When the `post_type_link` filter runs:
run: `wpa_course_post_link` as it has priority 3
if the object is a post, then:
if that post has a course term then:
search for %course% in the URL string and replace it
then add ".html" to the end
I leave conversion to PHP as a task for the reader, all the needed PHP is already available in the question.