Help with rewrite rules for two post types

The first rule is capturing all requests. If you add a $ anchor at the end of the rules, they’ll behave as expected (you could also just switch the order of the rules so the more specific one “wins”).

add_rewrite_rule(
    '^app/course/([^/]+)/?$',
    'index.php?page_id=3400&course=matches[1]', 
    'top'
);
add_rewrite_rule(
    '^app/course/([^/]+)/theme/([^/]+)/?$',
    'index.php?page_id=3400&course=$matches[1]&theme=$matches[2]',
    'top'
);

Check out Monkeyman Rewrite Analyzer for debugging rules.

Also note that theme is used by WordPress internally as a POST or GET var, I suggest changing that to something unique to prevent any potential conflict.