The filter you need is nav_menu_css_classes. You should be able to test for *in_category* on single post and archive pages, and add the appropriate class there.
The filter you need is nav_menu_css_classes. You should be able to test for *in_category* on single post and archive pages, and add the appropriate class there.