Admin filter/error if post title is too long

The key thing I think you’re missing is understanding that WordPress doesn’t use a linear path for saving posts. Because posts are autosaved you’ll want to handle this in a way that defines it according to it’s publish status rather than it’s save state. Basically, instead of trying to truncate or adjust the title, do your check in the wp_insert_post_data or via a post status transition and prevent the post from being set to a publish or future status until all of your conditions are met. The action for an admin notice is admin_notices.