How to moderate posts

This is a partial answer, because as I mentioned in my comment I’m not sure why your code is requiring two ‘really bad words’ at the moment. It’s also an untested attempt.

I’ll deal with your first question: how to send a user back to the posts page and display a suitable error message, rather than an error about editing posts in the trash.

A way to do this that comes to mind is using wp_redirect() with a custom querystring parameter, that you can then detect and display your error message with the admin_notices action.

However we can’t redirect right away, because we need the wp_insert_post_data hook to complete its work first. Looking in the source, we could potentially redirect at the end of the wp_insert_post() function, in the save_post or wp_insert_post actions. We’ll also need a way to check at this stage if a redirect is needed – which we can probably safely do by checking if the post_status is 'trash' and if it’s a new post (because when else would a new post be going to the trash?).

Here’s a potential workflow:

// if this is a new post but it's in the trash, redirect with a custom error

add_action( 'wp_insert_post', 'wpse_215752_redirect_from_trash', 10, 3 );

function wpse_215752_redirect_from_trash( $post_ID, $post, $update ) {
  if( !$update && 'trash' === $post->post_status ) {
    wp_redirect( admin_url( 'edit.php?custom_error=badwords' ) );
    die();
  }        
}

// if our custom error is set in the querystring, make sure we show it

if( isset( $_GET['custom_error'] ) && 'badwords' === $_GET['custom_error']) {
  add_action( 'admin_notices', 'wpse_215752_badwords_error' );
}

function wpse_215752_badwords_error() {
    $class="notice notice-error";
    $message = __( 'Sorry, that post cannot be made.', 'your-text-domain' );
    printf( '<div class="%1$s"><p>%2$s</p></div>', $class, $message ); 
}

I haven’t tested this, mind you, but I trust this gives you something to get started with!

Other options to look at might include the wp_transition_post_status actions, particularly new_to_trash, which – by looking at the source – should also be called in your situation.

Finally, if I was a user, I might want the opportunity to edit my post rather than have it automatically trashed, but this is a user experience decision on your part.

Disclaimer: This might not necessarily be the best way to do this, but I hope it gives you some direction nonetheless, and perhaps others can chime in if there’s a better way.