Yes, you’d use a JSON response containing an error
property.
<?php
echo json_encode( [
'error' => [
'code' => 'my_error_code',
'message' => 'My error message.',
'data' => [any additional data you need],
],
] );
Your AJAX callback should look for the error
and adapt accordingly.
For the regular POST, you’d want to generate HTML markup that produces a WP Notice, or that produces markup that you will style as a notice. See: Complete Guide to Notices