Custom form validation

Yes, you’d use a JSON response containing an error property.

<?php
echo json_encode( [
    'error' => [
        'code'    => 'my_error_code',
        'message' => 'My error message.',
        'data'    => [any additional data you need],
    ],
] );

Your AJAX callback should look for the error and adapt accordingly.

For the regular POST, you’d want to generate HTML markup that produces a WP Notice, or that produces markup that you will style as a notice. See: Complete Guide to Notices