WP_REST_Response() doesn’t seem to return the expected object

I believe WP_Rest_Response did return the proper response based on what your endpoint callback returns, but I think you just not understanding what fetch() returns, so please check the documentation on MDN for more details, but here’s an excerpt that might help you:

The fetch() method takes one mandatory argument, the path to the
resource you want to fetch. It returns a Promise that resolves to
the
Response
to that request, whether it is successful or not.

So referring to your screenshot, you’re actually viewing the above-mentioned Response object which includes the body, headers, URL, etc. And the response body is that body property there (click the body: (...)), but to get the actual JSON data in the response, you can use res.json():

fetch ( `${this.baseUrl}/wp-json/contact/v1/send`, {
    method: 'POST',
    body: formData,
} )
    .then(
        ( res ) => {
            res.json().then(
                ( data ) => console.log( data, res.status ),
                ( jsonError ) => console.log( jsonError )
            );
        },
        ( error ) => console.log( error )
    );

And actually, that status in your $response array is part of the response body, so that $response is actually the response body.

If you want to send a custom HTTP status code, you can use the second parameter for WP_HTTP_Response::__construct() which is inherited by the WP_REST_Response class which extends WP_HTTP_Response.

But, are you sure you want to use the 304 (“Not Modified”) status? Because that would return no response body.. So you should instead use 400 (“Bad Request”) to indicate an error in the request.

function sendContactMail( WP_REST_Request $request ) {
    // dummy param, used for emitting an error, if requested
    $ok = empty( $request['is_error'] );

    // Send the response with custom HTTP status code.
    if ( $ok ) {
        return new WP_REST_Response( array( 'message' => 'Form sent!' ), 200 );
    } else {
        return new WP_REST_Response( array( 'message' => 'Something wrong..' ), 400 );
    }
}

Or you could also just return a WP_Error instance and the status code will default to 400:

function sendContactMail( WP_REST_Request $request ) {
    // dummy param, used for emitting an error, if requested
    $ok = empty( $request['is_error'] );

    // Use the default HTTP status codes.
    if ( $ok ) {
        return new WP_REST_Response( array( 'message' => 'Form sent!' ) );
    } else {
        return new WP_Error( 'foo_code', 'Something wrong..' );
    }
}