Is there a way to add a new value into a stored custom field value?

That is because you are passing false as the third parameter, that means you are retrieving all of the meta with the same key.

The first time you store the value, the meta doesn’t exist and the value is stored into the DB as a string, the second time you store the value serialized that is the only time the value will be retrieved as an array with the values inside the single array.

The third time you’ll receive an array of array with the first to values stored within the first array and the third value within another index but outside of the first array.

To be more clear.

[
  [
    0 => 'a', 
    1 => 'b',
  ]
  [
    0 => 'c',
  ]
]

If you want to store your data within the same value you have to pass true as the third parameter and also, I suggest to pass true as the fourth parameter for add_post_meta so you make that post meta unique.

In any case just to be sure you retrieve always an array I would suggest to cast and filter the retrieved value.

So the first time the value converted to an array will be an array with one value 0 because get_post_meta will return false.

If you filter the array you’ll we’ll have only the right values.
Be aware btw, that values that can be evaluated as false can be stripped out of the meta.

So if you think you’ll have 0 as the value of your meta don’t use the array_filter.

$new_number = $_POST['number'];
$old_numbers = array_filter((array) get_post_meta( $post_id, 'numbers', true ));

if( empty( $old_numbers ) ){
    add_post_meta( $post_id, 'numbers', $new_number, true );
}else{
    array_push( $old_numbers, $new_number );
    update_post_meta( $post_id, 'numbers', $old_numbers );
}

Instead:

$new_number = $_POST['number'];
$old_numbers = get_post_meta( $post_id, 'numbers', true );

if( ! $old_numbers ) {
    $old_numbers = [];
}

if( empty( $old_numbers ) ) {
    add_post_meta( $post_id, 'numbers', array($new_number), true );
} else {
    $old_numbers = array_merge((array) $old_numbers, (array)$new_number);
    update_post_meta( 1, 'numbers', $old_numbers );
}

Extra, isn’t always a good idea to directly use $_POST values, use filter_input or filter_var to prevent tainted values to be stored in the DB.