set_query_params using custom params defined in functions file?

it just seems to be ignoring only the custom parameters I have defined
in my functions file

There are two issues in your code:

1. WP_REST_Request::set_query_params() actually accepts just one parameter, which is an array of arguments, so your code should look like so:

   $request->set_query_params([ // pass a single array containing one or more arguments
       'per_page' => 100,
       'size'     => 'large',
       'color'    => 'blue',
   ]);
   

2. I could see that you’re using the rest_<post type>_query filter which has a second parameter named $request which is a WP_REST_Request instance, so in your filter function, instead of using $_GET[ $field ], you should use $request[ $field ] (or you could also use the $request->get_param( $field ) method) like so:

   // 1. Make sure your function accepts the 2nd parameter:
   add_filter('rest_post_query', function($args, $request) {
     $fields = array('size', 'color');
   
     foreach ($fields as $field) {
       // 2. Use $request[ $field ] and not $_GET[ $field ].
       if (! empty($request[$field])) {
         $args['meta_query'][] = array(
           'key'   => $field,
           'value' => esc_sql($request[$field]),
         );
       }
     }
   
     return $args;
   }, 10, 2); // 3. Note the 3rd parameter (number of arguments for your function).
   

   And the reason why should the $request be used than relying upon the $_GET or $_POST superglobal, is because WP_REST_Request actually makes an internal API request, so no HTTP connections would be made, and unless the current URL (the page where you run new WP_REST_Request) contains the size or color parameter as in example.com/some-page/?color=white, then in your filter function, $_GET['color'] for example would be undefined (not set).

   On the other hand, $request['color'] or $request->get_param( 'color' ) would always give you the correct value, regardless the current URL above contained color or not in the query string, so long as your query parameters actually contained the color parameter (and other necessary parameters). (see the set_query_params() above)

   So during a REST API request, always use the $request to access the query parameters.

And actually, this part:

$server = rest_get_server();
$data = $server->response_to_data($response, false);

can simply be written as:

$data = $response->get_data();