How to make an meta_query optional?

It’s not so difficult. All you have to do is to add a condition and check if the book_id param is passed. You can even check if this param is correct (for example if the book_id should be a number, then you can check if it is):

add_filter( 'rest_gallery_query', function( $args ) {
    if ( trim($_GET['book_id']) ) {  // check if book_id is passed and is not empty, but you can modify this condition
        $args['meta_query'] = array(
            array(
                'key'   => 'book_id',
                'value' => trim( $_GET['book_id'] ),  // <- you don't need to esc_sql this
            )
        );
    }

    return $args;
} );