How to remove specific panels in the editor (Gutenberg) when editing a block (via right panel)?

Removing specific panels or features for a block is done on a per block basis using various hooks (https://github.com/WordPress/gutenberg/issues/15450#issuecomment-635255936)

Update (August 2021)

Removing specific panels (that would allow for customization of a block) can now be done through the theme.json aka global settings and styles which allows you to customize settings (colors, typography, etc) for an entire website or on a block-level.

Requires WordPress 5.8 or newer.

Note that only certain core blocks have these supports built in.

For specifically disabling the custom colors in the button block,

You’ll need to first create a theme.json file in base folder of your theme; need to add the required info

{
    "version": 1,
    "settings": { 

and then add the following:

  "core/button": {
      "color": {
          "custom": false
      }
    },

so your full theme.json would look like:

    {
        "version": 1,
        "settings": { 

    "blocks": {
      "core/button": {
          "color": {
              "custom": false
          }
        },
    }, 
}
}