How to prepare (compress/zip) a plugin to enable updating instead of adding new instance?

This is the default behaviour for WordPress. But there is a plugin called “Easy theme and plugin upgrades” that handles this for plugin and template updates: https://de.wordpress.org/plugins/easy-theme-and-plugin-upgrades/

After installing the plugin, you can just upload a new zip and it wont’t install another plugin/theme instance. Instead, the plugin generates a backup of the older version and overwrites the plugin folder.