Custom WordPress Plugin will install new and not update

A plugin is treated as the same as another plugin if it has the same directory name. That’s it. If your zip file contains the plugin files then the zip filename will be used as the directory name. If your zip has a directory in it, containing the plugin files, then that will be used as the plugin directory.

So the solution is to structure your zips like this:

my-plugin-1.0.0.zip
    ∟ my-plugin/
        ∟ my-plugin.php

my-plugin-1.1.0.zip
    ∟ my-plugin/
        ∟ my-plugin.php

This way the directory will always be my-plugin/ and those two zips will be treated as the same plugin.