Are you able to update standard WP repo plugins? I am using that library and it seems to work well.
What’s the exact error that WordPress gives on the admin screen when it tries to run the update?
I would take a look at permissions first. What are the permissions on the old plugin directory? Any different than any of your other plugin directories?
I would also take a look at the format of the zip file you’re feeding it. What software did you use to create the zip? When you unzip it, what does the directory structure look like? It should unzip as a folder with the same name as the plugin you’re trying to replace. Did you set the slug attribute in the JSON file? Does that also match the folder name?
Hope this helps. I was able to get that library going pretty quickly.