I can’t comment yet so I’ll comment here.
Start here:
https://rudrastyh.com/wordpress/self-hosted-plugin-update.html
That’ll give you the basics. Hope you realize the can of worms you’re getting into… payment gateway api’s, registration codes, managing updates, etc. Self-hosting a plugin on your own is not for the feint of heart.
I know a number of years ago there was a company that took care of a lot of that for you, including registration & payments. Can’t seem to find them now though.
edit found them. https://freemius.com/