You can use this ThemeUpdateChecker class
Basic outline of the process:
- Have a fixed spot on your server to both house a little bit of data about the current version and a zip of your theme files.
- Hook an action into
pre_set_site_transient_update_themes
- In that action, ping your server (with something like curl)
- Compare the versions of both the local and remote themes.
Two more references: