Why are you building a seperate user system in the first place? The wordpress builtin system is pretty flexible.
In theory all the login functions like wp_set_auth_cookie(), wp_generate_auth_cookie(), wp_parse_auth_cookie() etc. are all pluggable functions. Which means you can replace them with your own custom functions.
But to be realistic, it will be a lot of work to get this working as you won’t be able to use the default wp_set_auth_cookie() as long as you replace all login functions, or sync your virtual userbase with the real wp_users table.
The default function will not set a valid login cookie unless it finds that $user_id is a real wordpress user.
I think your best options are:
- Rewrite your login system to use real users instead of your CPT and
store your custom stuff in user_meta. - Sync your CPT users with real users (create a real user for each post of your CPT).
- Build your own login system, leave it totally seperate from wordpress users and create the auth_cookies your self.