How to use custom post type APIs, but use a different db table

I don’t think you can do this. Maybe you could use a whole new database, but it seems unlikely you could add a new table for managing a custom post type. For one thing, a custom post type is stored in the wp_posts table, but some of the data for it gets stored in the wp_postmeta table (especially any post meta fields). And if you have a taxonomy with it, that uses 3 different tables. And if your plugin needs to store any options those would be in another table.

I wouldn’t worry about polluting the wp tables, it is how they are made to work. Part of its power is the extendability that is built in and easy to use.

You could maybe provide a way to remove your post type and all of its posts if you wanted to be able to completely remove it.