How to migrate a website based on a custom database to wordpress?

I don’t see any reason for this not to work. I have done similar transfers few times, and if you transfer all data correctly, and if posts in the wp_posts are well formed, with all data present (status, type…), it will work.

The tricky part might be the data that can’t fit into wp_posts, and it should go into wp_postmeta, you need to take care of proper post ids. Also, make sure to use correct format for date field, make sure to have both post_date and post_date_gmt filled, guid (some unique string for each post, WP uses URL by default), and make sure author ID’s are correct.