Importing a database from a custom built CMS to WordPress

I’d dump part of the database and see how the posts/pages/ titles, date and other meta is arranged and compare it to WordPress’s DB structure.

See http://codex.wordpress.org/Database_Description and http://codex.wordpress.org/Importing_Content for existing importers.

Decide what you really need to import and what needs to be generated that is specific to WP.
It’s not going to be an easy move.

You may be able to work with the text database dump with grep and other text tools to manipulate it into an import file.

You can also export the database in CSV using phpmyadmin, work with the cells/tables, and try http://wordpress.org/extend/plugins/csv-importer/

One other alternative: does the old CMS have an RSS feed? That’s a good way to export and import into WP, if it’s available.