How to define the template for custom posts?

You can Use single-{posttype}.php for the single template.

And, if you have register your post type with the has_archive argument set to true, then you can use archive-{posttype}.php for your archive template.

You can check Template Hierarchy

Also for 404 Page, double check that page slug and post-type slug is NOT SAME. I mean check whether you have created any page with slug “books”.