Allow users to only CREATE one single (custom) post

You can use WP_Query to get the posts created by user in a specific category as well as a post type.

To find posts of a specific category you can use the following snippet:

$query = new WP_Query( 'category_name=staff' );

Where as to find posts of an author you can use the following snippet:

$query = new WP_Query( 'author=123' );

Combine this in a single query and if the any posts are returned that means user has already created a post.

For reference, check this link