How to have different background image based on post category

You have to have something to target, so that means adding a class with the necessary information for your styles.

I recommend adding categories (or terms from your custom taxonomy) to the body classes. You can do this with a simple filter. This targets single templates and pulls terms from the category taxonomy. You can easily change either of these to suit your needs.

add_filter( 'body_class', 'my_body_class' );
function my_body_class( $classes ) {
    if ( is_single() ) {
        $categories = get_the_terms( get_the_ID(), 'category' );
        if ( $categories ) {
            $category_slugs = array();
            foreach ( $categories as $category ) {
                $category_slugs[] = 'cat-' . $category->slug;
            }
            $classes = array_merge( $classes, $category_slugs );
        }
    }
    return $classes;
}

This will add new classes to your body tag like cat-uncategorized.