How to Organize and Sort Gallery of Images

NextGen gallery isn’t built on top of custom post types, so you won’t be able to easily link your custom Artwork taxonomies with NextGen. Your best bet is to create archive template files in your theme that can display the artwork like NextGen would. The nested approach you mentioned could be accomplished using hierarchical taxonomies.

For example, you might create an archive-album.php file in your theme with PHP like the following:

<?php
// get children of this taxonomy term so we print out folders for them
$this_term = get_queried_object();
$children = get_term_children( $this_term->term_id , 'album' );
foreach ( $children as $child_term ) {
    $child_term = get_term( $child_term , 'album' );

    if ( ! $child_term->count ) continue; // skip terms with no posts in them

    // get one random post in this child term
    $random_term_post = get_posts( array(
        'posts_per_page'    => 1, // only one post
        'orderby'           => 'rand', // selected randomly
        'tax_query'         => array(
            array(
                'taxonomy'  => 'album', // from this child album
                'terms'     => $child_term->term_id
            )
        )
    ) );

    echo '<a href="' . get_term_link( $child_term ) . '">';
    echo get_the_post_thumbnail( $random_term_post[0] ); // show the random post's thumbnail
    echo $child_term->name;
    echo '</a>';
}

// now print out all Artwork in this particular term
if ( have_posts() ) : while ( have_posts() ) : the_post(); ?>
    <!-- print out Artwork here -->
<?php endwhile; else : ?>
    <p><?php _e( 'Sorry, no posts matched your criteria.' ); ?></p>
<?php endif; ?>