show custom category posts from a custom post type

You should need to do any custom querying on taxonomy-subject.php, just use the normal loop api. WordPress will build a query for you.

As for your question:

$args = array(
  'post_type'      => 'documents',
  'posts_per_page' => 20,
);

WP_Query doesn’t take a second argument to its constructor. All its seeing is the arguments above. You’re asking WP_Query for 20 posts from the document post type. Its giving them to you. If you want posts from a specific term, you’ll need to tell it that with a taxonomy parameter.

$loop = new WP_Query(array(
    'post_type' => 'documents',
    'posts_per_page' => 20,
    'tax_query' => array(array(
        'taxonomy' => 'subject',
        'terms' => get_queried_object_id(),
    )),
));

get_queried_object_id should return the term ID on taxonomy-subject.php, you’ll need to use some other code if this loop is being used elsewhere.