How can I link a CSS file only on single posts?

Question

The Codex page for is_single should be updated, what you’re seeing is correct behavior. According to the main Conditional Tags Codex page:

is_single()

When any single Post (or attachment, or custom Post Type) page is being displayed. (False for Pages)

You can exclude attachments by also checking ! is_attachment().

Also, you may want to move that code into an enqueue style instead.

EDIT-

if ( is_single() && ! is_attachment() ) :
    echo 'is a single post and not an attachment';
endif;