Add a YouTube or Vimeo video as a post attachment?

Attachment represents a file on your server that you may include its url in your content. All attachments are supposed to be in the media library. It is easy to see that by this definition your idea doesn’t fit the wordpress attachment model (images that are off server are not attachments).

Your idea has merit but since it doesn’t fit the wordpress model right now (version 3.7) I am not sure how easy it will be to implement it.