A more appropriate hook may be delete_attachment($post_id). You may also need to hook edit_attachment. Note that the post_id is that of the attachment type post, not the page/blog post to which the media is attached (if any).
In case you want to know when media is added, hook add_attachment($post_id). Note that add_attachment is called before intermediate size files are generated but after the media has been placed in the upload directory.