Need help building a filter to edit the output of “image_send_to_editor”

Your best bet here would be to use jQuery to grab any link that links to an image and tell it to use fanceybox.

jQuery(document).ready(function($){
    $('a[href$="jpg"], a[href$="png"], a[href$="jpeg"]').fancybox();
});

If you want this to work just for your post content areas use this:

$('.post-content a[href$="jpg"], .post-content  a[href$="png"], .post-content a[href$="jpeg"]').fancybox();

You will need to replace .post-content with whatever HTML parent element wraps the content area.