wp_delete_attachment

Try the following:

1. Replace your current upload filter with the one below
2. Create a new custom post and upload some files
3. Check the files have been moved to the correct uploads folder
4. Delete the custom post and then check the files have been deleted

New uploads filter function:

function set_upload_dir( $args ) {
    if ( ! empty( $_REQUEST['post_id'] ) && $post_id = absint( $_REQUEST['post_id'] ) ) {
        if ( $post = get_post( $post_id ) ) {               
            if ( $post->post_type !== 'attachment' ) {
                $args['subdir'] = "/$post->post_type"; // Must be preceded with slash
                $args['path']   = $args['basedir'] . $args['subdir'];
                $args['url']    = $args['baseurl'] . $args['subdir'];
            }
        }
    }

    return $args;
}

The main differences between this and yours are:

1. We aggressively check the post context (post_id) is valid and that it’s not an attachment in itself. Remember, this filter is called all over the place, not just when an image is uploaded.
2. Use the basedir and baseurl arguments instead of string replacing. Your paths could otherwise end up with double slashes (or even missing ones).

Once an image is uploaded, it’s subdir is stored in the database, and WordPress will only ever use basedir when attempting to retrieve the filepath.

In other words, things won’t break if wp_upload_dir is called for a custom post type attachment out of context, and the returned subdir is not actually where the file resides.