how to upload and allow downloads of .mobi and .epub formats

Assuming that you are using the WordPress native Media uploader then you can use the upload_mimes filter hook to add or remove allowed file types, for example:

function custom_myme_types($mime_types){

    //Adding avi extension
    $mime_types['avi'] = 'video/avi'; 

    //Removing the pdf extension
    unset($mime_types['pdf']); 

    return $mime_types;
}

add_filter('upload_mimes', 'custom_myme_types', 1, 1);

You can see that to add a file type you just add to the array with the file extension as the key and the mime type as the value do the MIME type for .mobi format file is application/x-mobipocket-ebook and for .epub format is application/epub+zip