Rewriting get_next_posts_link() for custom loop

I don’t know what do you mean by a ‘custom loop’ (perhaps you are using a custom WP_Query object?) but here is how I would do it in a normal situation (code hasn’t been tested):

Note: My initial solution made use of the add_rewrite_endpoint function which turns out to be useless in terms of making pagination work.

function wt_rewrite_rule() {
    add_rewrite_tag('%selfie%', '([^&]+)');

    add_rewrite_rule( 'author/([^/]+)/selfie/page/([0-9]+)/?$', 'index.php?author_name=$matches[1]&selfie=1&paged=$matches[2]', 'top' );
    add_rewrite_rule( 'author/([^/]+)/selfie/?$', 'index.php?author_name=$matches[1]&selfie=1', 'top' );
}
add_action( 'init', 'wt_rewrite_rule' );

As for linking to the author page from within the loop you can simply build the URL like this:

$author_url = esc_url( get_author_posts_url( get_the_author_meta( 'ID' ) ) ) . 'selfie/';

And you can check for the presence of that var like so:

global $wp_query;

if ( isset( $wp_query->query_vars['selfie'] ) ) {
    // Your custom code here
}

Unlike using add_rewrite_endpoint, your pagination should also work.

Just to be clear, as far as I understand you’ve mentioned that a paged link like this:

http://localhost/author/ricky/selfie/page/2/

wouldn’t work. The above solution aims to fix this and make it work as expected.