as a shortcode within the loop?

Using the the_posts filter:

Here’s one idea using the the_posts filter, that fires before setup_postdata() is activated:

/**
 * Replace [nextpage] with <!--nextpage--> through the 'the_posts' filter.
 *
 * @see http://wordpress.stackexchange.com/a/183980/26350
 */

! is_admin() && add_filter( 'the_posts', function( $posts )
{
    $posts = array_map( function( $p )
    {
        if ( false !== strpos( $p->post_content, '[nextpage]' ) )
            $p->post_content = str_replace( '[nextpage]', '<!--nextpage-->', $p->post_content ); 
        return $p;
    }, $posts );
    return $posts;
});

The reason why your approach doesn’t work, is that your [nextpage] shortcode is generated after the setup_postdata(), that handles the post pagination, has checked for <!--nextpage--> in the post content.

Example:

Here I test it with some WordPress Lorem Ipsum:

with the following output on the front-end of the TwentyFifteen default theme: