How to register a user via modal after clicking a link in the shortcode?

What you want to do can be achieved in two steps. First, create a registration form, then send the response to server and create the user.

Since you have skill in JS and HTML, I’ll directly get to business. Below is a very simple registration form. You will set this form to be hidden, for now.

Registration Form

<div id="ajax-register-modal">
    <input type="text" id="ajax-username" placeholder="<?php _e( 'Username', 'text-domain' ); ?>"/>
    <input type="password" id="ajax-password" placeholder="<?php _e( 'Password', 'text-domain' ); ?>"/>
    <span id="register-modal-send"><?php _e( 'Register', 'text-domain' ); ?></span>
</div>

Then when you click the link, you will make it visible by jQuery.

jQuery('#YourLinkID').click(function($){
    $('#ajax-register-modal').toggle();
});

So it pops up when we click the link. Now we send the data when the user clicks the register button.

Let’s bind an action to our button:

jQuery('#register-modal-send').click(function($){
    registerTheUser();
});

Server-side Registration

Now the AJAX handler:

function registerTheUser(){
    // Get the field values for username and password
    var username = $('#ajax-username').val();
    var password = $('#ajax-password').val();
    // Send them to server via AJAX
    jQuery.ajax({
        type: 'GET',
        url: 'REST URL HERE',
        data: { ajaxUsername: username , ajaxPassword: password},
    });
}

But wait, where would the data go? The final stage is to create a REST handler in the back-end, to deal with the registration. We simply create a REST route and register the user.

add_action( 'rest_api_init', function () {
    //Path to REST route for registration
    register_rest_route( 'LarryG001/v2', '/ajax_registration/', array(
            'methods' => 'GET', // I suggest using POST
            'callback' => 'register_the_user' // Callback PHP function
    ) );
});

Alright, let the registration begin!

function register_the_user(){
    // Get the username and password
    $username = $_GET['ajaxUsername'];
    $password = $_GET['ajaxPassword'];
    // A basic creation of user based on username and password. Email is optional
    $id = wp_create_user( $username, $password, $email );
    // Return the ID of the new user
    return $id;
}

Remember the URL in our AJAX call? This is what we should use there, since our request goes to this URL:

http://example.com/wp-json/LarryG001/v2/ajax_registration/

Done. We just created a new user by using wp_create_user function.

I omitted a lot of stuff here, such as checking if the values are set ( By using if(isset($_GET['ajaxUsername']) ).

After the data is returned, you can use it as your AJAX response.