How to submit form data in the same page in WordPress without reloading the page?

Your jQuery event is bound to elements with the class submit, but your submit button in the form doesn’t have that class.

Either add that class to the submit button, or just target the form itself. Simply change

$(".button").click(function() {

to

$("#modalForm").submit(function() {

This will fire when the form is submitted, whether the user clicks the submit button or simply hits enter in a text field.