how to make custom ajax handler?

You have to include WP core function files, something like this…

<?php
//mimic the actuall admin-ajax
define('DOING_AJAX', true);

if (!isset( $_POST['action']))
    die('-1');

//make sure you update this line 
//to the relative location of the wp-load.php
require_once('../../../../../wp-load.php'); 

//Typical headers
header('Content-Type: text/html');
send_nosniff_header();

//Disable caching
header('Cache-Control: no-cache');
header('Pragma: no-cache');

You can read more about this here https://coderwall.com/p/of7y2q/faster-ajax-for-wordpress or here https://wp-dreams.com/articles/2014/03/better-ajax-handler-for-wordpress-super-fast-ajax/.