You should be using admin-ajax.php for your AJAX requests and WordPress has pretty good wrappers and actions to help you do that.
In terms of performance, 3-4s is pretty harsh. WordPress will typically serve a non-cached request in about 300ms out of the box, so some other theme or plugin is adding to that time, probably by doing something silly and time-consuming during init.
Also, calls to admin-ajax.php can also be cached and served from cache instantly, especially since your only variance seems like a post id.
Finally, you should probably be checking the status and post type of $post, because given your current code, anybody could read the fields you’re providing for any post in your database, including private posts, protected ones, drafts, schedule posts, WooCommerce items, debug data or logs from some plugin storing stuff as post types, etc.