Ok got it, knew there must be some built-in WP solution to this. Simply call wp_get_referer() in your callback (for details, see this).
At least it’s working as I need it, let me know if there’s any better solution.
UPDATE
Thanks to @Tom J Nowell, we should also mention that referrers could get stripped for privacy reasons or similar. To thus be on the safe side, there’s no way around passing the referrer’s URL explicitely to the data sent across AJAX, like using document.location.href.