The function WordPress ultimately uses for this is get_adjacent_post(), which has filters that allow you to modify the query directly. You’d have to modify the JOIN, WHERE and ORDER BY clauses to join the post meta table and order by your custom field.
Another option is the Ambrosite Next/Previous Post Link Plus plugin.