How to modify core when there is no hook?

You’re right that comment_form hardcodes the <form> tag, so you cannot modify it with a filter or action. Adding your custom attribute with jquery would be possible, but then it wouldn’t be in the source code and hence not be picked up by search engines you might be targeting. The alternative is to buffer the entire output of the function and do a search and replace there. Like this:

ob_start ('wpse193237_add_attribute_to_form');
comment_form();
ob_end_flush();

function wpse193237_add_attribute_to_form ($buffer) {
  return (str_replace ('<form', '<form data-my-custom="my_value"', $buffer));
  }