Check if do_action(‘custom_action’) is hooked into?

I believe you are looking for has_action, which should …

Check if any action has been registered for a hook.

if (has_action('custom_action')) {
  // stuff
} else {
  // different stuff
}