How can I apply a meta query for a single custom post type in the main query?

In the case that the start_date custom field only exists for the event post type, this works:

$query->set( 'meta_query', array(
    'relation' => 'OR',
    array(
        'key' => 'start_date',
        'compare' => 'NOT EXISTS',
    ),
    array(
        'key' => 'start_date',
        'value' => $today = date( 'Ymd' ),
        'compare' => '>=',
    ),
    array(
        'key' => 'end_date',
        'value' => $today = date( 'Ymd' ),
        'compare' => '>=',
    ),
) );

By comparing the start_date key to NOT EXISTS, the other post types are not filtered out. Obviously this is not an ideal solution, so I’m still curious if there are better ideas out there.