You are correct, this is not possible for serialised data. There are some hacks to try and get around this by searching for individual parts of the serialised string, but those throw up lots of false positives and can’t be sorted.
Generally, it’s an anti-pattern to store structured data inside a single meta value, and you can’t query the internal data in a meta_query field. If you absolutely must store it, store it as JSON, but keep in mind you can’t query or perform logic on the contents. ( serialised PHP values also open you up to object deserialisation attacks, and break when naive search replace queries are ran on the database if the length changes )
The solution here is to separate it out into individual atomic post meta, at least 5, do the sorting in PHP/JS,or to create child post types to represent your events so that you can use the standard date time columns