Let’s go simple

If I understand OP well, your problem is that urls containing a tilde are matched at all.

All other answers focus on the fact that sanitization for query strips out some characters before perform the query, however one should be capable to prevent a rewrite rule to don’t match under some circumstances.

And it is doable, not very easy, but doable.

Why it matches, in first place?

The reason why two urls like example.com/postname and example.com/postname~ match the same rewrite rule is because WP rewrite rule for posts use the rewrite tag %postname% that is replaced by the regex ([^/]+) when rewrite rules are created.

Problem is that regex ([^/]+) also matches the postname postname~ and, because of sanitization, the name queried will be postname ending up in a valid result.

It means that if we are able to change the regex from ([^/]+) to ([^~/]+) tilde will not match anymore so we actively prevent urls containing tilde in post name to be matched.

Since no rule will match, the url will end up to be a 404, which should be the expected behavior, I think.

Prevent matching

add_rewrite_tag is a function that, despite its name, can be used to update an existing rewrite tag like %postname%.

So, if we use the code:

add_action('init', function() {
  add_rewrite_tag( '%postname%', '([^~/]+)', 'name=" );
});

we will reach our target and example.com/postname~ will not match the rule for example.com/postname.

So, yes, the 3 lines above is the only code you”ll need.

However, before it works, you’ll need to flush rewrite rules by visiting permalink settings page on backend.

Note that regex ([^~/]+) prevent a tilde to be anywhere in post name, not only as trailing character, but since post names can’t actually contain tilde because of sanitization, that should be not a problem.