Does the WordPress core software handle bounces on system email?

The quick answer: no. At least not out of the box.

The bouncing email goes to the mail server and if there is no mailbox, catch-all or redirect address set, it will just bounce. You would need to connect WordPress to via pop3 or IMAP to receive emails.

There is at least one plugin for using WordPress as a mail client. I am not sure how it works, so I don’t know secure it is and how it (potentially) impacts WordPress performance.