You can create separated sitemap for each language like your example above, or add an alternate URL for each item for each language in a sitemap.
The first approach requires adding custom rewrite rules to WordPress, and add some queries to get all posts within a language. This is a big job and should be done by the plugins. I think SEO plugin should do that, if they go this way.
The second approach works better since it doesn’t require custom rewrite rules or queries. Instead of that, it only add alternate URLs for each language (which are provided by multilingual plugin like Polylang) to each post in the sitemap. You don’t need to change/add your sitemap URL, just add more content to that. To see the URLs, you must view the source code of the sitemap.
I think most SEO plugin automatically adds integration for popular multilingual plugins. You can verify this by viewing the source code of the sitemap for posts.
If you need more details about these approaches, Slim SEO wrote a detailed blog post about this.