I believe part of what you’re looking for is the WP_Embed class defined in wp-includes/media.php. It implements a framework for automatically replacing urls with the output of embed handlers. I’m afraid you’ll have to do the deeper research yourself (this was apparently introduces in 2.9.0, so documentation might still be thin).
I doubt WordPress core comes with a player solution or will ever do so. It would seem it installs a single default embed handler for googlevideo urls. For anything else you’ll have to write and register your own embed handler and if you want to host the files and player yourself, you’ll either need a 3rd party player (plugin) or rely on HTML5 video/audio tags.
Ahh…and the WP_Embed class seems to get added as a callback to the ‘the_content’ filters by default, so you shouldn’t need to do so manually. At least that’s what it looks like here. 😉
IF on the other hand you are not trying to replace URLs pointing to your media files, but rather complete tag enclosures like <video></video> or <embed></embed> and so on, then you’ll have to rely on regex or write your PHP XML manipulation functions, as i’m pretty sure WordPress doesn’t natively come with functions to do something like that.
You could, however, combine that with above, making your job a bit easier, by regex-erasing everything within those tags and the tags themselves and only leaving the url. If you regsiter the filter callback that does that with a higher priority than 10 on the ‘the_content’ filter hook, the WP_Embed filter callback will then react on these URLs and you can do the rest using the WP_Embed framework.