Delete everything but wp-content and wp-config.php, copy the fresh installation into the directory. On upgrading WordPress will use the database to see what should be done, not the files.
Not all files are deleted automatically, because some of them might still be used by outdated plugins or external scripts (the old feed files are good examples for this case).