Is there a way to download only the Rest API part of WordPress?

WP REST API depends on WordPress, as it is just another interface for interacting with its other APIs. If BackPress (https://backpress.org/) was still in development, you could have had some chance with it, but it seems the project is abandoned.

The plugin that was merged with WordPress Core is available here: https://github.com/WP-API/WP-API but is also abandoned, so you will not get the latest stuff that’s in 5.8. I don’t know whether this would be of help to you, as I don’t know your main goal, but I would rather try to extract whatever I need from nowadays’ Core, than messing with the old plugin.