Is it safe to hand over the admin rights?

In short no.

The long answer. As an admin they have complete control of the content and options of the site, and (usually) what code is executed on the server. You can disable plug-in & theme editing/installing:

define( 'DISALLOW_FILE_EDIT', true );
define( 'DISALLOW_FILE_MODS', true );

(in your wp-config.php) but they can still do ‘damage’ by irrevocably deleting data (backup?).

In your context you seem to what to preserve the user’s ability to “download plugins etc”. In which case you’re explitly allowing them to execute any code they want on your server – they can do this with just access to the theme/plugin editor. If you’ve got multiple installs in sub-directories to the root folder allocated to by your host, then in general those other installs would also be vulnerable.

(If you’re running multi-site, then yes, obviously each site in the network is vulnerable.)