Do I need to re-register all my images in the media library after a migration?

Assuming your old site is still accessible, I would just go Tools -> Export and create an export file containing your media.

In the new WordPress server, import the file and select the option to “Download and import file attachments”. (your old site needs to be public for this to work).

This seems to be good guidance: https://www.itjon.com/moving-posts-and-media-between-wordpress-blogs-with-wordpress-importexport/

If you just copy the uploads folder, WordPress knows nothing about this and all your image references will be 404. Even if you import the media by dragging everything from your uploads folder from old site, they will have different IDs, so I would advice to follow the import method.

Once you have done this, you might want to use a plugin to find stale images and remove them. There probably is a performance gain from getting rid of image references that are somewhere in the database.