Short Answer: No
A different question is:
Will uploading a large number of files in WordPress affect performance?
The answer to this: Most likely not.
Reason:
- WordPress uses the DB to get a list of uploaded files (almost no difference for many files)
- FTP uses the filesystem (slower for many files)
Details:
WordPress does not list the directory contents of the uploads folder, so you can dump any amount of files there without WordPress even noticing it.
When you upload a file via WordPress a new DB entry is made to your wp_posts table. WordPress only knows about the media files by looking into wp_posts, so uploading a ton of files will add more lines to that table and therefore makes SQL queries a bit slower.
Though honestly the amount (20.000 entries in DB) will not have a big impact on most hosting environments. Also WordPress has optimizations at every end to make it run fast (e.g. it caches SQL results and limits the number of queried rows)
However, as you noticed: When accessing the folder via FTP it freezes. This is caused by FTP requesting a list of all files the folder, and the server takes a while to answer with a list of 20.000 filesnames. But this performance issue is not affecting WordPress itself.