WordPress categorizes every upload as an attachment post-type which is built into your install by default in a similar manner as Pages, Posts, or Comments. If you delete row upon rows of attachment post types in your database you will lose availability to these in the Media Module ( Add Media button ) and won’t be able to edit the information ( title, alt, caption ), permanently delete, or attach the files onto additional posts. It will forever be lost in your /uploads/ folder with only a link in your post ( If you attached your file to a post ).
Imagine down the line your website is up and running for 10+ years and you want to go through and remove any old and unused media from your website and server. This can be easily done from the admin panel since WordPress knows ( using the database ) what files are attached to where by using post_parent. If you remove this database link you’ve orphaned your files onto your server and will need to reverse engineer what files are connected to what posts or if they’re connected to a post. You would spend years making charts and graphs stringing together what goes where and if you’ll need it again eventually spiraling into madness.
In short, it’s very unwise to delete these from the database and not get rid of them on the website. MySQL is extremely powerful and unless you have mountains of images and millions of rows then the actual speedboost you get from deleting these rows would probably be minimal.
TL;DR Stop it, don’t!