How to list all images in uploads directory except those that are attached to any post

Under Media / Edit, there’s a means to list them, so I suggest you reuse the existing API.

Else, a general method, which works anywhere, is to run a db query with a left join:

select ID
  from foo
left join bar
    on whatever
 where condition
   and bar.ID is null;