I think you have the correct approach when you want to use a meta data value for it. I would do it the same way. I don’t quite know how downloads are arranged, but I think it shouldn’t be to hard to add an action which adds 1 to the count, after download.
A server overload is not really an issue unless you get 100’s of click in a short time.