Git does not support downloading parts of the repository. You have to download all of it. But you should be able to do this with GitHub.
When you view a file it has a link to the “raw” version. The URL is constructed like so
https://raw.githubusercontent.com/user/repository/branch/filename
By filling in the blanks in the URL, you can use Wget or cURL (with the -L option, see below) or whatever to download a single file. Again, you won’t get any of the nice version control features used by Git by doing this.
Update: I noticed you mention this doesn’t work for binary files. You probably shouldn’t use binary files in your Git repository, but GitHub has a download section for each repository that you can use to upload files. If you need more than one binary, you can use a .zip file. The URL to download an uploaded file is:
https://github.com/downloads/user/repository/filename
Note that the URLs given above, from the links on github.com, will redirect to raw.githubusercontent.com. You should not directly use the URL given by this HTTP 302 redirect because, per RFC 2616: “Since the redirection might be altered on occasion, the client SHOULD continue to use the Request-URI for future requests.”