You must be doing something really wrong and no caching solution is going to solve it. Disable plug-ins on WordPress that you don’t really need. If you use it once a month, you can enable it, use it, then disable it again. For example, I had bbPress installed but wasn’t really using it and it took 11MB of memory.
500 visitors a day eating up 12GB of memory (are you sure that’s memory and not bandwidth?) is a huge red flag. If everything worked fine in a shared hosting, and is now broken in your dedicated server, it’s your server setup causing the problems. If you are on a managed server, I would have them fix the problem. If it’s un-managed, you should get a managed server or go back to shared or start searching for help on fixing the problems/setup.
What shared host were you on? What dedicated server service are you using?
I’m assuming you mean 500 visits is 500 unique visitors, with more page views. 500 page views would be even lower demand site and any basic hosting site should be able to handle that.
500 visits is a low-demand site. I’m on a shared host and have served 2000 visits a day (with 15k page views) with less than 128MB of memory with minimal slowness.