WordPress’ URL routing is really difficult to work with. I’m looking forward to the day it’s replaced with something more rational (and I hope that I get to be the one to write the code to replace it, but I digress…)
However I think your first problem is trying to do everything in .htaccess. I’ve probably got more experience with URL design and URL rewriting than anyone else I know and I really wouldn’t try to fight WordPress with .htaccess; you are most likely only going to get the kind of results that are currently driving you crazy. Instead I’d try to manage URL routing from within WordPress and using hooks for when you need to override something WordPress is doing really screwy.
However, for me to fully understand what you are trying to accomplish and prescribe a proper solution I’d want to see a full map of URLs you have on your site and those that you want.
Finally I’ve linked some related Q&As below. They are not directly answering your question but they might give you some insight and the last one is about scaling WordPress for high traffic.