You did not mention the amount of RAM for your server?
Anyway firstly, check if the server itself is overloaded or if it is coping fine.
Do you have things APC, Memcached, Redis, Varnish, etc installed and tuned in your server?
Is the server managed by your hosting provider, a third-party or do you manage the server yourself?
... View more