one of my clients webshop's has a problem with the performance when loading the cart page. Loading this page takes 2 - 5 seconds with just one product, so I think this can improved to a much faster speed.
Using AOE_profiler I've found out that something between the application_clean_cache and model_save_after steps when loading the cart is taking a huge amount of time. But aoe_profiler doesn't say what process is taking so long.
Clearing the Magento cache will speed it up a bit, but after a few hours the performance is slow again.
I hope that this can be improved, as the website will grow their product catalog and so on also their amount of customers. So it would be nice if there are ways to improve this.
Are you using any FPC extension?
From your post it seems clearing cache, seems to be improving the speed a bit. So it coul be an I/O issue or caching server issue ( if you are using ).