I changed "Enable External Cache" to "no" and flushed the cache for a few mins to see if that effected loading times, but they were about the same. Is that an effective way of testing if the cache is being using?
(I know the cache files are being created ok, I can see them in FTP, I just suspect that they are not being used when loading the pages)