Magento2 works differently to Magento1, ourselves with the consultants have spent the past 6mths designing a cluster architecture which achieves 98% performance ranking on categories and 95% on product pages, this was the evolution on AWS, avoiding 'local' files so an update to the 'master' was reflected across the cluster.
1x LB / 2x DB Galera (Write & Read) / 3x Web (or more) / 1x Elasticsearch/Redis / 1x Varnish on Magento 2.4.3 with 4websites having 1store each
First release was to use EFS however as expected it was intensely slow.
Second release was to use Gluster, but on testing with 10,000 products was no faster than EFS.
Third release was to use Ceph, this is fast, but a beyond miserable experience to install for the consultants, however provides cross web cluster RAID with effective instant replication.
The standard topology for a Magento install is a load balancer with two or more Varnish instances which talk to the web server cluster, however, if you are looking for organic Google ranking there is a problem, you cannot shard the Varnish cache across instances and 'warming' fails as the product count increases.
They implemented an alternative approach which provides better performance for organic ranking by making the web servers the first entry point after the load balancer, not varnish, within weeks we had 10% of the site ranked by Google with zero advertising on 15,000products, working our way to 250,000+ products.
Magento slows the higher the product count, and especially the more stores you have which was a major problem in Magento 1x, using the AWS methods such as RDS are less optimal than raw servers with these installed, as our product count will be so high we had to provide maximum performance from the outset.
Most likely the problem is the combination of opcache/redis/varnish/database not working in sync but when it comes to performance, the devil is in the detail, high store counts are the worst of the worst for Magento and using AWS (which is already slower compared to bare metal) with their RDS/Redis services the load times will become unsustainable.
Is there a solution, try the dual varnish from load balancer in to a web cluster connected to a single redis instance, after that you will need a Magento performance optimized web cluster, Nexcess have them but charge $5k to $10k/mth, we had issues paying the 80% of that as their monthly management fees, so instead went the upfront route with the consultant cost and run on 15-20% of that monthly, if you can separate the 20stores in to multiple websites it should help.
MochiGroup.net
