Redis cache vs File cached Magento 2.2.2 Benchmarks

[George_L]

First time Magento user who literally just did my first install yesterday via write up I made at https://github.com/centminmod/centminmod-magento2 so thought I'd share my first benchmarks of Redis backend, page cache and session caching versus file based caching native to Magento 2.2.2. Benchmarks used my custom forked wrk-cmm load testing tool.

Magento 2.2.2 install is done purely via SSH command line and the test KVM VPS server is of following specs:

• 4 CPU Intel 2.40Ghz cpu
• 4GB memory
• 80GB disk
• CentOS 7.4 64bit
• Centmin Mod 123.09beta01 installed via betainstaller71.sh
• Nginx 1.13.9
• PHP-FPM 7.1.15 + Zend Opcache
• MariaDB 10.1.31 MySQL Server
• CSF Firewall

Benchmarks

• Magento 2 Redis Benchmarks
• Magento 2 File Based Caching Benchmarks

Redis

n98-magerun2 sys:info --skip-root-check

                              
  Magento System Information  
                              

+------------------+-----------------------------------------------+
| name             | value                                         |
+------------------+-----------------------------------------------+
| Name             | Magento                                       |
| Version          | 2.2.2                                         |
| Edition          | Community                                     |
| Root             | /home/nginx/domains/magento.domain.com/public |
| Application Mode | production                                    |
| Session          | redis                                         |
| Crypt Key        | d45f000b87ce82*******************             |
| Install Date     | Sun, 11 Mar 2018 20:40:12 +0000               |
| Cache Backend    | Cm_Cache_Backend_Redis                        |
| Vendors          | Magento, MageHost, Dotdigitalgroup, Temando   |
| Attribute Count  | 134                                           |
| Customer Count   | 0                                             |
| Category Count   | 2                                             |
| Product Count    | 0                                             |
+------------------+-----------------------------------------------+

Redis TCP mode

domain=https://magento.domain.com
wrk-cmm -t2 -c2 -d10s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain
thread 1 created
thread 2 created
Running 10s test @ https://magento.domain.com
  2 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    31.15ms   10.16ms 158.71ms   97.59%
    Connect    14.00ms    5.32ms  17.76ms  100.00%
    TTFB       30.83ms   10.10ms 158.42ms   97.89%
    TTLB      316.32us  266.73us   4.59ms   97.25%
    Req/Sec    32.83      5.93    40.00     62.63%
  Latency Distribution
     50%   29.45ms
     75%   31.67ms
     90%   35.71ms
     99%   80.05ms
  655 requests in 10.06s, 4.64MB read
Requests/sec:     65.14
Transfer/sec:    472.44KB
thread 1 made 330 requests and got 328 responses
thread 2 made 328 requests and got 327 responses

Redis Unix Socket mode

domain=https://magento.domain.com
wrk-cmm -t2 -c2 -d10s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain
thread 1 created
thread 2 created
Running 10s test @ https://magento.domain.com
  2 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    29.77ms    9.12ms 134.42ms   96.97%
    Connect     9.30ms    4.76ms  12.67ms  100.00%
    TTFB       29.46ms    9.08ms 134.08ms   96.97%
    TTLB      306.88us  212.53us   2.83ms   97.51%
    Req/Sec    34.26      6.07    40.00     97.47%
  Latency Distribution
     50%   28.09ms
     75%   30.42ms
     90%   33.51ms
     99%   75.62ms
  684 requests in 10.05s, 4.84MB read
Requests/sec:     68.04
Transfer/sec:    493.39KB
thread 1 made 342 requests and got 340 responses
thread 2 made 345 requests and got 344 responses

File based caching

n98-magerun2 sys:info --skip-root-check

                              
  Magento System Information  
                              

+------------------+-----------------------------------------------+
| name             | value                                         |
+------------------+-----------------------------------------------+
| Name             | Magento                                       |
| Version          | 2.2.2                                         |
| Edition          | Community                                     |
| Root             | /home/nginx/domains/magento.domain.com/public |
| Application Mode | production                                    |
| Session          | files                                         |
| Crypt Key        | d45f000b87ce82*******************             |
| Install Date     | Sun, 11 Mar 2018 20:40:12 +0000               |
| Cache Backend    | Cm_Cache_Backend_File                         |
| Vendors          | Magento, MageHost, Dotdigitalgroup, Temando   |
| Attribute Count  | 134                                           |
| Customer Count   | 0                                             |
| Category Count   | 2                                             |
| Product Count    | 0                                             |
+------------------+-----------------------------------------------+

wrk-cmm -t2 -c2 -d10s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain
thread 1 created
thread 2 created
Running 10s test @ https://magento.domain.com
  2 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    39.78ms   43.07ms 357.86ms   95.21%
    Connect    15.02ms    1.14ms  15.83ms  100.00%
    TTFB       39.35ms   42.99ms 357.27ms   95.22%
    TTLB      426.65us  153.00us   3.21ms   90.85%
    Req/Sec    31.85      7.32    40.00     64.74%
  Latency Distribution
     50%   29.49ms
     75%   31.51ms
     90%   39.24ms
     99%  292.20ms
  612 requests in 10.05s, 4.33MB read
Requests/sec:     60.92
Transfer/sec:    441.84KB
thread 1 made 315 requests and got 313 responses
thread 2 made 300 requests and got 299 responses

Summary

Benchmarks with Redis caching disabled and switching caching to file based so only slightly less performance that Redis caching in terms of average throughput at 60 requests/s versus 65-68 requests/s with Redis caching. But 99% percentile latency response times were much slower with file based caching at 292 ms vs 75-80ms with Redis caching. Thread TTFB latency max was also much higher with file based caching at 357+ ms versus Redis cached thread TTFB latency at 134-158 ms.

It's very clear to me that Varnish Cache is really needed to scale Magento for Apache/Nginx setups as Magento's native Redis caching still needs Nginx/PHP to process the requests so not totally bypassing PHP processes like I'd imagine it would. With my tests with Redis Nginx level caching of a Wordpress install I managed to scale to 302 million hits/day or 3,500+ average and 5,100+ peak requests/sec.

During wrk-cmm load testing with Redis caching you can see the load still hitting Nginx and PHP-FPM.

top -c
top - 10:04:41 up  1:14,  2 users,  load average: 1.15, 0.94, 0.82
Tasks: 119 total,  20 running,  99 sleeping,   0 stopped,   0 zombie
%Cpu0  : 81.6 us, 16.4 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  1.6 si,  0.0 st
%Cpu1  : 79.5 us, 18.5 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu2  : 77.6 us, 20.1 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  2.0 si,  0.0 st
%Cpu3  : 76.9 us, 20.8 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
KiB Mem :  3881428 total,  2639780 free,   517272 used,   724376 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used.  3022348 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                     
  760 redis     20   0  159424   7524   1552 R  32.6  0.2   0:25.46 redis-server                                                
11543 nginx     10 -10  157724  45128   2300 S  20.0  1.2   0:12.43 nginx                                                       
13033 nginx     20   0  918384  27584  14580 R  20.0  0.7   0:01.47 php-fpm                                                     
13027 nginx     20   0  918384  27576  14572 R  19.0  0.7   0:01.56 php-fpm                                                     
13035 nginx     20   0  918384  27580  14576 R  18.7  0.7   0:01.59 php-fpm                                                     
13032 nginx     20   0  918384  27580  14576 R  18.4  0.7   0:01.52 php-fpm                                                     
13021 nginx     20   0  918480  27688  14588 R  18.1  0.7   0:02.20 php-fpm                                                     
13031 nginx     20   0  918384  27580  14576 R  18.1  0.7   0:01.48 php-fpm                                                     
13026 nginx     20   0  918384  27580  14576 R  17.7  0.7   0:01.85 php-fpm                                                     
13029 nginx     20   0  918384  27576  14572 R  17.7  0.7   0:01.68 php-fpm                                                     
13030 nginx     20   0  918384  27580  14576 R  17.7  0.7   0:01.58 php-fpm                                                     
13036 nginx     20   0  918384  27580  14576 S  17.7  0.7   0:01.48 php-fpm                                                     
13022 nginx     20   0  918384  27580  14576 R  17.4  0.7   0:02.20 php-fpm                                                     
13023 nginx     20   0  918420  27620  14584 R  17.4  0.7   0:02.07 php-fpm                                                     
13025 nginx     20   0  918384  27576  14576 R  17.4  0.7   0:02.26 php-fpm                                                     
13034 nginx     20   0  918384  27580  14576 R  17.4  0.7   0:01.49 php-fpm                                                     
13024 nginx     20   0  918468  27684  14588 R  17.1  0.7   0:02.28 php-fpm                                                     
13028 nginx     20   0  918384  27576  14572 R  16.5  0.7   0:01.61 php-fpm                                                     
11545 nginx     10 -10  153628  44512   2284 S  16.1  1.1   0:08.60 nginx                                                       
11547 nginx     10 -10  157724  44672   2224 R  14.8  1.2   0:10.76 nginx                                                       
11544 nginx     10 -10  153628  44512   2264 R  10.6  1.1   0:09.73 nginx                                                       
13018 root      20   0  307660   8072   2360 S   6.8  0.2   0:00.57 wrk-cmm

[George_L]

Next up is Magento 2 Varnish Cache Config & Benchmarks with Varnish Cache 5.2.x with HTTP/2 support proxying to a non-HTTPS magento.domain.com backend. For terminating HTTPS connections using Centmin Mod Nginx HTTP/2 HTTPS as reverse proxy in front of Varnish Cache as Varnish Cache can't fully handle HTTPS itself. The work flow looks like this:

Visitor > Nginx HTTP/2 HTTPS Proxy > Varnish Cache HTTP/2 Enabled > Nginx non-HTTPS backend

Quick benchmarks with Redis caching for sessions and backend and Varnish Cache for full page caching and a huge boost in performance compared to using Redis caching for full page caching jumping from 65-68 requests/sec and TTFB thread latency max at between 134-158ms.

first run without cache warm up with 569 requests/s and TTFB thread latency max at 581ms

domain=https://magento.domain.com
wrk-cmm -t2 -c2 -d10s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain
thread 1 created
thread 2 created
Running 10s test @ https://magento.domain.com
  2 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.03ms   53.21ms 581.69ms   97.69%
    Connect    12.42ms    9.14ms  28.15ms   55.00%
    TTFB       10.53ms   54.13ms 581.03ms   97.63%
    TTLB      709.65us    0.90ms  26.59ms   96.59%
    Req/Sec   294.81    119.88   620.00     64.95%
  Latency Distribution
     50%    2.97ms
     75%    3.94ms
     90%    7.20ms
     99%  356.80ms
  5736 requests in 10.08s, 39.96MB read
Requests/sec:    569.02
Transfer/sec:      3.96MB
thread 1 made 2680 requests and got 2678 responses
thread 2 made 3059 requests and got 3058 responses

second run with warmed up cache with 1,011 requests/second and TTFB thread latency max at 12.54ms

wrk-cmm -t2 -c2 -d10s --breakout -H 'Accept-Encoding: gzip' -s scripts/setup.lua --latency $domain
thread 1 created
thread 2 created
Running 10s test @ https://magento.domain.com
  2 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
   Latency     1.99ms    0.90ms  13.00ms   91.47%
   Connect     7.28ms    5.85ms  17.71ms   52.00%
   TTFB        1.46ms  679.66us  12.54ms   81.05%
   TTLB      505.80us  583.70us   6.74ms   91.19%
   Req/Sec   508.07    126.29   720.00     66.50%
  Latency Distribution
    50%    1.49ms
    75%    2.49ms
    90%    2.76ms
    99%    5.49ms
  10124 requests in 10.01s, 70.53MB read
Requests/sec:   1011.06
Transfer/sec:      7.04MB
thread 1 made 5153 requests and got 5151 responses

[MagenX]

with redis is much easier to autoscale servers. you have cache/session servers separately always connected.

[George_L]

---

@MagenXwrote:

with redis is much easier to autoscale servers. you have cache/session servers separately always connected.

---

You mean compared to file based cache or compared to Varnish cache ?

For file based cache comparison, I agree it's easier to scale redis servers as a sharded redis proxy setup, redis cluster or redis replication configuration but as long as Magento 2's Redis caching implementation requires requests to go through PHP, PHP will be the bottleneck at high concurrency traffic loads.So Redis cache for backend and session caching is fine But for Redis for full page caching doesn't seem like it will scale on it's own without scaling the underlying PHP setup.

Apart from redundancy, is there any point performance wise for Magento 2 to use redis in a replication master/slave setup ? I might try that later on as I wrote my own redis-generator.sh script so can try different redis configurations i.e. replication replicate master/slave + 3x sentinels

For Nginx proxy_cache you can do a multi-tier proxy cache reading to Nginx proxy_cached backends to scale as well. Wonder if there's a way to setup Varnish Cache to read the Redis full page caching stored content as a backend instead of Varnish going to Nginx backend ?

If using Varnish Cache in HTTP/2 HTTPS config setup, you'd have HTTP/2 HTTPS requests first going through Nginx as a reverse proxy to Varnish Cache.

I added a bit more info to my write up for Varnish Cache benchmarks  at https://github.com/centminmod/centminmod-magento2#varnish-cache-benchmark-test-results

Below are Top stats during wrk-cmm load testing benchmarks of a longer duration at 120 seconds which confirm this - that Nginx still receives the bulk of the work load. There will be at least 2 Nginx processes as wrk-cmm load tester is running with 2 threads.

top -c
top - 09:25:50 up 3 days, 36 min,  2 users,  load average: 2.45, 1.95, 1.36
Tasks: 114 total,   3 running, 111 sleeping,   0 stopped,   0 zombie
%Cpu0  : 50.3 us, 12.7 sy,  0.0 ni, 34.2 id,  0.0 wa,  0.0 hi,  2.4 si,  0.3 st
%Cpu1  : 44.2 us,  9.9 sy,  0.0 ni, 43.2 id,  0.0 wa,  0.0 hi,  2.4 si,  0.3 st
%Cpu2  : 44.4 us, 11.9 sy,  0.0 ni, 41.6 id,  0.0 wa,  0.0 hi,  1.7 si,  0.3 st
%Cpu3  : 20.2 us, 13.2 sy,  0.0 ni, 65.5 id,  0.0 wa,  0.0 hi,  0.7 si,  0.3 st
KiB Mem :  3881428 total,   384332 free,   703628 used,  2793468 buff/cache
KiB Swap:  4194300 total,  4193232 free,     1068 used.  2597960 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                     
 1600 nginx     10 -10  461340  44740   2312 S  73.1  1.2   1:14.43 nginx                                                       
 1599 nginx     10 -10  465436  44832   2384 R  62.1  1.2   1:24.64 nginx                                                       
 8260 root      20   0  307532   3440   2360 S  49.8  0.1   0:55.48 wrk-cmm                                                     
 1718 nginx     20   0  348612  38012   3636 S  11.6  1.0   4:22.14 amplify-agent                                               
 7715 nginx     20   0  916364  24888  13992 S   1.7  0.6   0:00.85 php-fpm                                                     
  771 root      20   0  550532  31332  17076 S   0.7  0.8  12:14.86 dockerd                                                     
 1127 mysql     20   0 1116544 304360  12460 S   0.7  7.8  30:28.38 mysqld                                                      
 1597 nginx     10 -10  461340  44668   2332 S   0.7  1.2   1:26.91 nginx                                                       
 8540 root      20   0  157784   2188   1472 R   0.3  0.1   0:00.09 top