Magento 2.1.6 Performace Problems

[Mitch Singerman]

Hi,

My store used to be very fast, but now I am getting >10 second load times.  Recent system changes are, upgraded to 2.1.6, installed system patches, upped PHP FPM servers/children.   Below are my configuration files and system specs.   Any idea why I'm slow?  I'm at my wits end.   When I use curl I do see no-cache directives I don't think I should see that though, should I?  I also have varnish logs for a main site load, not included here, its 900+ lines.   Let me know if its useful and I will include.

Thanks for any help you can give.

Edit: Time to first byte is really high

curl fetch

curl -I -v --location-trusted https://www.mysite.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to www.mysite.com (xxx.xxx.xxx.xxx) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* TLSv1.2, TLS handshake, Client hello (1):
* TLSv1.2, TLS handshake, Server hello (2):
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
* TLSv1.2, TLS handshake, Server key exchange (12):
* TLSv1.2, TLS handshake, Server finished (14):
* TLSv1.2, TLS handshake, Client key exchange (16):
* TLSv1.2, TLS change cipher, Client hello (1):
* TLSv1.2, TLS handshake, Unknown (67):
* TLSv1.2, TLS handshake, Finished (20):
* TLSv1.2, TLS change cipher, Client hello (1):
* TLSv1.2, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: C=US; postalCode=***; ST=***; L=***; street=***; street=***; O=***; OU=InstantSSL Pro; CN=www.mysite.com
*        start date: 2015-04-29 00:00:00 GMT
*        expire date: 2018-05-06 23:59:59 GMT
*        subjectAltName: www.mysite.com matched
*        issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Organization Validation Secure Server CA
*        SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: www.mysite.User
> com-Agent: curl/7.42.1
> Accept: */*
>
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 14 Apr 2017 14:45:58 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=99999999999999999999999999999999; expires=Fri, 14-Apr-2017 15:45:56 GMT; Max-Age=3600; path=/; domain=www.mysite.com; secure; HttpOnly
Set-Cookie: mage-messages=%5B%5D; expires=Sat, 14-Apr-2018 14:45:58 GMT; Max-Age=31536000; path=/
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Thu, 14 Apr 2016 14:45:56 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
Strict-Transport-Security: max-age=63072000; includeSubdomains;

System

• openSUSE 13.2 (Harlequin) (x86_64)
• Redis 3.2.3
• nginx/1.11.4
• varnish-4.1.3 revision 5e3b6d2
• mysql Ver 15.1 Distrib 10.0.27-MariaDB, for Linux (x86_64) using readline 5.2
  

Magento Config

• Merge JavaScript Files: Yes
• Enable JavaScript Bundling: Yes
• Minify JavaScript Files: Yes
• Merge CSS Files: Yes
• Minify CSS Files: Yes
  
• Production mode

memory

# free -m
             total       used       free     shared    buffers     cached
Mem:          3958       3816        141         32          0       2931
-/+ buffers/cache:        885       3073
Swap:         2055          0       2055

CPU

DAEMON_OPTS="-a :80 \
             -T 127.0.0.1:6082 \
             -f /usr/local/etc/varnish/default.vcl \
             -p cli_buffer=16384 \
             -P /var/run/varnishd.pid \
             -p http_resp_hdr_len=64000 \
             -S /usr/local/etc/varnish/secret \
             -s malloc,256m \
             -F"

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            15
Model:                 4
Model name:            Intel(R) Xeon(TM) CPU 3.00GHz
Stepping:              1
CPU MHz:               3000.191
BogoMIPS:              5999.17
L1d cache:             16K
L2 cache:              1024K
NUMA node0 CPU(s):     0-3

Varnish

DAEMON_OPTS="-a :80 \
             -T 127.0.0.1:6082 \
             -f /usr/local/etc/varnish/default.vcl \
             -p cli_buffer=16384 \
             -P /var/run/varnishd.pid \
             -p http_resp_hdr_len=64000 \
             -S /usr/local/etc/varnish/secret \
             -s malloc,256m \
             -F"

vcl 4.0;

import std;

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
}

backend mysite {
    .host = "xxx.xxx.xxx.xxx";
    .port = "8080";
    .first_byte_timeout = 600s;
}

acl purge {
    "localhost";
    "192.168.0.0"/24;
}

sub vcl_recv {
   if (req.http.host ~ "mysite.com") {
     set req.backend_hint = mysite;
   } else {
     set req.backend_hint = default;
   }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        if (!req.http.X-Magento-Tags-Pattern) {
            return (synth(400, "X-Magento-Tags-Pattern header required"));
        }
        ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove Google gclid parameters to minimize the cache objects
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

    # static files are always cacheable. remove SSL flag and cookie
        if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
        unset req.http.Https;
        unset req.http.X-Forwarded-Proto;
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
}

sub vcl_backend_response {
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # validate if we need to cache it and prevent from setting cookie
    # images, css and js are cacheable by default so we have to remove cookie also
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
        if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
            set beresp.http.Pragma = "no-cache";
            set beresp.http.Expires = "-1";
            set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
            set beresp.grace = 1m;
        }
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
        beresp.http.Surrogate-control ~ "no-store" ||
        (!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

nginx.conf

user wwwrun www;
worker_processes  auto;

events {
    worker_connections  1024;
    use epoll;
}

http {
    include mime.types;
    include /etc/nginx/fastcgi.conf;
    default_type  application/octet-stream;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    server_tokens off;

    upstream fastcgi_backend {
        server unix:/var/run/php7-fpm.sock;
    }

    #mysite.com NON-SSL
    server {
        listen xxx.xxx.xxx.xxx:8080 default_server;
        server_name mysite.com www.mysite.com;
        set $MAGE_ROOT /srv/www/vhosts/mysite.com/htdocs;
        include magento.nginx_2.conf;
    }

    #mysite.com SSL
    server {
        listen xxx.xxx.xxx.xxx:443 ssl;
        server_name mysite.com  www.mysite.com;
        ssl_certificate      /etc/nginx/ssl.crt/www_mysite_com_2015-04-23-nginx-bundle.crt;
        ssl_certificate_key  /etc/nginx/ssl.key/www.mysite.com_2015-04-23.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AE
        ssl_stapling on;
        ssl_stapling_verify on;
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
        location / {
            proxy_pass http://127.0.0.1:80;
            proxy_read_timeout 600;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header Host $host;
        }
        set $MAGE_ROOT /srv/www/vhosts/mysite.com/htdocs;
        root $MAGE_ROOT/pub;
    }
}

[Mitch Singerman · Accepted solution]

Fixed it.   I looked in Cache Management and all caches were disabled.  I don't think that happened any other time up upgraded.  Lesson learned, check that the caches are enabled. lol