Varnish Cache Validation beginning 1/2/18

[etalbott]

Keeping in line with our efforts to improve the quality of all extensions on Marketplace, as of 1/2/18 we will be implementing Varnish cache validation during Technical Review for all M2 submissions.The Varnish test verifies products do not break Varnish caching when in production mode. 

Unfamiliar with caching and Varnish?  Review the following DevDocs pages:

• Configuration Guide
• Configure and Use Varnish
• PHP Developer Guide - Partial caching
• PHP Developer Guide - Full page caching

To properly prepare for this new test, review the overview and specifications, and incorporate Varnish cache testing into your extension validation prior to submitting to Marketplace. While this is an automated test, be aware there will be some instances where a manual review is required.

Overview and Specifications

The Varnish test is a basic FTF (functional test framework) test with two iterations.  Each iteration uses different data sets and assertions.  The test uses Magento sample data and sitemap to navigate through the data pages, opens different types of pages (product, category, etc.) and checks for expected cache headers in the server’s response (HIT or MISS).

Environment

The test will be run on a specific configuration:

• PHP 7.0.x latest
• Apache 2.4.6
• MySQL 5.6.x
• Varnish 4
• Magento 2.x latest with sample data

Test steps

Both variations run the same steps before running assertions. These steps are:

1. Setup configuration: Setup all necessary application configuration (i.e. enable Varnish caching)
2. Create Sitemap: Creation of a sitemap
3. Generate Sitemap: Sitemap generation
4. Flush Magento cache: Sanitize cache and isolate variation from impact of other variations
5. Get pages from Sitemap: Fetch necessary set of pages and pass them to the assertions

Assertions

1. AssertMissHeader: validates that after sending request for test page, server returns MISS for expected header.
2. AssertHitHeader: validates that after sending request for test page, server returns HIT for expected header.
3. AssertCacheInvalidatesAfterPriceChange: validates that after a price change, the cache for the product page is invalidated. On the first page load attempt, there is a Cache MISS and on the second attempt, there is a Cache HIT.

Known Test Issue

If an extension has a customized sitemap creation/generation flow, the sitemap can’t be created/generated using standard Magento routes.

Solution: In this instance, the extension will have to be reviewed manually.

Errors and Potential Solutions

PHP Fatal error: Uncaught Exception: Configuration settings are not applied!...

Error happens during Setup Configuration (Step 1) and the test tries to save   configuration settings.

Reason for failure: Incorrect settings in etc/adminhtml/system.xml.

Possible Solutions:

• Ensure new sections and tabs added by the extension are properly defined and their respective configurations can be retrieved and saved correctly.
• Review how defined new sections and tabs relate to each other.

PHP Fatal error: Uncaught Exception: Form key is absent! Url:...

Error occurs during Setup Configuration (Step 1) when the test tries to save configuration settings.

Reason for failure: Incorrect settings in etc/adminhtml/system.xml

Possible Solutions:

• Ensure new sections and tabs added by the extension are properly defined and their respective configurations can be retrieved and saved correctly.
• Review how defined new sections and tabs relate to each other.

PHPUnit_Extensions_Selenium2TestCase_WebDriverException: Timed out after 90 seconds…

Error occurs when Selenium cannot open a page during the test steps or assertions.

Reason for failure: Extension doesn’t work correctly in production mode and has crashed existing Magento functionality or cannot find some of the CSS selectors.

Possible Solutions: Due to the number of possible reasons for the failure, there is no simple answer for this failure other than to thoroughly test your extension in production mode and verify functionality is as intended.

Recommended Validation Steps

There isn’t a standalone test we can provide to the community for use in validation testing but here are the steps for manually testing your code before submitting to Marketplace.

1. Create a test environment as defined above under the Environment section.
2. Configure the environment to be the same as the automated test defined above under the Test Steps section.
3. Install extension.
4. Set Magento to production mode
5. Inspect page headers for all product pages.

         Success criteria:

• The first time accessing the page will always be a MISS.
• Subsequent times accessing the page will be a HIT.

6. Go to Magento 2 admin and change a product’s price.
7. Inspect page headers for all product pages (same as step 5).

         Success criteria:

• The first time accessing the page will be a MISS since data has changed
• Subsequent times accessing the page will be a HIT.

8. Resolve all failures before submitting to Marketplace.