Keeping in line with our efforts to improve the quality of all extensions on Marketplace, as of 1/2/18 we will be implementing Varnish cache validation during Technical Review for all M2 submissions.The Varnish test verifies products do not break Varnish caching when in production mode.
Unfamiliar with caching and Varnish? Review the following DevDocs pages:
To properly prepare for this new test, review the overview and specifications, and incorporate Varnish cache testing into your extension validation prior to submitting to Marketplace. While this is an automated test, be aware there will be some instances where a manual review is required.
Overview and Specifications
The Varnish test is a basic FTF (functional test framework) test with two iterations. Each iteration uses different data sets and assertions. The test uses Magento sample data and sitemap to navigate through the data pages, opens different types of pages (product, category, etc.) and checks for expected cache headers in the server’s response (HIT or MISS).
The test will be run on a specific configuration:
Both variations run the same steps before running assertions. These steps are:
Known Test Issue
If an extension has a customized sitemap creation/generation flow, the sitemap can’t be created/generated using standard Magento routes.
Solution: In this instance, the extension will have to be reviewed manually.
Errors and Potential Solutions
PHP Fatal error: Uncaught Exception: Configuration settings are not applied!...
Error happens during Setup Configuration (Step 1) and the test tries to save configuration settings.
Reason for failure: Incorrect settings in etc/adminhtml/system.xml.
PHP Fatal error: Uncaught Exception: Form key is absent! Url:...
Error occurs during Setup Configuration (Step 1) when the test tries to save configuration settings.
Reason for failure: Incorrect settings in etc/adminhtml/system.xml
PHPUnit_Extensions_Selenium2TestCase_WebDriverException: Timed out after 90 seconds…
Error occurs when Selenium cannot open a page during the test steps or assertions.
Reason for failure: Extension doesn’t work correctly in production mode and has crashed existing Magento functionality or cannot find some of the CSS selectors.
Possible Solutions: Due to the number of possible reasons for the failure, there is no simple answer for this failure other than to thoroughly test your extension in production mode and verify functionality is as intended.
There isn’t a standalone test we can provide to the community for use in validation testing but here are the steps for manually testing your code before submitting to Marketplace.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.