MailPoet Performance Tests
Automated k6 performance tests for MailPoet. To be used for benchmarking performance (both single user and under load) by simulating and measuring the response time of browser-level and protocol-lever tests (protocol-level yet to be implemented).
Table of contents
Pre-requisites
Installing k6
You don't need to install it - it's automatic!
To execute the tests locally, check the below instructions under Running Scenarios
The k6 installation is automatic. It will download the binary file in /tools/vendor/ folder. Associated file for auto installing is /tools/k6.php.
Configuration
Test Environment
This test suite is using original MailPoet's environment with installed WordPress, MySQL and stuff like that.
The local env's url of the performance test site is: https://localhost:9500
When you're done with testing, you may want to stop the environment by the following command:
./do test:performance-clean
Config Variables
config.js comes with some test data and __ENV values stored in your .env file. You don't need to change the values there if you want to use different test site, user and password, but just include parameters when executing the test such as:
./do test:performance --scenario pullrequests --url="yoururl" --pw="yourpassword"
Config Variables List
These are some of the very important variables:
baseURL = this is the test site url
scenario = this is used for performing different scenarios, currently supported pull requests and nightlytests only
projectName = this is used for streaming results to Grafana Cloud k6. No need to use if testing locally, this is for CI.
k6CloudID = this is used for streaming results to Grafana Cloud k6. Same as above.
adminPassword = this is used for logging to a test site, locally or against online test site.
Grafana k6 Cloud
We use Grafana k6 Cloud to streamline our test results. This is happening only for the CI tests. However, you can even stream the local tests in case you need it.
For the Grafana k6 Cloud access, please refer to the secret store and search for MailPoet: Grafana
Currently, the local test env. checks if your .env file contain the token for Grafana K6_CLOUD_TOKEN=, and if not, streaming to Grafana won't happen. In case you add it, it will automatically stream (there's cloud token in CI saved so it streams there).
You can have K6_CLOUD_ID= applied in your .env, but keep the token var empty.
Running Tests
When refering to running k6 tests usually it means executing some of the scenarios available. The test scenario file in turn determines which requests we run and what metrics and thresholds will be applied to them. It is also possible to execute individual test files containing requests and pass in scenario config as a CLI flag but scenario files allow for more configuration options.
Running Individual Tests
To execute an individual test file (for example newsletter-listing.js):
./do test:performance tests/performance/tests/newsletter-listing.js
This will run the individual test for 1 iteration.
Running Scenarios
To execute a test scenario for pull requests, as an example:
./do test:performance --scenario pullrequests
This will run scenario with associated tests and options specificed inside scenarios.js.
Included in the tests folder is a set of test scripts which includes all scenarios and that can be ran or used as a starting point to be modified to suit the context in which tests are being ran.
Another aspect that affects the traffic pattern of the tests is the amount of “Think Time” in between requests. In real world usage of an application users will spend some amount of time before doing some action. For some situations there will be a need to simulate this as part of the test traffic and it is common to have this as part of load tests.
To do this a sleep step is included between each request sleep(randomIntBetween(`${think_time_min}`, `${think_time_max}`)).
The amount of think time can be controlled from config.js.
Note for Protocol-level tests: It’s important to note to be very careful when adding load to a scenario. By accident a dangerous amount of load could be ran aginst the test environment that could effectively be like a denial-of-service attack on the test environment. Also important to consider any other consequences of running large load such as triggering of emails.
Debugging Tests
Browser-level tests: the easiest way is to turn off headless mode by running individual or scenario with this argument --head and it will execute test with browser. The example command would be:
./do test:performance [test script path here] --head
Protocol-level tests: to help with getting a test working, the --http-debug="full" flag prints to console the full log of requests and their responses. It is also useful to use console.log() to print messages when debugging.
Results Output
Default Terminal Output
When you run a single test script, there will be a default k6 output in the end, but without applied thresholds. Threshold are applied only on the scenarios.js script, so when running a set of test scripts.
HTML Output with k6 Reporter
The k6 reporter should have already been implemented. There's generated file called k6report.html inside performance folder after every suite/scenario execution. You can find the path and open it in a local browser or see it as artefact in CircleCI.
See this guide for more details
Writing Tests
Start by adding a new file under tests folder and make sure it follows the name order with the other tests.
Then, you can see and copy/paste the imports from the other tests, as they might be the same... along with the comments on the top.
Then, begin with adding a new function called with your new test, for example:
export async function yourNewTest() {
and make sure to add browser and page constants.
Then you're ready to start your first test.
The very first step is to add the method goto(), here's an example:
await page.goto(${baseURL}/wp-admin... as you can see, we have baseURL here for the domain name and the rest is the page url where you want to land.
The second step is to add authenticating method, currently we use authenticate(page).
The third step is to tak a screenshot (you can later add more in the test), we want to have screenshots for easier debugging especially in CI.
When adding screenshot name, please stay in line with other screenshots as they are stored altogether in a single folder _screenshots.
The screenshot name should consist of the test name, example: Subscribers_Filtering_01.png. Please add one in the beginning and at least one in the end, but also in between if the test is longer.
The rest of the test should contain a very simple E2E steps using k6 browser module, here are some links for the commands: Selecting elements, or checking the full list of k6 browser possibilities. Note: Because k6 does not run in NodeJS, the browser module APIs will slightly differ from their Playwright counterparts.
To make assertions, we are using k6chai.js library, here is an example how you should assert things:
describe(subscribersPageTitle, () => {
describe('should be able to see Lists Filter', () => {
expect(page.locator('[data-automation-id="listing_filter_segment"]')).to
.exist;
});
In order to keep the output results in a clear view, we use describe to group them.
There's required sleep at the end, above is explanation in Running Scenarios section.
To complete and export the test to scenarios.js, you need to end it with export default async function yourNewTestTest() {.
After doing that, make sure to add the test in scenarios.js in appropriate section by pasting yourNewTest() function.
Finally, you need to check if your new test require any change to the existing options, thresholds, tags or maxDuration in scenarios.js.
That's it. Running the scenarios will include also your test. In nightly() scenario, you can include test that relies on premium, or even sending. These tests are executing against mpperftesting.mystagingwebsite.com website hosted on Pressable.com and have a premium key applied.
Capturing Requests
Browser-level tests: k6 tests rely on Playwright technology to make E2E tests where then k6 measure performance with its metrics, tresholds and checks. There are also groups to divide one scenario intro parts like one group for creating newsletter and going through pages and the one for sending it, so we could measure performance of the whole scenario in single or multiple parts.
Protocol-level tests: k6 tests rely on HTTP requests in order to test the backend. They can either be constructed from scratch, by using the k6 recorder, or by converting a HAR file. The k6 recorder is a browser extension which captures http requests generated as you perform actions in a tab. It generates a test with all the HTTP requests from your actions which can then be modified to make it execuatable.
Request Headers For Protocol-level Tests
Every HTTP requests tested includes the headers.
To make it easier to manage the headers they have been moved to a separate file so any changes can be made to all the requests at once.
In headers.js the common headers are grouped by type and then can be imported in for use in the requests. However if an individual request uniquely needs a specific header this can still be added in as an extra member of the headers object literal of that request.
If you don't see headers.js inside performance folder, that means we don't have yet protocol-level tests included. Feel free to ignore this section then.
Groups (deprecated in our tests)
Groups are used to organize common logic in the test scripts and can help with the test result analysis. For example the group "Proceed through the newsletter creation flow until Send step" groups together multiple requests triggered by this action.
Checks
There are checks inside the tests but they aren't failing tests, instead, they are more of a check that the process we use in tests are finished.
If you find less than this in our CI runs over to Grafana streaming, that means some of the tests are failing in execution and they require investigation, whether due to maintance or bug found.
Other Resources
k6 documention is a very useful resource for test creation and execution. using k6 browser doc is a very useful resource for test creation and execution.