Playwright

Playwright is a modern end-to-end testing framework for web applications. It supports multiple browsers (Chromium, Firefox, WebKit) and provides reliable, fast testing capabilities.

Further information

Link: Playwright Documentation

Installation

npm install -D @playwright/test
npx playwright install

Modes

There are several modes to run the playwright tests.

Limiting/Filtering tests

If you run just the basic playwright test command, all identified tests will be executed:

npx playwright test

You can filter the tests in several ways.

Single testfile

Add a testfile to the command to execute only the tests in this file:

npx playwright test mytests.spec.ts

Single test

If you declare a single file, you can add the line number of the test to only execute this one test:

npx playwright test mytests.spec.ts:30

Filtering tests

You can add a text parameter to only execute those files which contain the passed text in their test name:

npx playwright test mytests.spec.ts 'should show error'

Headed Mode

By default, Playwright tests run in headless mode (no visible browser). To see the browser:

npx playwright test --headed

UI mode

In UI mode a small application launches and gives you a GUI with all available tests, where you can select the test, check the results and watch the executed steps.

npx playwright test --ui

Debug mode

In this mode you can execute the tests step-by-step.

npx playwright test --debug

Report

After running the tests you can open a visual report about the results:

npx playwright show-report

Debug Flaky Tests

Identifying flaky tests might be very time-consuming, especially if they fail on CI and work locally.

There are several steps you can try:

Disable retries

Playwright can be configured to retry failed tests. In that case you might not see the flaky test, if the test works by the next try.

Define the retries in playwright.config.ts

Check playwright.config.ts if retries defined > 0
retries: isCIEnvironment ? 1 : 0, // Retry once in CI environment

You can define the retries within the playwright test command:

npx playwright test --retries 0

Use the --fail-on-flaky-tests flag, which will fail if any test is flagged as flaky:

npx playwright test --fail-on-flaky-tests

Repeat tests

You can try to force the flakiness by executing the test several times:

npx playwright test --repeat-each 10

Machine stress test

Local machines are most of the time stronger than CI systems. You could stress your system by running multiple tests in parallel to reproduce the flakiness:

npx playwright test --workers 10

Stop after first failure

Running the tests multiple times can take some time and mess up your logs. Use the -x parameter to stop the execution as soon as a test fails:

px playwright test -x

An example with all options

npx playwright test --fail-on-flaky-tests --repeat-each 10 --workers 10 -x

Best practices

Avoid non-retry assertions

Always prefer auto-retrying assertions over non-retrying ones. Non-retrying assertions don't wait for conditions and can lead to flaky tests.

Further information

Link: Playwright Assertions

Use toPass instead of timeouts

If you have cases where you have to wait until some case happens, don't use timeout. Instead, wrap a condition into a function and call toPass. This will make Playwright retry the condition several times.

await expect(async () => {
  const response = await page.request.get('https://api.example.com');
  expect(response.status()).toBe(200);
}).toPass();

You can also specify custom timeout and retry intervals:

await expect(async () => {
  const response = await page.request.get('https://api.example.com');
  expect(response.status()).toBe(200);
}).toPass({
  // Probe, wait 1s, probe, wait 2s, probe, wait 10s, probe, wait 10s, probe
  // ... Defaults to [100, 250, 500, 1000].
  intervals: [1_000, 2_000, 10_000],
  timeout: 60_000
});

Wait for responses

Sometimes you have to wait for an response of an api call, before you can continue. In that case you can use wait for response

await page.waitForResponse('http://www.demopage.com/api/example/1')

Beware of floating promises

Floating promises are missing await calls or missing error handling in async functions.

That could lead to wrong functionality, as other code could get executed before the promise is finished. That could lead to errors or flaky tests.

To identify floating promises, enable the ESLint check for no-floating-promises.

Use the Locator for Element Identification

You can generate an optimized locator generated by Playwright with the help of codegen, playwright-ui, or the VS Code plugin.

Assertions

Generally, there are two types of assertions:

• Auto-retrying
• Non-retrying

Auto-retrying

Those assertions will retry until the assertion passes. All of them are async; therefore, they require await.

Examples

• await expect(locator).toContainText()
• await expect(locator).toBeChecked()
• await expect(locator).toHaveValue()

Non-retrying

Those assertions test without auto-retry.

Be aware that most of the time content is loaded asynchronously. Using non-retrying assertions may lead to flaky tests.

Use auto-retrying assertions whenever possible.

Examples

• expect(value).toContain()
• expect(value).toHaveLength()
• expect(value).toMatchObject()

Structuring and Documentation

Here are some best practices for structuring the tests and improving the result of the test report.

Annotations

With annotations you can link further information or structure the html report

test('test login page', {
  annotation: {
    type: 'issue',
    description: 'https://github.com/microsoft/playwright/issues/23180',
  },
}, async ({ page }) => {
  // ...
});

Smaller tips

• Never use waitForTimeout
• Use locators (getByTestId, getByText, getByRole) instead of selectors like CSS
• Do not use locator.all() - It returns immediately and does not wait for all elements. If you have to wait for the page to load, use page.waitForLoadState('networkidle'). It will wait until the page is fully loaded.

Codegen

With codegen it is very easy to generate playwright tests. This tool records your interaction with the browser ui an selects the best selector for you.

pnpm exec playwright codegen playwright.dev

Networking API

Avoid testing external systems/dependencies

If your application depends on external systems or other dependencies, try to avoid them, as they can make the tests complex, slow them down, or even make them fail.

Instead use the Playwright API to guarantee the result.

Return expected result

In Playwright it is very easy to return a result for a specific URL call:

await page.route('**/api/fetch_data_third_party_dependency', route => route.fulfill({
status: 200,
body: testData,
}));

You can define the route in general for all tests by using browserContext instead of page.

Abort request

await page.route('**/*.{png,jpg,jpeg}', route => route.abort());

Modify response

await page.route('**/title.html', async route => {
  // Fetch original response.
  const response = await route.fetch();
  // Add a prefix to the title.
  let body = await response.text();
  body = body.replace('<title>', '<title>My prefix:');
  await route.fulfill({
    // Pass all fields from the response.
    response,
    // Override response body.
    body,
    // Force content type to be html.
    headers: {
      ...response.headers(),
      'content-type': 'text/html'
    }
  });
});

Plugin

There is a very good plugin to integrate Playwright into VS Code, with several options.

Playwright Test for VS Code

Use Recorder

With the recorder, it is easy to record tests by clicking through the application. The recorded steps can be copied to the test suites.

Configuration

There are several important configurations you can tweak for your tests. The centralized config file is playwright.config.ts.

Timeouts

• timeout - Timeout for the test
• expect.timeout - Timeout for expect

Alternatively, you can overwrite them in the test code:

• test.setTimeout()
• test.slow() - Shortcut for triple timeout

Retries

You can define how many times Playwright should retry a failed test. This might solve some flaky tests.

// In playwright.config.ts
retries: 1

Traces

Playwright can generate a trace file about the test execution. A trace file is a zip containing all information to reproduce the test execution afterwards.

• On: All traces stored
• Off: None
• On first retry: First retry stored
• on-all-retries: Every retry
• retain-on-failure: All failures kept, success deleted

Screenshots

As many other UI test frameworks, Playwright can make screenshots of the tests. You can control when to take a screenshot with the following settings:

• on
• off
• only-on-failure

Videos

If you prefer videos of the execution of the tests, you can define when the videos should be recorded.

• on
• off
• only-on-failure
• on-first-retry