Skip to main content

Visual Regression Testing

This document is a guideline for running and creating visual regression tests in ATLAS project.

In order to safeguard the visual consistency of our components, we employ visual regression tests. A standard snapshot test captures a component's screenshot in the browser and compares it with reference snapshots. If a failure occurs and the two snapshots don't match, it indicates either an unexpected change or the need to update the reference snapshot to the new version.

For this purpose we use Cypress, a headless version of Chrome, and cypress visual regression plugin. In cypress we connect to our docusaurus host where we have a dedicated route for visual testing of components.

note

Visual regression tests run on GitHub Actions to ensure consistency across all test runs, regardless of the developer's local machine setup.

Running Visual Regression Tests​

Visual regression tests automatically run on every pull request via GitHub Actions. This ensures consistent results across all pull requests.

Automatic Test Runs​

  • Tests run automatically on every PR

Generating Fresh Baselines​

When you need to update baseline snapshots (e.g., after intentional visual changes):

Option 1: Add a Label to the PR

  1. Go to your pull request on GitHub
  2. Add the label generate-baseline-snapshots to your PR
  3. The baseline generation workflow will run automatically
  4. New baseline snapshots will be committed to your branch

Option 2: Manual Workflow Trigger

  1. Go to Atlas GitHub repository
  2. Navigate to Actions tab
  3. Select "Generate Visual Regression Baselines" workflow
  4. Click "Run workflow" button
  5. Select your branch and click "Run workflow"

Local Testing (Limited)​

You can run visual regression tests locally for development, but note that local tests may fail due to environment differences (fonts, rendering, OS-specific differences).

# Start the development server (in one terminal)
yarn start

# Run visual regression tests (in another terminal)
yarn visual-test

# Run a specific test file
yarn visual-test --spec "cypress/e2e/Button.cy.js"

# Open Cypress UI for interactive testing
yarn visual-test-ui
warning

Local baseline generation is not recommended. Always use GitHub Actions to generate baseline snapshots to ensure consistency.

If you need to update baseline snapshots locally for quick iteration:

yarn visual-test-update-snapshots

However, these should not be committed. Instead, use the GitHub Actions workflow to generate the baseline snapshots.

Creating New Visual Regression Tests​

1. Create a Test File​

Create a new test file in the cypress/e2e directory with the naming convention ComponentName.cy.js.

For example, if you are testing a Button component:

cypress/e2e/Button.cy.js

2. Import Necessary Support​

Start your test file by importing the required support packages:

import 'cypress-real-events/support';

const waitTime = Cypress.env('waitTime');

3. Define Test Description and beforeEach Block​

Begin your test by using the describe() function to provide a clear description of what the test is going to cover.

Inside the describe() block, create a beforeEach() block. This block sets up the environment that will be used before each test in the file.

4. Visit the URL and Set Viewport​

Inside the beforeEach() block, set up the URL for the component you want to test. This URL is constructed using the visualRegressionBaseURL and path variables.

Use cy.visit(url) to visit the specified URL in the Cypress browser.

Call cy.setViewport() to set the viewport. This ensures consistent viewport settings for your tests.

Putting it all together, the starting code for your first test in Cypress would look like this:

// Import necessary support
import 'cypress-real-events/support';

// Define waitTime if needed
const waitTime = 1000; // Replace 1000 with your desired wait time in milliseconds

// Write the test
describe('Description of Test', () => {
  beforeEach(() => {
    // Set up the environment for each test
    const visualRegressionBaseURL = Cypress.env('visualRegressionBaseURL');
    const path = 'component_name'; // Replace 'component_name' with the actual path to your component
    const url = `${visualRegressionBaseURL}${path}`;
    cy.visit(url);
    cy.setViewport(); // Set the viewport to your desired settings
  });

  // Your test cases go here...
});

5. Write Test Code​

Once you have set up the test file and the beforeEach() block, you can proceed to write your actual tests using the it() blocks. Each it() represents an individual test case. Inside these blocks, you'll identify the element you want to focus on using cy.get() and then use compareSnapshotWithConfig() to perform visual regression testing by comparing screenshots.

Look at an example test file for the Button component's visual regression tests:

import 'cypress-real-events/support';
const waitTime = Cypress.env('waitTime');

describe('Button component's visual regression tests', () => {
  beforeEach(() => {
    const visualRegressionBaseURL = Cypress.env('visualRegressionBaseURL');
    const path = 'Button'; 
    const url = `${visualRegressionBaseURL}${path}`;
    cy.visit(url);
    cy.setViewport(); 
  });

  // First test case
  it('Icon-left', () => {
    // Identify the element you want to test using cy.get()
    cy.get('#icon-left').compareSnapshotWithConfig('#icon-left', 'icon-left');
  });

  // Additional test cases can be added here...
});

In this example, we have a test case labeled 'Icon-left'. Inside this test case, we use cy.get('#icon-left') to select the element with the ID #icon-left (you can replace this with the appropriate selector for your component). We then call compareSnapshotWithConfig() to compare the screenshot of this element with the reference snapshot for the same element.

You can add more test cases by creating additional it() blocks. Each test case should focus on a specific aspect or state of the component you want to test, ensuring comprehensive coverage of its visual appearance.

tip

Look at existing visual regression pages in packages/docusaurus/src/pages/visual-regression/ for examples of how to set up component pages for testing.

note

If you want to add new features for your component test, be sure that the function is included in the design page. You can find page designs under packages/docusaurus/src/pages/visual-regression path.

Understanding Base and Actual Modes​

Base Mode​

The "Base" mode refers to the reference screenshots stored in cypress/snapshots/base/. These represent the expected output and are considered the source of truth.

  • Generated on GitHub Actions with consistent environment
  • Used as reference for comparison during test runs
  • Should be updated when intentional visual changes are made

Actual Mode​

The "Actual" mode contains screenshots from current test runs stored in cypress/snapshots/actual/. These are compared against base snapshots.

  • Generated during test runs
  • Compared pixel-by-pixel with base snapshots
  • Differences trigger test failures

Diff Images​

When tests fail, diff images are generated in cypress/snapshots/diff/ showing the differences between base and actual snapshots.

note

The actual/ and diff/ directories are gitignored and only exist locally or in CI artifacts.

When to Update Baselines​

You should regenerate baselines when:

  • ✅ Component visual styles are intentionally changed
  • ✅ New visual tests are added
  • ✅ Dependencies affecting rendering are updated (React, CSS libraries, etc.)
  • ✅ Design tokens are modified
note

It's crucial to update the "Base" screenshots whenever there are intentional changes in the application that reflect the desired behavior. This ensures that the tests accurately represent the expected output as the application evolves.