Blink layout tests

How to land Skia changes that change Blink layout test results. See https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/web_tests.md for more detail on running the Blink layout tests.

General tips about layout tests

  • Layout tests come in two flavors: “compare 2 html pages” and “compare html page and .png file” When rebaselining, most of the effort comes from regenerating the .png files for the second kind. The first kind will be something like third_party/blink/web_tests/.../something.html and have a companion file .../something-expected.html as a sibling file. (example; companion html) The second type won’t have the companion html file, but might have a companion .png file, or multiple .png files in other directories when the html should render differently on other platforms or settings. (example; expected png) For this second type, using https://cs.chromium.org is a good way to find these if you need to look at rebaselining history or something

  • Layout tests (of both kinds) can be given fuzzy matching by adding a meta HTML tag to the test file. <meta name="fuzzy" content="maxDifference=0-4; totalPixels=0-100" />

  • Some non-layout tests (also called pixel tests) will fail as a result of rendering changes because they have their own checked-in images. Look at the logs of failing tests, as these will hopefully output base64 encoded pngs of the expected and actual image. Open up a browser tab, use Dev Tools to create an <img src="[base64]" /> with the actual base64 data and right-click to save the image as the new expected data. (example; expected png)

  • Some tests compare Skia and PyCairo. Since Skia makes different choices to Cairo, it’s best to increase fuzzy tolerance for these. Look for a fuzzy entry in the .yaml file that generates the tests and then regenerate things (or just use find and replace 🫣).

  • Failing CQ tests usually have a “Show Reproduction Instructions” for running locally. This can help verify fuzzy tolerances. Be sure to use –gtest_filter to limit what you are testing for faster iteration time.

Changes that affect a large number of test results

Where a ‘large number’ or ‘many’ means more than about 20, it’s a bit of a process to get things rebaselined in Chromium.

  1. Add a staging define to the Skia code that allows a client to (at compile time) opt-in to the old code path. If only Chromium needs rebaselining, it’s probably easier to set it up like if !defined(SK_USE_LEGACY_xxx). If this needs to be staged across multiple clients, if defined(SK_USE_NEW_xxx) is better to let clients “opt-in” one at a time. (example CL)

  2. Tell Chromium to use the old code path using a staging define in their SkUserConfig.h (or their //skia/BUILD.gn if it impacts only specific builds). (example CL)

  3. Land and wait for the autoroller to roll Skia into Chromium. (example CL autoroll CL)

  4. Create Chromium CL to use the new code path (by removing the define) and update expectations. Follow the rebaselining steps to update layout tests that use a reference image. For other types of tests (including the .html and -expected.html type), observe the above tips to manually update them. To update the images, you may have to repeat the flow of “sync”, “run tryjobs” and “rebaseline images from them” a few times due to other simultaneous changes or flaky tests. Feel free to add some <meta name="fuzzy" tags to any of the flaky tests.

    (example CL)

  5. Remove staging define from Skia. (example CL)

Changes that affect a small number of layout test results

Changes affecting fewer than ~20 layout tests can be rebaselined without a staging define using these steps:

  1. Prepare your Skia change. Run the Chromium-Canary tryjob and take note of which layout tests will turn red.
  2. Ahead of the Skia auto roll including your change, manually push a change to the Blink LayoutTests/TestExpectations file, flagging tests expected to fail as a result of your change as follows: 
    foo/bar/test-name.html [ Failure Pass ]  # Needs rebaseline
    There’s a section Skia roll test suppressions to use (to avoid conflicts with other changes).
  3. Check in your code to the Skia repo.
  4. Wait for the Skia roll to land successfully.
  5. In your Chromium checkout, create a new branch (e.g. git co main && gclient sync -D && git cl new-branch update-expectations). Follow the rebaselining steps and remove the suppressions from TestExpectations.