1 .. SPDX-License-Identifier: GPL-2.0+
3 =========================================
4 Automated testing of the DRM subsystem
5 =========================================
10 Making sure that changes to the core or drivers don't introduce regressions can
11 be very time-consuming when lots of different hardware configurations need to
12 be tested. Moreover, it isn't practical for each person interested in this
13 testing to have to acquire and maintain what can be a considerable amount of
16 Also, it is desirable for developers to check for regressions in their code by
17 themselves, instead of relying on the maintainers to find them and then
20 There are facilities in gitlab.freedesktop.org to automatically test Mesa that
21 can be used as well for testing the DRM subsystem. This document explains how
22 people interested in testing it can use this shared infrastructure to save
23 quite some time and effort.
29 drivers/gpu/drm/ci/gitlab-ci.yml
30 --------------------------------
32 This is the root configuration file for GitLab CI. Among other less interesting
33 bits, it specifies the specific version of the scripts to be used. There are
34 some variables that can be modified to change the behavior of the pipeline:
37 Repository that contains the Mesa software infrastructure for CI
40 A particular revision to use from that repository
43 URL to git repository containing the target branch
46 Branch to which this branch is to be merged into
49 Revision of igt-gpu-tools being used, from
50 https://gitlab.freedesktop.org/drm/igt-gpu-tools
52 drivers/gpu/drm/ci/testlist.txt
53 -------------------------------
55 IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
58 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
59 ----------------------------------------------------------
61 Lists the known failures for a given driver on a specific hardware revision.
63 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
64 -----------------------------------------------------------
66 Lists the tests that for a given driver on a specific hardware revision are
67 known to behave unreliably. These tests won't cause a job to fail regardless of
68 the result. They will still be run.
70 Each new flake entry must be associated with a link to the email reporting the
71 bug to the author of the affected driver, the board name or Device Tree name of
72 the board, the first kernel version affected, and an approximation of the
75 They should be provided under the following format::
77 # Bug Report: $LORE_OR_PATCHWORK_URL
78 # Board Name: broken-board.dtb
83 drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
84 -----------------------------------------------------------
86 Lists the tests that won't be run for a given driver on a specific hardware
87 revision. These are usually tests that interfere with the running of the test
88 list due to hanging the machine, causing OOM, taking too long, etc.
91 How to enable automated testing on your tree
92 ============================================
94 1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
97 2. In your kernel repo's configuration (eg.
98 https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
99 CI/CD configuration file from .gitlab-ci.yml to
100 drivers/gpu/drm/ci/gitlab-ci.yml.
102 3. Request to be added to the drm/ci-ok group so that your user has the
103 necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok
105 4. Next time you push to this repository, you will see a CI pipeline being
106 created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
108 5. The various jobs will be run and when the pipeline is finished, all jobs
109 should be green unless a regression has been found.
112 How to update test expectations
113 ===============================
115 If your changes to the code fix any tests, you will have to remove one or more
116 lines from one or more of the files in
117 drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
118 affected by the change.
121 How to expand coverage
122 ======================
124 If your code changes make it possible to run more tests (by solving reliability
125 issues, for example), you can remove tests from the flakes and/or skips lists,
126 and then the expected results if there are any known failures.
128 If there is a need for updating the version of IGT being used (maybe you have
129 added more tests to it), update the IGT_VERSION variable at the top of the
133 How to test your changes to the scripts
134 =======================================
136 For testing changes to the scripts in the drm-ci repo, change the
137 DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
138 drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
139 janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
142 How to incorporate external fixes in your testing
143 =================================================
145 Often, regressions in other trees will prevent testing changes local to the
146 tree under test. These fixes will be automatically merged in during the build
147 jobs from a branch in the target tree that is named as
148 ${TARGET_BRANCH}-external-fixes.
150 If the pipeline is not in a merge request and a branch with the same name
151 exists in the local tree, commits from that branch will be merged in as well.
154 How to deal with automated testing labs that may be down
155 ========================================================
157 If a hardware farm is down and thus causing pipelines to fail that would
158 otherwise pass, one can disable all jobs that would be submitted to that farm
159 by editing the file at
160 https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.