1 ===========================================
2 Fault injection capabilities infrastructure
3 ===========================================
5 See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
8 Available fault injection capabilities
9 --------------------------------------
13 injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
17 injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
21 injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
25 injects futex deadlock and uaddr fault errors.
29 injects kernel RPC client and server failures.
33 injects disk IO errors on devices permitted by setting
34 /sys/block/<device>/make-it-fail or
35 /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
39 injects MMC data errors on devices permitted by setting
40 debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
44 injects error return on specific functions, which are marked by
45 ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
46 under /sys/kernel/debug/fail_function. No boot option supported.
48 - NVMe fault injection
50 inject NVMe status code and retry flag on devices permitted by setting
51 debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
52 status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
53 retry flag can be set via the debugfs.
55 - Null test block driver fault injection
57 inject IO timeouts by setting config items under
58 /sys/kernel/config/nullb/<disk>/timeout_inject,
59 inject requeue requests by setting config items under
60 /sys/kernel/config/nullb/<disk>/requeue_inject, and
61 inject init_hctx() errors by setting config items under
62 /sys/kernel/config/nullb/<disk>/init_hctx_fault_inject.
64 Configure fault-injection capabilities behavior
65 -----------------------------------------------
70 fault-inject-debugfs kernel module provides some debugfs entries for runtime
71 configuration of fault-injection capabilities.
73 - /sys/kernel/debug/fail*/probability:
75 likelihood of failure injection, in percent.
79 Note that one-failure-per-hundred is a very high error rate
80 for some testcases. Consider setting probability=100 and configure
81 /sys/kernel/debug/fail*/interval for such testcases.
83 - /sys/kernel/debug/fail*/interval:
85 specifies the interval between failures, for calls to
86 should_fail() that pass all the other tests.
88 Note that if you enable this, by setting interval>1, you will
89 probably want to set probability=100.
91 - /sys/kernel/debug/fail*/times:
93 specifies how many times failures may happen at most. A value of -1
96 - /sys/kernel/debug/fail*/space:
98 specifies an initial resource "budget", decremented by "size"
99 on each call to should_fail(,size). Failure injection is
100 suppressed until "space" reaches zero.
102 - /sys/kernel/debug/fail*/verbose
104 Format: { 0 | 1 | 2 }
106 specifies the verbosity of the messages when failure is
107 injected. '0' means no messages; '1' will print only a single
108 log line per failure; '2' will print a call trace too -- useful
109 to debug the problems revealed by fault injection.
111 - /sys/kernel/debug/fail*/task-filter:
113 Format: { 'Y' | 'N' }
115 A value of 'N' disables filtering by process (default).
116 Any positive value limits failures to only processes indicated by
117 /proc/<pid>/make-it-fail==1.
119 - /sys/kernel/debug/fail*/require-start,
120 /sys/kernel/debug/fail*/require-end,
121 /sys/kernel/debug/fail*/reject-start,
122 /sys/kernel/debug/fail*/reject-end:
124 specifies the range of virtual addresses tested during
125 stacktrace walking. Failure is injected only if some caller
126 in the walked stacktrace lies within the required range, and
127 none lies within the rejected range.
128 Default required range is [0,ULONG_MAX) (whole of virtual address space).
129 Default rejected range is [0,0).
131 - /sys/kernel/debug/fail*/stacktrace-depth:
133 specifies the maximum stacktrace depth walked during search
134 for a caller within [require-start,require-end) OR
135 [reject-start,reject-end).
137 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
139 Format: { 'Y' | 'N' }
141 default is 'Y', setting it to 'N' will also inject failures into
142 highmem/user allocations (__GFP_HIGHMEM allocations).
144 - /sys/kernel/debug/failslab/ignore-gfp-wait:
145 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
147 Format: { 'Y' | 'N' }
149 default is 'Y', setting it to 'N' will also inject failures
150 into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations).
152 - /sys/kernel/debug/fail_page_alloc/min-order:
154 specifies the minimum page allocation order to be injected
157 - /sys/kernel/debug/fail_futex/ignore-private:
159 Format: { 'Y' | 'N' }
161 default is 'N', setting it to 'Y' will disable failure injections
162 when dealing with private (address space) futexes.
164 - /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
166 Format: { 'Y' | 'N' }
168 default is 'N', setting it to 'Y' will disable disconnect
169 injection on the RPC client.
171 - /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
173 Format: { 'Y' | 'N' }
175 default is 'N', setting it to 'Y' will disable disconnect
176 injection on the RPC server.
178 - /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
180 Format: { 'Y' | 'N' }
182 default is 'N', setting it to 'Y' will disable cache wait
183 injection on the RPC server.
185 - /sys/kernel/debug/fail_function/inject:
187 Format: { 'function-name' | '!function-name' | '' }
189 specifies the target function of error injection by name.
190 If the function name leads '!' prefix, given function is
191 removed from injection list. If nothing specified ('')
192 injection list is cleared.
194 - /sys/kernel/debug/fail_function/injectable:
196 (read only) shows error injectable functions and what type of
197 error values can be specified. The error type will be one of
199 - NULL: retval must be 0.
200 - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
201 - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
203 - /sys/kernel/debug/fail_function/<function-name>/retval:
205 specifies the "error" return value to inject to the given function.
206 This will be created when the user specifies a new injection entry.
207 Note that this file only accepts unsigned values. So, if you want to
208 use a negative errno, you better use 'printf' instead of 'echo', e.g.:
209 $ printf %#x -12 > retval
214 In order to inject faults while debugfs is not available (early boot time),
215 use the boot option::
222 mmc_core.fail_request=<interval>,<probability>,<space>,<times>
227 - /proc/<pid>/fail-nth,
228 /proc/self/task/<tid>/fail-nth:
230 Write to this file of integer N makes N-th call in the task fail.
231 Read from this file returns a integer value. A value of '0' indicates
232 that the fault setup with a previous write to this file was injected.
233 A positive integer N indicates that the fault wasn't yet injected.
234 Note that this file enables all types of faults (slab, futex, etc).
235 This setting takes precedence over all other generic debugfs settings
236 like probability, interval, times, etc. But per-capability settings
237 (e.g. fail_futex/ignore-private) take precedence over it.
239 This feature is intended for systematic testing of faults in a single
240 system call. See an example below.
243 Error Injectable Functions
244 --------------------------
246 This part is for the kernel developers considering to add a function to
247 ALLOW_ERROR_INJECTION() macro.
249 Requirements for the Error Injectable Functions
250 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
252 Since the function-level error injection forcibly changes the code path
253 and returns an error even if the input and conditions are proper, this can
254 cause unexpected kernel crash if you allow error injection on the function
255 which is NOT error injectable. Thus, you (and reviewers) must ensure;
257 - The function returns an error code if it fails, and the callers must check
258 it correctly (need to recover from it).
260 - The function does not execute any code which can change any state before
261 the first error return. The state includes global or local, or input
262 variable. For example, clear output address storage (e.g. `*ret = NULL`),
263 increments/decrements counter, set a flag, preempt/irq disable or get
264 a lock (if those are recovered before returning error, that will be OK.)
266 The first requirement is important, and it will result in that the release
267 (free objects) functions are usually harder to inject errors than allocate
268 functions. If errors of such release functions are not correctly handled
269 it will cause a memory leak easily (the caller will confuse that the object
270 has been released or corrupted.)
272 The second one is for the caller which expects the function should always
273 does something. Thus if the function error injection skips whole of the
274 function, the expectation is betrayed and causes an unexpected error.
276 Type of the Error Injectable Functions
277 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
279 Each error injectable functions will have the error type specified by the
280 ALLOW_ERROR_INJECTION() macro. You have to choose it carefully if you add
281 a new error injectable function. If the wrong error type is chosen, the
282 kernel may crash because it may not be able to handle the error.
283 There are 4 types of errors defined in include/asm-generic/error-injection.h
286 This function will return `NULL` if it fails. e.g. return an allocateed
290 This function will return an `-errno` error code if it fails. e.g. return
291 -EINVAL if the input is wrong. This will include the functions which will
292 return an address which encodes `-errno` by ERR_PTR() macro.
295 This function will return an `-errno` or `NULL` if it fails. If the caller
296 of this function checks the return value with IS_ERR_OR_NULL() macro, this
297 type will be appropriate.
300 This function will return `true` (non-zero positive value) if it fails.
302 If you specifies a wrong type, for example, EI_TYPE_ERRNO for the function
303 which returns an allocated object, it may cause a problem because the returned
304 value is not an object address and the caller can not access to the address.
307 How to add new fault injection capability
308 -----------------------------------------
310 - #include <linux/fault-inject.h>
312 - define the fault attributes
314 DECLARE_FAULT_ATTR(name);
316 Please see the definition of struct fault_attr in fault-inject.h
319 - provide a way to configure fault attributes
323 If you need to enable the fault injection capability from boot time, you can
324 provide boot option to configure it. There is a helper function for it:
326 setup_fault_attr(attr, str);
330 failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
333 fault_create_debugfs_attr(name, parent, attr);
337 If the scope of the fault injection capability is limited to a
338 single kernel module, it is better to provide module parameters to
339 configure the fault attributes.
341 - add a hook to insert failures
343 Upon should_fail() returning true, client code should inject a failure:
345 should_fail(attr, size);
350 - Inject slab allocation failures into module init/exit code::
355 echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
356 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
357 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
358 echo -1 > /sys/kernel/debug/$FAILTYPE/times
359 echo 0 > /sys/kernel/debug/$FAILTYPE/space
360 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
361 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
365 bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
370 echo "Usage: $0 modulename [ modulename ... ]"
377 faulty_system modprobe $m
380 faulty_system modprobe -r $m
383 ------------------------------------------------------------------------------
385 - Inject page allocation failures only for a specific module::
389 FAILTYPE=fail_page_alloc
394 echo "Usage: $0 <modulename>"
400 if [ ! -d /sys/module/$module/sections ]
402 echo Module $module is not loaded
406 cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
407 cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
409 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
410 echo 10 > /sys/kernel/debug/$FAILTYPE/probability
411 echo 100 > /sys/kernel/debug/$FAILTYPE/interval
412 echo -1 > /sys/kernel/debug/$FAILTYPE/times
413 echo 0 > /sys/kernel/debug/$FAILTYPE/space
414 echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
415 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
416 echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
417 echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
419 trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
421 echo "Injecting errors into the module $module... (interrupt to stop)"
424 ------------------------------------------------------------------------------
426 - Inject open_ctree error while btrfs mount::
431 dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
432 DEVICE=$(losetup --show -f testfile.img)
433 mkfs.btrfs -f $DEVICE
436 FAILTYPE=fail_function
438 echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
439 printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
440 echo N > /sys/kernel/debug/$FAILTYPE/task-filter
441 echo 100 > /sys/kernel/debug/$FAILTYPE/probability
442 echo 0 > /sys/kernel/debug/$FAILTYPE/interval
443 echo -1 > /sys/kernel/debug/$FAILTYPE/times
444 echo 0 > /sys/kernel/debug/$FAILTYPE/space
445 echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
447 mount -t btrfs $DEVICE tmpmnt
456 echo > /sys/kernel/debug/$FAILTYPE/inject
463 Tool to run command with failslab or fail_page_alloc
464 ----------------------------------------------------
465 In order to make it easier to accomplish the tasks mentioned above, we can use
466 tools/testing/fault-injection/failcmd.sh. Please run a command
467 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
468 see the following examples.
472 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
475 # ./tools/testing/fault-injection/failcmd.sh \
476 -- make -C tools/testing/selftests/ run_tests
478 Same as above except to specify 100 times failures at most instead of one time
481 # ./tools/testing/fault-injection/failcmd.sh --times=100 \
482 -- make -C tools/testing/selftests/ run_tests
484 Same as above except to inject page allocation failure instead of slab
487 # env FAILCMD_TYPE=fail_page_alloc \
488 ./tools/testing/fault-injection/failcmd.sh --times=100 \
489 -- make -C tools/testing/selftests/ run_tests
491 Systematic faults using fail-nth
492 ---------------------------------
494 The following code systematically faults 0-th, 1-st, 2-nd and so on
495 capabilities in the socketpair() system call::
497 #include <sys/types.h>
498 #include <sys/stat.h>
499 #include <sys/socket.h>
500 #include <sys/syscall.h>
510 int i, err, res, fail_nth, fds[2];
513 system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
514 sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
515 fail_nth = open(buf, O_RDWR);
517 sprintf(buf, "%d", i);
518 write(fail_nth, buf, strlen(buf));
519 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
521 pread(fail_nth, buf, sizeof(buf), 0);
526 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
536 1-th fault Y: res=-1/23
537 2-th fault Y: res=-1/23
538 3-th fault Y: res=-1/12
539 4-th fault Y: res=-1/12
540 5-th fault Y: res=-1/23
541 6-th fault Y: res=-1/23
542 7-th fault Y: res=-1/23
543 8-th fault Y: res=-1/12
544 9-th fault Y: res=-1/12
545 10-th fault Y: res=-1/12
546 11-th fault Y: res=-1/12
547 12-th fault Y: res=-1/12
548 13-th fault Y: res=-1/12
549 14-th fault Y: res=-1/12
550 15-th fault Y: res=-1/12
551 16-th fault N: res=0/12