6 :Updated: Li Zefan and Tom Zanussi
11 Tracepoints (see Documentation/trace/tracepoints.rst) can be used
12 without creating custom kernel modules to register probe functions
13 using the event tracing infrastructure.
15 Not all tracepoints can be traced using the event tracing system;
16 the kernel developer must provide code snippets which define how the
17 tracing information is saved into the tracing buffer, and how the
18 tracing information should be printed.
20 2. Using Event Tracing
21 ======================
23 2.1 Via the 'set_event' interface
24 ---------------------------------
26 The events which are available for tracing can be found in the file
27 /sys/kernel/debug/tracing/available_events.
29 To enable a particular event, such as 'sched_wakeup', simply echo it
30 to /sys/kernel/debug/tracing/set_event. For example::
32 # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event
34 .. Note:: '>>' is necessary, otherwise it will firstly disable all the events.
36 To disable an event, echo the event name to the set_event file prefixed
37 with an exclamation point::
39 # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event
41 To disable all events, echo an empty line to the set_event file::
43 # echo > /sys/kernel/debug/tracing/set_event
45 To enable all events, echo ``*:*`` or ``*:`` to the set_event file::
47 # echo *:* > /sys/kernel/debug/tracing/set_event
49 The events are organized into subsystems, such as ext4, irq, sched,
50 etc., and a full event name looks like this: <subsystem>:<event>. The
51 subsystem name is optional, but it is displayed in the available_events
52 file. All of the events in a subsystem can be specified via the syntax
53 ``<subsystem>:*``; for example, to enable all irq events, you can use the
56 # echo 'irq:*' > /sys/kernel/debug/tracing/set_event
58 2.2 Via the 'enable' toggle
59 ---------------------------
61 The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy
64 To enable event 'sched_wakeup'::
66 # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
70 # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable
72 To enable all events in sched subsystem::
74 # echo 1 > /sys/kernel/debug/tracing/events/sched/enable
76 To enable all events::
78 # echo 1 > /sys/kernel/debug/tracing/events/enable
80 When reading one of these enable files, there are four results:
82 - 0 - all events this file affects are disabled
83 - 1 - all events this file affects are enabled
84 - X - there is a mixture of events enabled and disabled
85 - ? - this file does not affect any event
90 In order to facilitate early boot debugging, use boot option::
92 trace_event=[event-list]
94 event-list is a comma separated list of events. See section 2.1 for event
97 3. Defining an event-enabled tracepoint
98 =======================================
100 See The example provided in samples/trace_events
105 Each trace event has a 'format' file associated with it that contains
106 a description of each field in a logged event. This information can
107 be used to parse the binary trace stream, and is also the place to
108 find the field names that can be used in event filters (see section 5).
110 It also displays the format string that will be used to print the
111 event in text mode, along with the event name and ID used for
114 Every event has a set of ``common`` fields associated with it; these are
115 the fields prefixed with ``common_``. The other fields vary between
116 events and correspond to the fields defined in the TRACE_EVENT
117 definition for that event.
119 Each field in the format has the form::
121 field:field-type field-name; offset:N; size:N;
123 where offset is the offset of the field in the trace record and size
124 is the size of the data item, in bytes.
126 For example, here's the information displayed for the 'sched_wakeup'
129 # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format
134 field:unsigned short common_type; offset:0; size:2;
135 field:unsigned char common_flags; offset:2; size:1;
136 field:unsigned char common_preempt_count; offset:3; size:1;
137 field:int common_pid; offset:4; size:4;
138 field:int common_tgid; offset:8; size:4;
140 field:char comm[TASK_COMM_LEN]; offset:12; size:16;
141 field:pid_t pid; offset:28; size:4;
142 field:int prio; offset:32; size:4;
143 field:int success; offset:36; size:4;
144 field:int cpu; offset:40; size:4;
146 print fmt: "task %s:%d [%d] success=%d [%03d]", REC->comm, REC->pid,
147 REC->prio, REC->success, REC->cpu
149 This event contains 10 fields, the first 5 common and the remaining 5
150 event-specific. All the fields for this event are numeric, except for
151 'comm' which is a string, a distinction important for event filtering.
156 Trace events can be filtered in the kernel by associating boolean
157 'filter expressions' with them. As soon as an event is logged into
158 the trace buffer, its fields are checked against the filter expression
159 associated with that event type. An event with field values that
160 'match' the filter will appear in the trace output, and an event whose
161 values don't match will be discarded. An event with no filter
162 associated with it matches everything, and is the default when no
163 filter has been set for an event.
165 5.1 Expression syntax
166 ---------------------
168 A filter expression consists of one or more 'predicates' that can be
169 combined using the logical operators '&&' and '||'. A predicate is
170 simply a clause that compares the value of a field contained within a
171 logged event with a constant value and returns either 0 or 1 depending
172 on whether the field value matched (1) or didn't match (0)::
174 field-name relational-operator value
176 Parentheses can be used to provide arbitrary logical groupings and
177 double-quotes can be used to prevent the shell from interpreting
178 operators as shell metacharacters.
180 The field-names available for use in filters can be found in the
181 'format' files for trace events (see section 4).
183 The relational-operators depend on the type of the field being tested:
185 The operators available for numeric fields are:
187 ==, !=, <, <=, >, >=, &
189 And for string fields they are:
193 The glob (~) accepts a wild card character (\*,?) and character classes
201 If the field is a pointer that points into user space (for example
202 "filename" from sys_enter_openat), then you have to append ".ustring" to the
205 filename.ustring ~ "password"
207 As the kernel will have to know how to retrieve the memory that the pointer
208 is at from user space.
213 A filter for an individual event is set by writing a filter expression
214 to the 'filter' file for the given event.
218 # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup
219 # echo "common_preempt_count > 4" > filter
221 A slightly more involved example::
223 # cd /sys/kernel/debug/tracing/events/signal/signal_generate
224 # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter
226 If there is an error in the expression, you'll get an 'Invalid
227 argument' error when setting it, and the erroneous string along with
228 an error message can be seen by looking at the filter e.g.::
230 # cd /sys/kernel/debug/tracing/events/signal/signal_generate
231 # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter
232 -bash: echo: write error: Invalid argument
234 ((sig >= 10 && sig < 15) || dsig == 17) && comm != bash
236 parse_error: Field not found
238 Currently the caret ('^') for an error always appears at the beginning of
239 the filter string; the error message should still be useful though
240 even without more accurate position info.
242 5.2.1 Filter limitations
243 ------------------------
245 If a filter is placed on a string pointer ``(char *)`` that does not point
246 to a string on the ring buffer, but instead points to kernel or user space
247 memory, then, for safety reasons, at most 1024 bytes of the content is
248 copied onto a temporary buffer to do the compare. If the copy of the memory
249 faults (the pointer points to memory that should not be accessed), then the
250 string compare will be treated as not matching.
255 To clear the filter for an event, write a '0' to the event's filter
258 To clear the filters for all events in a subsystem, write a '0' to the
259 subsystem's filter file.
261 5.3 Subsystem filters
262 ---------------------
264 For convenience, filters for every event in a subsystem can be set or
265 cleared as a group by writing a filter expression into the filter file
266 at the root of the subsystem. Note however, that if a filter for any
267 event within the subsystem lacks a field specified in the subsystem
268 filter, or if the filter can't be applied for any other reason, the
269 filter for that event will retain its previous setting. This can
270 result in an unintended mixture of filters which could lead to
271 confusing (to the user who might think different filters are in
272 effect) trace output. Only filters that reference just the common
273 fields can be guaranteed to propagate successfully to all events.
275 Here are a few subsystem filter examples that also illustrate the
278 Clear the filters on all events in the sched subsystem::
280 # cd /sys/kernel/debug/tracing/events/sched
282 # cat sched_switch/filter
284 # cat sched_wakeup/filter
287 Set a filter using only common fields for all events in the sched
288 subsystem (all events end up with the same filter)::
290 # cd /sys/kernel/debug/tracing/events/sched
291 # echo common_pid == 0 > filter
292 # cat sched_switch/filter
294 # cat sched_wakeup/filter
297 Attempt to set a filter using a non-common field for all events in the
298 sched subsystem (all events but those that have a prev_pid field retain
301 # cd /sys/kernel/debug/tracing/events/sched
302 # echo prev_pid == 0 > filter
303 # cat sched_switch/filter
305 # cat sched_wakeup/filter
311 The set_event_pid file in the same directory as the top events directory
312 exists, will filter all events from tracing any task that does not have the
313 PID listed in the set_event_pid file.
316 # cd /sys/kernel/debug/tracing
317 # echo $$ > set_event_pid
318 # echo 1 > events/enable
320 Will only trace events for the current task.
322 To add more PIDs without losing the PIDs already included, use '>>'.
325 # echo 123 244 1 >> set_event_pid
331 Trace events can be made to conditionally invoke trigger 'commands'
332 which can take various forms and are described in detail below;
333 examples would be enabling or disabling other trace events or invoking
334 a stack trace whenever the trace event is hit. Whenever a trace event
335 with attached triggers is invoked, the set of trigger commands
336 associated with that event is invoked. Any given trigger can
337 additionally have an event filter of the same form as described in
338 section 5 (Event filtering) associated with it - the command will only
339 be invoked if the event being invoked passes the associated filter.
340 If no filter is associated with the trigger, it always passes.
342 Triggers are added to and removed from a particular event by writing
343 trigger expressions to the 'trigger' file for the given event.
345 A given event can have any number of triggers associated with it,
346 subject to any restrictions that individual commands may have in that
349 Event triggers are implemented on top of "soft" mode, which means that
350 whenever a trace event has one or more triggers associated with it,
351 the event is activated even if it isn't actually enabled, but is
352 disabled in a "soft" mode. That is, the tracepoint will be called,
353 but just will not be traced, unless of course it's actually enabled.
354 This scheme allows triggers to be invoked even for events that aren't
355 enabled, and also allows the current event filter implementation to be
356 used for conditionally invoking triggers.
358 The syntax for event triggers is roughly based on the syntax for
359 set_ftrace_filter 'ftrace filter commands' (see the 'Filter commands'
360 section of Documentation/trace/ftrace.rst), but there are major
361 differences and the implementation isn't currently tied to it in any
362 way, so beware about making generalizations between the two.
365 Writing into trace_marker (See Documentation/trace/ftrace.rst)
366 can also enable triggers that are written into
367 /sys/kernel/tracing/events/ftrace/print/trigger
369 6.1 Expression syntax
370 ---------------------
372 Triggers are added by echoing the command to the 'trigger' file::
374 # echo 'command[:count] [if filter]' > trigger
376 Triggers are removed by echoing the same command but starting with '!'
377 to the 'trigger' file::
379 # echo '!command[:count] [if filter]' > trigger
381 The [if filter] part isn't used in matching commands when removing, so
382 leaving that off in a '!' command will accomplish the same thing as
385 The filter syntax is the same as that described in the 'Event
386 filtering' section above.
388 For ease of use, writing to the trigger file using '>' currently just
389 adds or removes a single trigger and there's no explicit '>>' support
390 ('>' actually behaves like '>>') or truncation support to remove all
391 triggers (you have to use '!' for each one added.)
393 6.2 Supported trigger commands
394 ------------------------------
396 The following commands are supported:
398 - enable_event/disable_event
400 These commands can enable or disable another trace event whenever
401 the triggering event is hit. When these commands are registered,
402 the other trace event is activated, but disabled in a "soft" mode.
403 That is, the tracepoint will be called, but just will not be traced.
404 The event tracepoint stays in this mode as long as there's a trigger
405 in effect that can trigger it.
407 For example, the following trigger causes kmalloc events to be
408 traced when a read system call is entered, and the :1 at the end
409 specifies that this enablement happens only once::
411 # echo 'enable_event:kmem:kmalloc:1' > \
412 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
414 The following trigger causes kmalloc events to stop being traced
415 when a read system call exits. This disablement happens on every
416 read system call exit::
418 # echo 'disable_event:kmem:kmalloc' > \
419 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
423 enable_event:<system>:<event>[:count]
424 disable_event:<system>:<event>[:count]
426 To remove the above commands::
428 # echo '!enable_event:kmem:kmalloc:1' > \
429 /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger
431 # echo '!disable_event:kmem:kmalloc' > \
432 /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger
434 Note that there can be any number of enable/disable_event triggers
435 per triggering event, but there can only be one trigger per
436 triggered event. e.g. sys_enter_read can have triggers enabling both
437 kmem:kmalloc and sched:sched_switch, but can't have two kmem:kmalloc
438 versions such as kmem:kmalloc and kmem:kmalloc:1 or 'kmem:kmalloc if
439 bytes_req == 256' and 'kmem:kmalloc if bytes_alloc == 256' (they
440 could be combined into a single filter on kmem:kmalloc though).
444 This command dumps a stacktrace in the trace buffer whenever the
445 triggering event occurs.
447 For example, the following trigger dumps a stacktrace every time the
448 kmalloc tracepoint is hit::
450 # echo 'stacktrace' > \
451 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
453 The following trigger dumps a stacktrace the first 5 times a kmalloc
454 request happens with a size >= 64K::
456 # echo 'stacktrace:5 if bytes_req >= 65536' > \
457 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
463 To remove the above commands::
465 # echo '!stacktrace' > \
466 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
468 # echo '!stacktrace:5 if bytes_req >= 65536' > \
469 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
471 The latter can also be removed more simply by the following (without
474 # echo '!stacktrace:5' > \
475 /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger
477 Note that there can be only one stacktrace trigger per triggering
482 This command causes a snapshot to be triggered whenever the
483 triggering event occurs.
485 The following command creates a snapshot every time a block request
486 queue is unplugged with a depth > 1. If you were tracing a set of
487 events or functions at the time, the snapshot trace buffer would
488 capture those events when the trigger event occurred::
490 # echo 'snapshot if nr_rq > 1' > \
491 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
493 To only snapshot once::
495 # echo 'snapshot:1 if nr_rq > 1' > \
496 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
498 To remove the above commands::
500 # echo '!snapshot if nr_rq > 1' > \
501 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
503 # echo '!snapshot:1 if nr_rq > 1' > \
504 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
506 Note that there can be only one snapshot trigger per triggering
511 These commands turn tracing on and off when the specified events are
512 hit. The parameter determines how many times the tracing system is
513 turned on and off. If unspecified, there is no limit.
515 The following command turns tracing off the first time a block
516 request queue is unplugged with a depth > 1. If you were tracing a
517 set of events or functions at the time, you could then examine the
518 trace buffer to see the sequence of events that led up to the
521 # echo 'traceoff:1 if nr_rq > 1' > \
522 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
524 To always disable tracing when nr_rq > 1::
526 # echo 'traceoff if nr_rq > 1' > \
527 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
529 To remove the above commands::
531 # echo '!traceoff:1 if nr_rq > 1' > \
532 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
534 # echo '!traceoff if nr_rq > 1' > \
535 /sys/kernel/debug/tracing/events/block/block_unplug/trigger
537 Note that there can be only one traceon or traceoff trigger per
542 This command aggregates event hits into a hash table keyed on one or
543 more trace event format fields (or stacktrace) and a set of running
544 totals derived from one or more trace event format fields and/or
545 event counts (hitcount).
547 See Documentation/trace/histogram.rst for details and examples.
549 7. In-kernel trace event API
550 ============================
552 In most cases, the command-line interface to trace events is more than
553 sufficient. Sometimes, however, applications might find the need for
554 more complex relationships than can be expressed through a simple
555 series of linked command-line expressions, or putting together sets of
556 commands may be simply too cumbersome. An example might be an
557 application that needs to 'listen' to the trace stream in order to
558 maintain an in-kernel state machine detecting, for instance, when an
559 illegal kernel state occurs in the scheduler.
561 The trace event subsystem provides an in-kernel API allowing modules
562 or other kernel code to generate user-defined 'synthetic' events at
563 will, which can be used to either augment the existing trace stream
564 and/or signal that a particular important state has occurred.
566 A similar in-kernel API is also available for creating kprobe and
569 Both the synthetic event and k/ret/probe event APIs are built on top
570 of a lower-level "dynevent_cmd" event command API, which is also
571 available for more specialized applications, or as the basis of other
572 higher-level trace event APIs.
574 The API provided for these purposes is describe below and allows the
577 - dynamically creating synthetic event definitions
578 - dynamically creating kprobe and kretprobe event definitions
579 - tracing synthetic events from in-kernel code
580 - the low-level "dynevent_cmd" API
582 7.1 Dyamically creating synthetic event definitions
583 ---------------------------------------------------
585 There are a couple ways to create a new synthetic event from a kernel
586 module or other kernel code.
588 The first creates the event in one step, using synth_event_create().
589 In this method, the name of the event to create and an array defining
590 the fields is supplied to synth_event_create(). If successful, a
591 synthetic event with that name and fields will exist following that
592 call. For example, to create a new "schedtest" synthetic event::
594 ret = synth_event_create("schedtest", sched_fields,
595 ARRAY_SIZE(sched_fields), THIS_MODULE);
597 The sched_fields param in this example points to an array of struct
598 synth_field_desc, each of which describes an event field by type and
601 static struct synth_field_desc sched_fields[] = {
602 { .type = "pid_t", .name = "next_pid_field" },
603 { .type = "char[16]", .name = "next_comm_field" },
604 { .type = "u64", .name = "ts_ns" },
605 { .type = "u64", .name = "ts_ms" },
606 { .type = "unsigned int", .name = "cpu" },
607 { .type = "char[64]", .name = "my_string_field" },
608 { .type = "int", .name = "my_int_field" },
611 See synth_field_size() for available types.
613 If field_name contains [n], the field is considered to be a static array.
615 If field_names contains[] (no subscript), the field is considered to
616 be a dynamic array, which will only take as much space in the event as
617 is required to hold the array.
619 Because space for an event is reserved before assigning field values
620 to the event, using dynamic arrays implies that the piecewise
621 in-kernel API described below can't be used with dynamic arrays. The
622 other non-piecewise in-kernel APIs can, however, be used with dynamic
625 If the event is created from within a module, a pointer to the module
626 must be passed to synth_event_create(). This will ensure that the
627 trace buffer won't contain unreadable events when the module is
630 At this point, the event object is ready to be used for generating new
633 In the second method, the event is created in several steps. This
634 allows events to be created dynamically and without the need to create
635 and populate an array of fields beforehand.
637 To use this method, an empty or partially empty synthetic event should
638 first be created using synth_event_gen_cmd_start() or
639 synth_event_gen_cmd_array_start(). For synth_event_gen_cmd_start(),
640 the name of the event along with one or more pairs of args each pair
641 representing a 'type field_name;' field specification should be
642 supplied. For synth_event_gen_cmd_array_start(), the name of the
643 event along with an array of struct synth_field_desc should be
644 supplied. Before calling synth_event_gen_cmd_start() or
645 synth_event_gen_cmd_array_start(), the user should create and
646 initialize a dynevent_cmd object using synth_event_cmd_init().
648 For example, to create a new "schedtest" synthetic event with two
651 struct dynevent_cmd cmd;
654 /* Create a buffer to hold the generated command */
655 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
657 /* Before generating the command, initialize the cmd object */
658 synth_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
660 ret = synth_event_gen_cmd_start(&cmd, "schedtest", THIS_MODULE,
661 "pid_t", "next_pid_field",
664 Alternatively, using an array of struct synth_field_desc fields
665 containing the same information::
667 ret = synth_event_gen_cmd_array_start(&cmd, "schedtest", THIS_MODULE,
670 Once the synthetic event object has been created, it can then be
671 populated with more fields. Fields are added one by one using
672 synth_event_add_field(), supplying the dynevent_cmd object, a field
673 type, and a field name. For example, to add a new int field named
674 "intfield", the following call should be made::
676 ret = synth_event_add_field(&cmd, "int", "intfield");
678 See synth_field_size() for available types. If field_name contains [n]
679 the field is considered to be an array.
681 A group of fields can also be added all at once using an array of
682 synth_field_desc with add_synth_fields(). For example, this would add
683 just the first four sched_fields::
685 ret = synth_event_add_fields(&cmd, sched_fields, 4);
687 If you already have a string of the form 'type field_name',
688 synth_event_add_field_str() can be used to add it as-is; it will
689 also automatically append a ';' to the string.
691 Once all the fields have been added, the event should be finalized and
692 registered by calling the synth_event_gen_cmd_end() function::
694 ret = synth_event_gen_cmd_end(&cmd);
696 At this point, the event object is ready to be used for tracing new
699 7.2 Tracing synthetic events from in-kernel code
700 ------------------------------------------------
702 To trace a synthetic event, there are several options. The first
703 option is to trace the event in one call, using synth_event_trace()
704 with a variable number of values, or synth_event_trace_array() with an
705 array of values to be set. A second option can be used to avoid the
706 need for a pre-formed array of values or list of arguments, via
707 synth_event_trace_start() and synth_event_trace_end() along with
708 synth_event_add_next_val() or synth_event_add_val() to add the values
711 7.2.1 Tracing a synthetic event all at once
712 -------------------------------------------
714 To trace a synthetic event all at once, the synth_event_trace() or
715 synth_event_trace_array() functions can be used.
717 The synth_event_trace() function is passed the trace_event_file
718 representing the synthetic event (which can be retrieved using
719 trace_get_event_file() using the synthetic event name, "synthetic" as
720 the system name, and the trace instance name (NULL if using the global
721 trace array)), along with an variable number of u64 args, one for each
722 synthetic event field, and the number of values being passed.
724 So, to trace an event corresponding to the synthetic event definition
725 above, code like the following could be used::
727 ret = synth_event_trace(create_synth_test, 7, /* number of values */
728 444, /* next_pid_field */
729 (u64)"clackers", /* next_comm_field */
732 smp_processor_id(),/* cpu */
733 (u64)"Thneed", /* my_string_field */
734 999); /* my_int_field */
736 All vals should be cast to u64, and string vals are just pointers to
737 strings, cast to u64. Strings will be copied into space reserved in
738 the event for the string, using these pointers.
740 Alternatively, the synth_event_trace_array() function can be used to
741 accomplish the same thing. It is passed the trace_event_file
742 representing the synthetic event (which can be retrieved using
743 trace_get_event_file() using the synthetic event name, "synthetic" as
744 the system name, and the trace instance name (NULL if using the global
745 trace array)), along with an array of u64, one for each synthetic
748 To trace an event corresponding to the synthetic event definition
749 above, code like the following could be used::
753 vals[0] = 777; /* next_pid_field */
754 vals[1] = (u64)"tiddlywinks"; /* next_comm_field */
755 vals[2] = 1000000; /* ts_ns */
756 vals[3] = 1000; /* ts_ms */
757 vals[4] = smp_processor_id(); /* cpu */
758 vals[5] = (u64)"thneed"; /* my_string_field */
759 vals[6] = 398; /* my_int_field */
761 The 'vals' array is just an array of u64, the number of which must
762 match the number of field in the synthetic event, and which must be in
763 the same order as the synthetic event fields.
765 All vals should be cast to u64, and string vals are just pointers to
766 strings, cast to u64. Strings will be copied into space reserved in
767 the event for the string, using these pointers.
769 In order to trace a synthetic event, a pointer to the trace event file
770 is needed. The trace_get_event_file() function can be used to get
771 it - it will find the file in the given trace instance (in this case
772 NULL since the top trace array is being used) while at the same time
773 preventing the instance containing it from going away::
775 schedtest_event_file = trace_get_event_file(NULL, "synthetic",
778 Before tracing the event, it should be enabled in some way, otherwise
779 the synthetic event won't actually show up in the trace buffer.
781 To enable a synthetic event from the kernel, trace_array_set_clr_event()
782 can be used (which is not specific to synthetic events, so does need
783 the "synthetic" system name to be specified explicitly).
785 To enable the event, pass 'true' to it::
787 trace_array_set_clr_event(schedtest_event_file->tr,
788 "synthetic", "schedtest", true);
790 To disable it pass false::
792 trace_array_set_clr_event(schedtest_event_file->tr,
793 "synthetic", "schedtest", false);
795 Finally, synth_event_trace_array() can be used to actually trace the
796 event, which should be visible in the trace buffer afterwards::
798 ret = synth_event_trace_array(schedtest_event_file, vals,
801 To remove the synthetic event, the event should be disabled, and the
802 trace instance should be 'put' back using trace_put_event_file()::
804 trace_array_set_clr_event(schedtest_event_file->tr,
805 "synthetic", "schedtest", false);
806 trace_put_event_file(schedtest_event_file);
808 If those have been successful, synth_event_delete() can be called to
811 ret = synth_event_delete("schedtest");
813 7.2.2 Tracing a synthetic event piecewise
814 -----------------------------------------
816 To trace a synthetic using the piecewise method described above, the
817 synth_event_trace_start() function is used to 'open' the synthetic
820 struct synth_trace_state trace_state;
822 ret = synth_event_trace_start(schedtest_event_file, &trace_state);
824 It's passed the trace_event_file representing the synthetic event
825 using the same methods as described above, along with a pointer to a
826 struct synth_trace_state object, which will be zeroed before use and
827 used to maintain state between this and following calls.
829 Once the event has been opened, which means space for it has been
830 reserved in the trace buffer, the individual fields can be set. There
831 are two ways to do that, either one after another for each field in
832 the event, which requires no lookups, or by name, which does. The
833 tradeoff is flexibility in doing the assignments vs the cost of a
836 To assign the values one after the other without lookups,
837 synth_event_add_next_val() should be used. Each call is passed the
838 same synth_trace_state object used in the synth_event_trace_start(),
839 along with the value to set the next field in the event. After each
840 field is set, the 'cursor' points to the next field, which will be set
841 by the subsequent call, continuing until all the fields have been set
842 in order. The same sequence of calls as in the above examples using
843 this method would be (without error-handling code)::
846 ret = synth_event_add_next_val(777, &trace_state);
848 /* next_comm_field */
849 ret = synth_event_add_next_val((u64)"slinky", &trace_state);
852 ret = synth_event_add_next_val(1000000, &trace_state);
855 ret = synth_event_add_next_val(1000, &trace_state);
858 ret = synth_event_add_next_val(smp_processor_id(), &trace_state);
860 /* my_string_field */
861 ret = synth_event_add_next_val((u64)"thneed_2.01", &trace_state);
864 ret = synth_event_add_next_val(395, &trace_state);
866 To assign the values in any order, synth_event_add_val() should be
867 used. Each call is passed the same synth_trace_state object used in
868 the synth_event_trace_start(), along with the field name of the field
869 to set and the value to set it to. The same sequence of calls as in
870 the above examples using this method would be (without error-handling
873 ret = synth_event_add_val("next_pid_field", 777, &trace_state);
874 ret = synth_event_add_val("next_comm_field", (u64)"silly putty",
876 ret = synth_event_add_val("ts_ns", 1000000, &trace_state);
877 ret = synth_event_add_val("ts_ms", 1000, &trace_state);
878 ret = synth_event_add_val("cpu", smp_processor_id(), &trace_state);
879 ret = synth_event_add_val("my_string_field", (u64)"thneed_9",
881 ret = synth_event_add_val("my_int_field", 3999, &trace_state);
883 Note that synth_event_add_next_val() and synth_event_add_val() are
884 incompatible if used within the same trace of an event - either one
885 can be used but not both at the same time.
887 Finally, the event won't be actually traced until it's 'closed',
888 which is done using synth_event_trace_end(), which takes only the
889 struct synth_trace_state object used in the previous calls::
891 ret = synth_event_trace_end(&trace_state);
893 Note that synth_event_trace_end() must be called at the end regardless
894 of whether any of the add calls failed (say due to a bad field name
897 7.3 Dyamically creating kprobe and kretprobe event definitions
898 --------------------------------------------------------------
900 To create a kprobe or kretprobe trace event from kernel code, the
901 kprobe_event_gen_cmd_start() or kretprobe_event_gen_cmd_start()
902 functions can be used.
904 To create a kprobe event, an empty or partially empty kprobe event
905 should first be created using kprobe_event_gen_cmd_start(). The name
906 of the event and the probe location should be specfied along with one
907 or args each representing a probe field should be supplied to this
908 function. Before calling kprobe_event_gen_cmd_start(), the user
909 should create and initialize a dynevent_cmd object using
910 kprobe_event_cmd_init().
912 For example, to create a new "schedtest" kprobe event with two fields::
914 struct dynevent_cmd cmd;
917 /* Create a buffer to hold the generated command */
918 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
920 /* Before generating the command, initialize the cmd object */
921 kprobe_event_cmd_init(&cmd, buf, MAX_DYNEVENT_CMD_LEN);
924 * Define the gen_kprobe_test event with the first 2 kprobe
927 ret = kprobe_event_gen_cmd_start(&cmd, "gen_kprobe_test", "do_sys_open",
928 "dfd=%ax", "filename=%dx");
930 Once the kprobe event object has been created, it can then be
931 populated with more fields. Fields can be added using
932 kprobe_event_add_fields(), supplying the dynevent_cmd object along
933 with a variable arg list of probe fields. For example, to add a
934 couple additional fields, the following call could be made::
936 ret = kprobe_event_add_fields(&cmd, "flags=%cx", "mode=+4($stack)");
938 Once all the fields have been added, the event should be finalized and
939 registered by calling the kprobe_event_gen_cmd_end() or
940 kretprobe_event_gen_cmd_end() functions, depending on whether a kprobe
941 or kretprobe command was started::
943 ret = kprobe_event_gen_cmd_end(&cmd);
947 ret = kretprobe_event_gen_cmd_end(&cmd);
949 At this point, the event object is ready to be used for tracing new
952 Similarly, a kretprobe event can be created using
953 kretprobe_event_gen_cmd_start() with a probe name and location and
954 additional params such as $retval::
956 ret = kretprobe_event_gen_cmd_start(&cmd, "gen_kretprobe_test",
957 "do_sys_open", "$retval");
959 Similar to the synthetic event case, code like the following can be
960 used to enable the newly created kprobe event::
962 gen_kprobe_test = trace_get_event_file(NULL, "kprobes", "gen_kprobe_test");
964 ret = trace_array_set_clr_event(gen_kprobe_test->tr,
965 "kprobes", "gen_kprobe_test", true);
967 Finally, also similar to synthetic events, the following code can be
968 used to give the kprobe event file back and delete the event::
970 trace_put_event_file(gen_kprobe_test);
972 ret = kprobe_event_delete("gen_kprobe_test");
974 7.4 The "dynevent_cmd" low-level API
975 ------------------------------------
977 Both the in-kernel synthetic event and kprobe interfaces are built on
978 top of a lower-level "dynevent_cmd" interface. This interface is
979 meant to provide the basis for higher-level interfaces such as the
980 synthetic and kprobe interfaces, which can be used as examples.
982 The basic idea is simple and amounts to providing a general-purpose
983 layer that can be used to generate trace event commands. The
984 generated command strings can then be passed to the command-parsing
985 and event creation code that already exists in the trace event
986 subystem for creating the corresponding trace events.
988 In a nutshell, the way it works is that the higher-level interface
989 code creates a struct dynevent_cmd object, then uses a couple
990 functions, dynevent_arg_add() and dynevent_arg_pair_add() to build up
991 a command string, which finally causes the command to be executed
992 using the dynevent_create() function. The details of the interface
995 The first step in building a new command string is to create and
996 initialize an instance of a dynevent_cmd. Here, for instance, we
997 create a dynevent_cmd on the stack and initialize it::
999 struct dynevent_cmd cmd;
1003 buf = kzalloc(MAX_DYNEVENT_CMD_LEN, GFP_KERNEL);
1005 dynevent_cmd_init(cmd, buf, maxlen, DYNEVENT_TYPE_FOO,
1006 foo_event_run_command);
1008 The dynevent_cmd initialization needs to be given a user-specified
1009 buffer and the length of the buffer (MAX_DYNEVENT_CMD_LEN can be used
1010 for this purpose - at 2k it's generally too big to be comfortably put
1011 on the stack, so is dynamically allocated), a dynevent type id, which
1012 is meant to be used to check that further API calls are for the
1013 correct command type, and a pointer to an event-specific run_command()
1014 callback that will be called to actually execute the event-specific
1017 Once that's done, the command string can by built up by successive
1018 calls to argument-adding functions.
1020 To add a single argument, define and initialize a struct dynevent_arg
1021 or struct dynevent_arg_pair object. Here's an example of the simplest
1022 possible arg addition, which is simply to append the given string as
1023 a whitespace-separated argument to the command::
1025 struct dynevent_arg arg;
1027 dynevent_arg_init(&arg, NULL, 0);
1031 ret = dynevent_arg_add(cmd, &arg);
1033 The arg object is first initialized using dynevent_arg_init() and in
1034 this case the parameters are NULL or 0, which means there's no
1035 optional sanity-checking function or separator appended to the end of
1038 Here's another more complicated example using an 'arg pair', which is
1039 used to create an argument that consists of a couple components added
1040 together as a unit, for example, a 'type field_name;' arg or a simple
1041 expression arg e.g. 'flags=%cx'::
1043 struct dynevent_arg_pair arg_pair;
1045 dynevent_arg_pair_init(&arg_pair, dynevent_foo_check_arg_fn, 0, ';');
1047 arg_pair.lhs = type;
1048 arg_pair.rhs = name;
1050 ret = dynevent_arg_pair_add(cmd, &arg_pair);
1052 Again, the arg_pair is first initialized, in this case with a callback
1053 function used to check the sanity of the args (for example, that
1054 neither part of the pair is NULL), along with a character to be used
1055 to add an operator between the pair (here none) and a separator to be
1056 appended onto the end of the arg pair (here ';').
1058 There's also a dynevent_str_add() function that can be used to simply
1059 add a string as-is, with no spaces, delimeters, or arg check.
1061 Any number of dynevent_*_add() calls can be made to build up the string
1062 (until its length surpasses cmd->maxlen). When all the arguments have
1063 been added and the command string is complete, the only thing left to
1064 do is run the command, which happens by simply calling
1067 ret = dynevent_create(&cmd);
1069 At that point, if the return value is 0, the dynamic event has been
1070 created and is ready to use.
1072 See the dynevent_cmd function definitions themselves for the details