Documentation/networking/devlink/devlink-health.rst

   1 .. SPDX-License-Identifier: GPL-2.0
   2
   3 ==============
   4 Devlink Health
   5 ==============
   6
   7 Background
   8 ==========
   9
  10 The ``devlink`` health mechanism is targeted for Real Time Alerting, in
  11 order to know when something bad happened to a PCI device.
  12
  13   * Provide alert debug information.
  14   * Self healing.
  15   * If problem needs vendor support, provide a way to gather all needed
  16     debugging information.
  17
  18 Overview
  19 ========
  20
  21 The main idea is to unify and centralize driver health reports in the
  22 generic ``devlink`` instance and allow the user to set different
  23 attributes of the health reporting and recovery procedures.
  24
  25 The ``devlink`` health reporter:
  26 Device driver creates a "health reporter" per each error/health type.
  27 Error/Health type can be a known/generic (e.g. PCI error, fw error, rx/tx error)
  28 or unknown (driver specific).
  29 For each registered health reporter a driver can issue error/health reports
  30 asynchronously. All health reports handling is done by ``devlink``.
  31 Device driver can provide specific callbacks for each "health reporter", e.g.:
  32
  33   * Recovery procedures
  34   * Diagnostics procedures
  35   * Object dump procedures
  36   * Out Of Box initial parameters
  37
  38 Different parts of the driver can register different types of health reporters
  39 with different handlers.
  40
  41 Actions
  42 =======
  43
  44 Once an error is reported, devlink health will perform the following actions:
  45
  46   * A log is being send to the kernel trace events buffer
  47   * Health status and statistics are being updated for the reporter instance
  48   * Object dump is being taken and saved at the reporter instance (as long as
  49     auto-dump is set and there is no other dump which is already stored)
  50   * Auto recovery attempt is being done. Depends on:
  51
  52     - Auto-recovery configuration
  53     - Grace period vs. time passed since last recover
  54
  55 Devlink formatted message
  56 =========================
  57
  58 To handle devlink health diagnose and health dump requests, devlink creates a
  59 formatted message structure ``devlink_fmsg`` and send it to the driver's callback
  60 to fill the data in using the devlink fmsg API.
  61
  62 Devlink fmsg is a mechanism to pass descriptors between drivers and devlink, in
  63 json-like format. The API allows the driver to add nested attributes such as
  64 object, object pair and value array, in addition to attributes such as name and
  65 value.
  66
  67 Driver should use this API to fill the fmsg context in a format which will be
  68 translated by the devlink to the netlink message later. When it needs to send
  69 the data using SKBs to the netlink layer, it fragments the data between
  70 different SKBs. In order to do this fragmentation, it uses virtual nests
  71 attributes, to avoid actual nesting use which cannot be divided between
  72 different SKBs.
  73
  74 User Interface
  75 ==============
  76
  77 User can access/change each reporter's parameters and driver specific callbacks
  78 via ``devlink``, e.g per error type (per health reporter):
  79
  80   * Configure reporter's generic parameters (like: disable/enable auto recovery)
  81   * Invoke recovery procedure
  82   * Run diagnostics
  83   * Object dump
  84
  85 .. list-table:: List of devlink health interfaces
  86    :widths: 10 90
  87
  88    * - Name
  89      - Description
  90    * - ``DEVLINK_CMD_HEALTH_REPORTER_GET``
  91      - Retrieves status and configuration info per DEV and reporter.
  92    * - ``DEVLINK_CMD_HEALTH_REPORTER_SET``
  93      - Allows reporter-related configuration setting.
  94    * - ``DEVLINK_CMD_HEALTH_REPORTER_RECOVER``
  95      - Triggers reporter's recovery procedure.
  96    * - ``DEVLINK_CMD_HEALTH_REPORTER_TEST``
  97      - Triggers a fake health event on the reporter. The effects of the test
  98        event in terms of recovery flow should follow closely that of a real
  99        event.
 100    * - ``DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE``
 101      - Retrieves current device state related to the reporter.
 102    * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET``
 103      - Retrieves the last stored dump. Devlink health
 104        saves a single dump. If an dump is not already stored by devlink
 105        for this reporter, devlink generates a new dump.
 106        Dump output is defined by the reporter.
 107    * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR``
 108      - Clears the last saved dump file for the specified reporter.
 109
 110 The following diagram provides a general overview of ``devlink-health``::
 111
 112                                                    netlink
 113                                           +--------------------------+
 114                                           |                          |
 115                                           |            +             |
 116                                           |            |             |
 117                                           +--------------------------+
 118                                                        |request for ops
 119                                                        |(diagnose,
 120       driver                               devlink     |recover,
 121                                                        |dump)
 122     +--------+                            +--------------------------+
 123     |        |                            |    reporter|             |
 124     |        |                            |  +---------v----------+  |
 125     |        |   ops execution            |  |                    |  |
 126     |     <----------------------------------+                    |  |
 127     |        |                            |  |                    |  |
 128     |        |                            |  + ^------------------+  |
 129     |        |                            |    | request for ops     |
 130     |        |                            |    | (recover, dump)     |
 131     |        |                            |    |                     |
 132     |        |                            |  +-+------------------+  |
 133     |        |     health report          |  | health handler     |  |
 134     |        +------------------------------->                    |  |
 135     |        |                            |  +--------------------+  |
 136     |        |     health reporter create |                          |
 137     |        +---------------------------->                          |
 138     +--------+                            +--------------------------+