1 .. SPDX-License-Identifier: GPL-2.0
3 ============================================================
4 Linux kernel driver for Elastic Network Adapter (ENA) family
5 ============================================================
10 ENA is a networking interface designed to make good use of modern CPU
11 features and system architectures.
13 The ENA device exposes a lightweight management interface with a
14 minimal set of memory mapped registers and extendible command set
15 through an Admin Queue.
17 The driver supports a range of ENA devices, is link-speed independent
18 (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc), and has
19 a negotiated and extendible feature set.
21 Some ENA devices support SR-IOV. This driver is used for both the
22 SR-IOV Physical Function (PF) and Virtual Function (VF) devices.
24 ENA devices enable high speed and low overhead network traffic
25 processing by providing multiple Tx/Rx queue pairs (the maximum number
26 is advertised by the device via the Admin Queue), a dedicated MSI-X
27 interrupt vector per Tx/Rx queue pair, adaptive interrupt moderation,
28 and CPU cacheline optimized data placement.
30 The ENA driver supports industry standard TCP/IP offload features such as
31 checksum offload. Receive-side scaling (RSS) is supported for multi-core
34 The ENA driver and its corresponding devices implement health
35 monitoring mechanisms such as watchdog, enabling the device and driver
36 to recover in a manner transparent to the application, as well as
39 Some of the ENA devices support a working mode called Low-latency
40 Queue (LLQ), which saves several more microseconds.
42 ENA Source Code Directory Structure
43 ===================================
45 ================= ======================================================
46 ena_com.[ch] Management communication layer. This layer is
47 responsible for the handling all the management
48 (admin) communication between the device and the
50 ena_eth_com.[ch] Tx/Rx data path.
51 ena_admin_defs.h Definition of ENA management interface.
52 ena_eth_io_defs.h Definition of ENA data path interface.
53 ena_common_defs.h Common definitions for ena_com layer.
54 ena_regs_defs.h Definition of ENA PCI memory-mapped (MMIO) registers.
55 ena_netdev.[ch] Main Linux kernel driver.
56 ena_ethtool.c ethtool callbacks.
57 ena_pci_id_tbl.h Supported device IDs.
58 ================= ======================================================
63 ENA management interface is exposed by means of:
65 - PCIe Configuration Space
67 - Admin Queue (AQ) and Admin Completion Queue (ACQ)
68 - Asynchronous Event Notification Queue (AENQ)
70 ENA device MMIO Registers are accessed only during driver
71 initialization and are not used during further normal device
74 AQ is used for submitting management commands, and the
75 results/responses are reported asynchronously through ACQ.
77 ENA introduces a small set of management commands with room for
78 vendor-specific extensions. Most of the management operations are
79 framed in a generic Get/Set feature command.
81 The following admin queue commands are supported:
83 - Create I/O submission queue
84 - Create I/O completion queue
85 - Destroy I/O submission queue
86 - Destroy I/O completion queue
92 Refer to ena_admin_defs.h for the list of supported Get/Set Feature
95 The Asynchronous Event Notification Queue (AENQ) is a uni-directional
96 queue used by the ENA device to send to the driver events that cannot
97 be reported using ACQ. AENQ events are subdivided into groups. Each
98 group may have multiple syndromes, as shown below
102 ==================== ===============
104 ==================== ===============
105 Link state change **X**
107 Notification Suspend traffic
108 Notification Resume traffic
110 ==================== ===============
112 ACQ and AENQ share the same MSI-X vector.
114 Keep-Alive is a special mechanism that allows monitoring the device's health.
115 A Keep-Alive event is delivered by the device every second.
116 The driver maintains a watchdog (WD) handler which logs the current state and
117 statistics. If the keep-alive events aren't delivered as expected the WD resets
118 the device and the driver.
123 I/O operations are based on Tx and Rx Submission Queues (Tx SQ and Rx
124 SQ correspondingly). Each SQ has a completion queue (CQ) associated
127 The SQs and CQs are implemented as descriptor rings in contiguous
130 The ENA driver supports two Queue Operation modes for Tx SQs:
133 In this mode the Tx SQs reside in the host's memory. The ENA
134 device fetches the ENA Tx descriptors and packet data from host
137 - **Low Latency Queue (LLQ) mode or "push-mode":**
138 In this mode the driver pushes the transmit descriptors and the
139 first 96 bytes of the packet directly to the ENA device memory
140 space. The rest of the packet payload is fetched by the
141 device. For this operation mode, the driver uses a dedicated PCI
142 device memory BAR, which is mapped with write-combine capability.
144 **Note that** not all ENA devices support LLQ, and this feature is negotiated
145 with the device upon initialization. If the ENA device does not
146 support LLQ mode, the driver falls back to the regular mode.
148 The Rx SQs support only the regular mode.
150 The driver supports multi-queue for both Tx and Rx. This has various
153 - Reduced CPU/thread/process contention on a given Ethernet interface.
154 - Cache miss rate on completion is reduced, particularly for data
155 cache lines that hold the sk_buff structures.
156 - Increased process-level parallelism when handling received packets.
157 - Increased data cache hit rate, by steering kernel processing of
158 packets to the CPU, where the application thread consuming the
160 - In hardware interrupt re-direction.
165 The driver assigns a single MSI-X vector per queue pair (for both Tx
166 and Rx directions). The driver assigns an additional dedicated MSI-X vector
167 for management (for ACQ and AENQ).
169 Management interrupt registration is performed when the Linux kernel
170 probes the adapter, and it is de-registered when the adapter is
171 removed. I/O queue interrupt registration is performed when the Linux
172 interface of the adapter is opened, and it is de-registered when the
175 The management interrupt is named::
177 ena-mgmnt@pci:<PCI domain:bus:slot.function>
179 and for each queue pair, an interrupt is named::
181 <interface name>-Tx-Rx-<queue index>
183 The ENA device operates in auto-mask and auto-clear interrupt
184 modes. That is, once MSI-X is delivered to the host, its Cause bit is
185 automatically cleared and the interrupt is masked. The interrupt is
186 unmasked by the driver after NAPI processing is complete.
191 ENA driver and device can operate in conventional or adaptive interrupt
194 **In conventional mode** the driver instructs device to postpone interrupt
195 posting according to static interrupt delay value. The interrupt delay
196 value can be configured through `ethtool(8)`. The following `ethtool`
197 parameters are supported by the driver: ``tx-usecs``, ``rx-usecs``
199 **In adaptive interrupt** moderation mode the interrupt delay value is
200 updated by the driver dynamically and adjusted every NAPI cycle
201 according to the traffic nature.
203 Adaptive coalescing can be switched on/off through `ethtool(8)`'s
204 :code:`adaptive_rx on|off` parameter.
206 More information about Adaptive Interrupt Moderation (DIM) can be found in
207 Documentation/networking/net_dim.rst
213 The rx_copybreak is initialized by default to ENA_DEFAULT_RX_COPYBREAK
214 and can be configured by the ETHTOOL_STUNABLE command of the
220 The user can obtain ENA device and driver statistics using `ethtool`.
221 The driver can collect regular or extended statistics (including
222 per-queue stats) from the device.
224 In addition the driver logs the stats to syslog upon device reset.
229 The driver supports an arbitrarily large MTU with a maximum that is
230 negotiated with the device. The driver configures MTU using the
231 SetFeature command (ENA_ADMIN_MTU property). The user can change MTU
232 via `ip(8)` and similar legacy tools.
237 The ENA driver supports:
239 - IPv4 header checksum offload
240 - TCP/UDP over IPv4/IPv6 checksum offloads
245 - The ENA device supports RSS that allows flexible Rx traffic
247 - Toeplitz and CRC32 hash functions are supported.
248 - Different combinations of L2/L3/L4 fields can be configured as
249 inputs for hash functions.
250 - The driver configures RSS settings using the AQ SetFeature command
251 (ENA_ADMIN_RSS_HASH_FUNCTION, ENA_ADMIN_RSS_HASH_INPUT and
252 ENA_ADMIN_RSS_INDIRECTION_TABLE_CONFIG properties).
253 - If the NETIF_F_RXHASH flag is set, the 32-bit result of the hash
254 function delivered in the Rx CQ descriptor is set in the received
256 - The user can provide a hash key, hash function, and configure the
257 indirection table through `ethtool(8)`.
265 :code:`ena_start_xmit()` is called by the stack. This function does the following:
267 - Maps data buffers (``skb->data`` and frags).
268 - Populates ``ena_buf`` for the push buffer (if the driver and device are
270 - Prepares ENA bufs for the remaining frags.
271 - Allocates a new request ID from the empty ``req_id`` ring. The request
272 ID is the index of the packet in the Tx info. This is used for
273 out-of-order Tx completions.
274 - Adds the packet to the proper place in the Tx ring.
275 - Calls :code:`ena_com_prepare_tx()`, an ENA communication layer that converts
276 the ``ena_bufs`` to ENA descriptors (and adds meta ENA descriptors as
279 * This function also copies the ENA descriptors and the push buffer
280 to the Device memory space (if in push mode).
282 - Writes a doorbell to the ENA device.
283 - When the ENA device finishes sending the packet, a completion
285 - The interrupt handler schedules NAPI.
286 - The :code:`ena_clean_tx_irq()` function is called. This function handles the
287 completion descriptors generated by the ENA, with a single
288 completion descriptor per completed packet.
290 * ``req_id`` is retrieved from the completion descriptor. The ``tx_info`` of
291 the packet is retrieved via the ``req_id``. The data buffers are
292 unmapped and ``req_id`` is returned to the empty ``req_id`` ring.
293 * The function stops when the completion descriptors are completed or
294 the budget is reached.
299 - When a packet is received from the ENA device.
300 - The interrupt handler schedules NAPI.
301 - The :code:`ena_clean_rx_irq()` function is called. This function calls
302 :code:`ena_com_rx_pkt()`, an ENA communication layer function, which returns the
303 number of descriptors used for a new packet, and zero if
304 no new packet is found.
305 - :code:`ena_rx_skb()` checks packet length:
307 * If the packet is small (len < rx_copybreak), the driver allocates
308 a SKB for the new packet, and copies the packet payload into the
311 - In this way the original data buffer is not passed to the stack
312 and is reused for future Rx packets.
314 * Otherwise the function unmaps the Rx buffer, sets the first
315 descriptor as `skb`'s linear part and the other descriptors as the
318 - The new SKB is updated with the necessary information (protocol,
319 checksum hw verify result, etc), and then passed to the network
320 stack, using the NAPI interface function :code:`napi_gro_receive()`.
322 Dynamic RX Buffers (DRB)
323 ------------------------
325 Each RX descriptor in the RX ring is a single memory page (which is either 4KB
326 or 16KB long depending on system's configurations).
327 To reduce the memory allocations required when dealing with a high rate of small
328 packets, the driver tries to reuse the remaining RX descriptor's space if more
329 than 2KB of this page remain unused.
331 A simple example of this mechanism is the following sequence of events:
335 1. Driver allocates page-sized RX buffer and passes it to hardware
336 +----------------------+
338 +----------------------+
340 2. A 300Bytes packet is received on this buffer
342 3. The driver increases the ref count on this page and returns it back to
343 HW as an RX buffer of size 4KB - 300Bytes = 3796 Bytes
344 +----+--------------------+
345 |****|3796 Bytes RX Buffer|
346 +----+--------------------+
348 This mechanism isn't used when an XDP program is loaded, or when the
349 RX packet is less than rx_copybreak bytes (in which case the packet is
350 copied out of the RX buffer into the linear part of a new skb allocated
351 for it and the RX buffer remains the same size, see `RX copybreak`_).