1 .. SPDX-License-Identifier: GPL-2.0
3 Media Controller devices
4 ------------------------
9 The media controller userspace API is documented in
10 :ref:`the Media Controller uAPI book <media_controller>`. This document focus
11 on the kernel-side implementation of the media framework.
13 Abstract media device model
14 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
16 Discovering a device internal topology, and configuring it at runtime, is one
17 of the goals of the media framework. To achieve this, hardware devices are
18 modelled as an oriented graph of building blocks called entities connected
21 An entity is a basic media hardware building block. It can correspond to
22 a large variety of logical blocks such as physical hardware devices
23 (CMOS sensor for instance), logical hardware devices (a building block
24 in a System-on-Chip image processing pipeline), DMA channels or physical
27 A pad is a connection endpoint through which an entity can interact with
28 other entities. Data (not restricted to video) produced by an entity
29 flows from the entity's output to one or more entity inputs. Pads should
30 not be confused with physical pins at chip boundaries.
32 A link is a point-to-point oriented connection between two pads, either
33 on the same entity or on different entities. Data flows from a source
39 A media device is represented by a struct media_device
40 instance, defined in ``include/media/media-device.h``.
41 Allocation of the structure is handled by the media device driver, usually by
42 embedding the :c:type:`media_device` instance in a larger driver-specific
45 Drivers initialise media device instances by calling
46 :c:func:`media_device_init()`. After initialising a media device instance, it is
47 registered by calling :c:func:`__media_device_register()` via the macro
48 ``media_device_register()`` and unregistered by calling
49 :c:func:`media_device_unregister()`. An initialised media device must be
50 eventually cleaned up by calling :c:func:`media_device_cleanup()`.
52 Note that it is not allowed to unregister a media device instance that was not
53 previously registered, or clean up a media device instance that was not
54 previously initialised.
59 Entities are represented by a struct media_entity
60 instance, defined in ``include/media/media-entity.h``. The structure is usually
61 embedded into a higher-level structure, such as
62 :c:type:`v4l2_subdev` or :c:type:`video_device`
63 instances, although drivers can allocate entities directly.
65 Drivers initialize entity pads by calling
66 :c:func:`media_entity_pads_init()`.
68 Drivers register entities with a media device by calling
69 :c:func:`media_device_register_entity()`
70 and unregistered by calling
71 :c:func:`media_device_unregister_entity()`.
76 Interfaces are represented by a
77 struct media_interface instance, defined in
78 ``include/media/media-entity.h``. Currently, only one type of interface is
79 defined: a device node. Such interfaces are represented by a
80 struct media_intf_devnode.
82 Drivers initialize and create device node interfaces by calling
83 :c:func:`media_devnode_create()`
84 and remove them by calling:
85 :c:func:`media_devnode_remove()`.
89 Pads are represented by a struct media_pad instance,
90 defined in ``include/media/media-entity.h``. Each entity stores its pads in
91 a pads array managed by the entity driver. Drivers usually embed the array in
92 a driver-specific structure.
94 Pads are identified by their entity and their 0-based index in the pads
97 Both information are stored in the struct media_pad,
98 making the struct media_pad pointer the canonical way
99 to store and pass link references.
101 Pads have flags that describe the pad capabilities and state.
103 ``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.
104 ``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.
108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must
114 Links are represented by a struct media_link instance,
115 defined in ``include/media/media-entity.h``. There are two types of links:
117 **1. pad to pad links**:
119 Associate two entities via their PADs. Each entity has a list that points
120 to all links originating at or targeting any of its pads.
121 A given link is thus stored twice, once in the source entity and once in
124 Drivers create pad to pad links by calling:
125 :c:func:`media_create_pad_link()` and remove with
126 :c:func:`media_entity_remove_links()`.
128 **2. interface to entity links**:
130 Associate one interface to a Link.
132 Drivers create interface to entity links by calling:
133 :c:func:`media_create_intf_link()` and remove with
134 :c:func:`media_remove_intf_links()`.
138 Links can only be created after having both ends already created.
140 Links have flags that describe the link capabilities and state. The
141 valid values are described at :c:func:`media_create_pad_link()` and
142 :c:func:`media_create_intf_link()`.
147 The media framework provides APIs to iterate over entities in a graph.
149 To iterate over all entities belonging to a media device, drivers can use
150 the media_device_for_each_entity macro, defined in
151 ``include/media/media-device.h``.
155 struct media_entity *entity;
157 media_device_for_each_entity(entity, mdev) {
158 // entity will point to each entity in turn
162 Drivers might also need to iterate over all entities in a graph that can be
163 reached only through enabled links starting at a given entity. The media
164 framework provides a depth-first graph traversal API for that purpose.
168 Graphs with cycles (whether directed or undirected) are **NOT**
169 supported by the graph traversal API. To prevent infinite loops, the graph
170 traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``,
171 currently defined as 16.
173 Drivers initiate a graph traversal by calling
174 :c:func:`media_graph_walk_start()`
176 The graph structure, provided by the caller, is initialized to start graph
177 traversal at the given entity.
179 Drivers can then retrieve the next entity by calling
180 :c:func:`media_graph_walk_next()`
182 When the graph traversal is complete the function will return ``NULL``.
184 Graph traversal can be interrupted at any moment. No cleanup function call
185 is required and the graph structure can be freed normally.
187 Helper functions can be used to find a link between two given pads, or a pad
188 connected to another pad through an enabled link
189 (:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`,
190 :c:func:`media_entity_remote_source_pad_unique()` and
191 :c:func:`media_pad_remote_pad_unique()`).
193 Use count and power handling
194 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
196 Due to the wide differences between drivers regarding power management
197 needs, the media controller does not implement power management. However,
198 the struct media_entity includes a ``use_count``
199 field that media drivers
200 can use to track the number of users of every entity for power management
203 The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by
204 media drivers and must not be
205 touched by entity drivers. Access to the field must be protected by the
206 :c:type:`media_device`.\ ``graph_mutex`` lock.
211 Link properties can be modified at runtime by calling
212 :c:func:`media_entity_setup_link()`.
214 Pipelines and media streams
215 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
217 A media stream is a stream of pixels or metadata originating from one or more
218 source devices (such as a sensors) and flowing through media entity pads
219 towards the final sinks. The stream can be modified on the route by the
220 devices (e.g. scaling or pixel format conversions), or it can be split into
221 multiple branches, or multiple branches can be merged.
223 A media pipeline is a set of media streams which are interdependent. This
224 interdependency can be caused by the hardware (e.g. configuration of a second
225 stream cannot be changed if the first stream has been enabled) or by the driver
226 due to the software design. Most commonly a media pipeline consists of a single
227 stream which does not branch.
229 When starting streaming, drivers must notify all entities in the pipeline to
230 prevent link states from being modified during streaming by calling
231 :c:func:`media_pipeline_start()`.
233 The function will mark all the pads which are part of the pipeline as streaming.
235 The struct media_pipeline instance pointed to by the pipe argument will be
236 stored in every pad in the pipeline. Drivers should embed the struct
237 media_pipeline in higher-level pipeline structures and can then access the
238 pipeline through the struct media_pad pipe field.
240 Calls to :c:func:`media_pipeline_start()` can be nested.
241 The pipeline pointer must be identical for all nested calls to the function.
243 :c:func:`media_pipeline_start()` may return an error. In that case,
244 it will clean up any of the changes it did by itself.
246 When stopping the stream, drivers must notify the entities with
247 :c:func:`media_pipeline_stop()`.
249 If multiple calls to :c:func:`media_pipeline_start()` have been
250 made the same number of :c:func:`media_pipeline_stop()` calls
251 are required to stop streaming.
252 The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last
255 Link configuration will fail with ``-EBUSY`` by default if either end of the
256 link is a streaming entity. Links that can be modified while streaming must
257 be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.
259 If other operations need to be disallowed on streaming entities (such as
260 changing entities configuration parameters) drivers can explicitly check the
261 media_entity stream_count field to find out if an entity is streaming. This
262 operation must be done with the media_device graph_mutex held.
267 Link validation is performed by :c:func:`media_pipeline_start()`
268 for any entity which has sink pads in the pipeline. The
269 :c:type:`media_entity`.\ ``link_validate()`` callback is used for that
270 purpose. In ``link_validate()`` callback, entity driver should check
271 that the properties of the source pad of the connected entity and its own
272 sink pad match. It is up to the type of the entity (and in the end, the
273 properties of the hardware) what matching actually means.
275 Subsystems should facilitate link validation by providing subsystem specific
276 helper functions to provide easy access for commonly needed information, and
277 in the end provide a way to use driver-specific callbacks.
279 Media Controller Device Allocator API
280 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
282 When the media device belongs to more than one driver, the shared media
283 device is allocated with the shared struct device as the key for look ups.
285 The shared media device should stay in registered state until the last
286 driver unregisters it. In addition, the media device should be released when
287 all the references are released. Each driver gets a reference to the media
288 device during probe, when it allocates the media device. If media device is
289 already allocated, the allocate API bumps up the refcount and returns the
290 existing media device. The driver puts the reference back in its disconnect
291 routine when it calls :c:func:`media_device_delete()`.
293 The media device is unregistered and cleaned up from the kref put handler to
294 ensure that the media device stays in registered state until the last driver
295 unregisters the media device.
299 Drivers should use the appropriate media-core routines to manage the shared
300 media device life-time handling the two states:
301 1. allocate -> register -> delete
302 2. get reference to already registered device -> delete
304 call :c:func:`media_device_delete()` routine to make sure the shared media
305 device delete is handled correctly.
308 Call :c:func:`media_device_usb_allocate()` to allocate or get a reference
309 Call :c:func:`media_device_register()`, if media devnode isn't registered
311 **driver disconnect:**
312 Call :c:func:`media_device_delete()` to free the media_device. Freeing is
313 handled by the kref put handler.
318 .. kernel-doc:: include/media/media-device.h
320 .. kernel-doc:: include/media/media-devnode.h
322 .. kernel-doc:: include/media/media-entity.h
324 .. kernel-doc:: include/media/media-request.h
326 .. kernel-doc:: include/media/media-dev-allocator.h