--- /dev/null
+# Ubuntu Kernel Annotations
+
+## Overview
+
+Ubuntu provides a wide variety of kernels for many different architectures and
+flavours, such as generic, lowlatency, cloud kernels, kvm, etc.
+
+Maintaining a separate `.config` for each of these kernels would be extremely
+challenging, Kconfig options need to always match the desired value in order to
+have a kernel functional for the scope that it was designed for.
+
+Starting with Ubuntu 22.04 (Jammy Jellyfish) the kernel team introduced a new
+format of annotations: a JSON-like file to store all the kernel config options
+for the supported architectures and flavours.
+
+## Quick start
+
+Generate the Ubuntu kernel .config for a specific architecture & flavour
+(i.e., `amd64`, `generic`):
+
+```
+$ annotations --arch amd64 --flavour generic --export > .config
+```
+
+Import your own kernel .config in the Ubuntu kernel for a specific architecture
+and flavour (i.e., `amd64`, `generic`):
+
+```
+$ annotations --arch amd64 --flavour generic --import .config
+```
+
+After a new config is imported run `fakeroot debian/rules clean updateconfigs`
+to update the dependencies, then you can review the actual changes doing a
+simple `git diff`.
+
+## Ubuntu kernel annotations
+
+There is a main annotations file for each Ubuntu kernel in:
+`debian.<kernel_name>/config/annotations`
+
+The main annotations file can include other annotations (recursively).
+
+The annotations has 4 different sections:
+
+ - a header (that defines the format version, the list of supported
+ architectures, the list of supported flavours paired with each architecture
+ and a list of flavour inheritance rules)
+
+ - a list of includes (annotations can include other annotations files to
+ create a hierarchy of configs with overrides)
+
+ - a subset of mandatory config options that have a note associated (typically
+ config options that must be enforced for certain reasons, the note can be
+ simply a reference to a tracking bug, explaining why the specific config
+ needs to be set to a certain value, or just a description, like security
+ implications for that config value, etc.)
+
+ - a subset of config options that are either generated as dependency of the
+ previous subset of configs or configs that have been set using the default
+ Ubuntu policy (enable everything as module when possible, except for debugging,
+ testing or deprecated features), so they don't need an explicit note.
+
+The annotations format looks like the following:
+
+```
+# Menu: HEADER
+# FORMAT: 4
+# ARCH: <ARCH_1> <ARCH_2> <ARCH_3> ... <ARCH_N>
+# FLAVOUR: <ARCH_1>-<FLAVOUR_1> <ARCH1_FLAVOUR_2> ... <ARCH_N>-<FLAVOUR_M>
+# FLAVOUR_DEP: {'<ARCH_1>-<FLAVOUR_1>': '<PARENT_ARCH_1>-<PARENT_FLAVOUR_1>', ..., '<ARCH_N>-<FLAVOUR_M>': '<PARENT_ARCH_N>-<PARENT_FLAVOUR_M>'}
+
+include <FILE_1>
+include <FILE_2>
+...
+include <FILE_N>
+
+CONFIG_<FOO> policy<{'<ARCH_1>[-<FLAVOUR_1>]': '<VALUE_1_1>', ..., '<ARCH_N>[-<FLAVOUR_M>]': '<VALUE_N_M>'}>
+CONFIG_<FOO> note<'<DESCRIPTION>'>
+...
+
+# ---- Annotations without notes ----
+
+CONFIG_<BAR> policy<{'<ARCH_1>[-<FLAVOUR_1>]': '<VALUE_1_1>', ..., '<ARCH_N>[-<FLAVOUR_M>]': '<VALUE_N_M>'}>
+...
+
+```
+
+ - **ARCH**: contains the list of comma-separated values that defines all the
+ supported architectures for this kernel
+
+ - **FLAVOUR**: contains the list of comma-separated values that defines the
+ list of flavours associated to each supported architecture
+
+ - **FLAVOUR_DEP**: contains a mapping of `ARCH-FLAVOUR` pairs mapped to a
+ parent `ARCH-FLAVOUR` pair; this can be used to define flavours that derives
+ from other flavours, for example `amd64-lowlatency` derives from
+ `amd64-generic` (then the local annotations file will only contain an
+ include rule of the generic kernel plus the small subset of config options
+ to override the defaults from `amd64-generic`)
+
+ - **FILE_1 .. FILE_N**: are the other included annotations (e.g., used by
+ derivative kernels that want to include the annotations from the kernel they
+ derive from)
+
+ - **FOO, BAR**: kernel config options with the associated values across
+ architectures and flavours, represented by the `policy<...>` definition
+
+A simple example of an annotations file is the following (from the
+`lunar/linux-lowlatency` kernel, that derives from the generic kernel
+`lunar/linux`):
+
+```
+# Menu: HEADER
+# FORMAT: 4
+# ARCH: amd64 arm64
+# FLAVOUR: amd64-lowlatency arm64-lowlatency arm64-lowlatency-64k
+# FLAVOUR_DEP: {'amd64-lowlatency': 'amd64-generic', 'arm64-lowlatency': 'arm64-generic', 'arm64-lowlatency-64k': 'arm64-generic-64k'}
+
+include "../../debian.master/config/annotations"
+
+CONFIG_HZ_1000 policy<{'amd64': 'y', 'arm64': 'y'}>
+CONFIG_HZ_1000 note<'HZ for lowlatency must be set to 1000 to provide better system responsiveness'>
+
+CONFIG_HZ_250 policy<{'amd64': 'n', 'arm64': 'n'}>
+CONFIG_HZ_250 note<'Override default HZ used in generic'>
+
+CONFIG_LATENCYTOP policy<{'amd64': 'y', 'arm64': 'y'}>
+CONFIG_LATENCYTOP note<'https://lists.ubuntu.com/archives/kernel-team/2014-July/045006.html, LP#1655986'>
+
+CONFIG_PREEMPT policy<{'amd64': 'y', 'arm64': 'y'}>
+CONFIG_PREEMPT note<'Enable fully preemptible kernel'>
+
+CONFIG_PREEMPT_VOLUNTARY policy<{'amd64': 'n', 'arm64': 'n'}>
+CONFIG_PREEMPT_VOLUNTARY note<'Disable voluntary preemption model'>
+
+
+# ---- Annotations without notes ----
+
+CONFIG_HZ policy<{'amd64': '1000', 'arm64': '1000'}>
+```
+
+## Managing the annotations file
+
+To help the management of annotations each Ubuntu kernel provides a helper
+script in `debian/scripts/misc/annotations` (use `--help` for an overview of
+the supported actions).
+
+Here some typical examples of what you can do with the annotations script:
+
+ - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across all the
+ supported architectures and flavours:
+
+```
+$ annotations --config CONFIG_DEBUG_INFO_BTF
+{
+ "policy": {
+ "amd64": "y",
+ "arm64": "y",
+ "armhf": "n",
+ "ppc64el": "y",
+ "riscv64": "y",
+ "s390x": "y"
+ },
+ "note": "'Needs newer pahole for armhf'"
+}
+```
+
+ - Dump kernel `.config` for `arm64` and flavour `generic-64k`:
+
+```
+$ annotations --arch arm64 --flavour generic-64k --export
+CONFIG_DEBUG_FS=y
+CONFIG_DEBUG_KERNEL=y
+CONFIG_COMPAT=y
+...
+```
+
+ - Update annotations file with a new kernel .config for `amd64` flavour
+ `generic`:
+
+```
+$ annotations --arch amd64 --flavour generic --import .config
+```
+
+ - Enable `CONFIG_PROVE_LOCKING` on `amd64` for flavour `generic`:
+
+```
+$ annotations -c CONFIG_PROVE_LOCKING --arch amd64 --flavour generic --write y
+{
+ "CONFIG_PROVE_LOCKING": {
+ "policy": {
+ "amd64": "y",
+ "arm64": "n",
+ "armhf": "n",
+ "ppc64el": "n",
+ "riscv64": "n",
+ "s390x": "n"
+ },
+ "oneline": false,
+ "note": "'prove locking enabled as a test'"
+ }
+}
+
+$ git diff
+diff --git a/debian.master/config/annotations b/debian.master/config/annotations
+index 24cec55b1b20b..e331bbd6de640 100644
+--- a/debian.master/config/annotations
++++ b/debian.master/config/annotations
+@@ -483,6 +483,9 @@ CONFIG_PPC_SECVAR_SYSFS note<'LP: #1866909'>
+ CONFIG_PREEMPT_NONE policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'y'}>
+ CONFIG_PREEMPT_NONE note<'LP: #1543165'>
+
++CONFIG_PROVE_LOCKING policy<{'amd64': 'y', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
++CONFIG_PROVE_LOCKING note<'prove locking enabled as a test'>
++
+ CONFIG_PSI_DEFAULT_DISABLED policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'y'}>
+ CONFIG_PSI_DEFAULT_DISABLED note<'LP: #1876044'>
+
+@@ -9892,7 +9895,6 @@ CONFIG_PROFILE_ALL_BRANCHES policy<{'riscv64': 'n'}>
+ CONFIG_PROFILE_ANNOTATED_BRANCHES policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
+ CONFIG_PROFILING policy<{'amd64': 'y', 'arm64': 'y', 'armhf': 'y', 'ppc64el': 'y', 'riscv64': 'y', 's390x': 'y'}>
+ CONFIG_PROTECTED_VIRTUALIZATION_GUEST policy<{'s390x': 'y'}>
+-CONFIG_PROVE_LOCKING policy<{'amd64': 'n', 'arm64': 'n', 'armhf': 'n', 'ppc64el': 'n', 'riscv64': 'n', 's390x': 'n'}>
+ CONFIG_PROVIDE_OHCI1394_DMA_INIT policy<{'amd64': 'n'}>
+ CONFIG_PRU_REMOTEPROC policy<{'arm64': 'm', 'armhf': 'm'}>
+ CONFIG_PSAMPLE policy<{'amd64': 'm', 'arm64': 'm', 'armhf': 'm', 'ppc64el': 'm', 'riscv64': 'm', 's390x': 'm'}>
+```
+
+Every time the annotations file is changed we need to make sure that all the
+dependent config options are refreshed.
+
+To do so we need to run the following command:
+
+```
+$ fakeroot debian/rules clean updateconfigs
+...
+```
+
+This command will take care of interfacing with the kernel Kconfig subsystem
+and it will interactively ask the value for all the potentially new enabled
+dependent options.
+
+## Conclusion
+
+The annotations format introduced in Lunar provides an easier and more
+efficient way of managing kernel .config's across the large variety of kernels,
+architectures and flavours that Ubuntu supports.
+
+If you want to submit a patch to kernel-team@lists.ubuntu.com that needs to
+change the kernel `.config`, or simply if you want to recompile the Ubuntu
+kernel using a different `.config` you just need to modify the annotations file
+and then you can simply upload your custom modified kernel to a ppa to test it,
+or you can recompile it locally using the typical Ubuntu way of rebuilding
+packages.
+
+## See also
+
+ - https://lists.ubuntu.com/archives/kernel-team/2023-June/140230.html
+ - https://discourse.ubuntu.com/t/kernel-configuration-in-ubuntu/35857
+++ /dev/null
-==================
-Config Annotations
-==================
-
-:Author: Andrea Righi
-
-Overview
-========
-
-Each Ubuntu kernel needs to maintain its own .config for each supported
-architecture and each flavour.
-
-Every time a new patch is applied or a kernel is rebased on top of a new
-one, we need to update the .config's accordingly (config options can be
-added, removed and also renamed).
-
-So, we need to make sure that some critical config options are always
-matching the desired value in order to have a functional kernel.
-
-State of the art
-================
-
-At the moment configs are maintained as a set of Kconfig chunks (inside
-`debian.<kernel>/config/`): a global one, plus per-arch / per-flavour
-chunks.
-
-In addition to that, we need to maintain also a file called
-'annotations'; the purpose of this file is to make sure that some
-critical config options are not silently removed or changed when the
-real .config is re-generated (for example after a rebase or after
-applying a new set of patches).
-
-The main problem with this approach is that, often, we have duplicate
-information that is stored both in the Kconfig chunks *and* in the
-annotations files and, at the same time, the whole .config's information
-is distributed between Kconfig chunks and annotations, making it hard to
-maintain, review and manage in general.
-
-Proposed solution
-=================
-
-The proposed solution is to store all the config information into the
-"annotations" format and get rid of the config chunks (basically the
-real .config's can be produced "compiling" annotations).
-
-Implementation
-==============
-
-To help the management of the annotations an helper script is provided
-(`debian/scripts/misc/annotations`):
-
-```
-usage: annotations [-h] [--version] [--file FILE] [--arch ARCH] [--flavour FLAVOUR] [--config CONFIG]
- (--query | --export | --import FILE | --update FILE | --check FILE)
-
-Manage Ubuntu kernel .config and annotations
-
-options:
- -h, --help show this help message and exit
- --version, -v show program's version number and exit
- --file FILE, -f FILE Pass annotations or .config file to be parsed
- --arch ARCH, -a ARCH Select architecture
- --flavour FLAVOUR, -l FLAVOUR
- Select flavour (default is "generic")
- --config CONFIG, -c CONFIG
- Select a specific config option
-
-Action:
- --query, -q Query annotations
- --export, -e Convert annotations to .config format
- --import FILE, -i FILE
- Import a full .config for a specific arch and flavour into annotations
- --update FILE, -u FILE
- Import a partial .config into annotations (only resync configs specified in FILE)
- --check FILE, -k FILE
- Validate kernel .config with annotations
-```
-
-This script allows to query config settings (per arch/flavour/config),
-export them into the Kconfig format (generating the real .config files)
-and check if the final .config matches the rules defined in the
-annotations.
-
-Examples (annotations is defined as an alias to `debian/scripts/annotations`):
-
- - Show settings for `CONFIG_DEBUG_INFO_BTF` for master kernel across all the
- supported architectures and flavours:
-
-```
-$ annotations --query --config CONFIG_DEBUG_INFO_BTF
-{
- "policy": {
- "amd64": "y",
- "arm64": "y",
- "armhf": "n",
- "ppc64el": "y",
- "riscv64": "y",
- "s390x": "y"
- },
- "note": "'Needs newer pahole for armhf'"
-}
-```
-
- - Dump kernel .config for arm64 and flavour generic-64k:
-
-```
-$ annotations --arch arm64 --flavour generic-64k --export
-CONFIG_DEBUG_FS=y
-CONFIG_DEBUG_KERNEL=y
-CONFIG_COMPAT=y
-...
-```
-
- - Update annotations file with a new kernel .config for amd64 flavour
- generic:
-
-```
-$ annotations --arch amd64 --flavour generic --import build/.config
-```
-
-Moreover, an additional kernelconfig commands are provided
-(via debian/rules targets):
- - `migrateconfigs`: automatically merge all the previous configs into
- annotations (local changes still need to be committed)
-
-Annotations headers
-===================
-
-The main annotations file should contain a header to define the architectures
-and flavours that are supported.
-
-Here is the format of the header for the generic kernel:
-```
-# Menu: HEADER
-# FORMAT: 4
-# ARCH: amd64 arm64 armhf ppc64el riscv64 s390x
-# FLAVOUR: amd64-generic arm64-generic arm64-generic-64k armhf-generic armhf-generic-lpae ppc64el-generic riscv64-generic s390x-generic
-
-```
-
-Example header of a derivative (linux-aws):
-```
-# Menu: HEADER
-# FORMAT: 4
-# ARCH: amd64 arm64
-# FLAVOUR: amd64-aws arm64-aws
-# FLAVOUR_DEP: {'amd64-aws': 'amd64-generic', 'arm64-aws': 'arm64-generic'}
-
-include "../../debian.master/config/annotations"
-
-# Below you can define only the specific linux-aws configs that differ from linux generic
-
-```
-
-Pros and Cons
-=============
-
- Pros:
- - avoid duplicate information in .config's and annotations
- - allow to easily define groups of config settings (for a specific
- environment or feature, such as annotations.clouds, annotations.ubuntu,
- annotations.snapd, etc.)
- - config options are more accessible, easy to change and review
- - we can easily document how config options are managed (and external
- contributors won't be discouraged anymore when they need to to change a
- config option)
-
- Cons:
- - potential regressions: the new tool/scripts can have potential bugs,
- so we could experience regressions due to some missed config changes
- - kernel team need to understand the new process (even if everything
- is transparent, kernel cranking process is the same, there might be
- corner cases that need to be addressed and resolved manually)
-
-TODO
-====
-
- - Migrate all flavour and arch definitions into annotations (rather
- than having this information defined in multiple places inside
- debian/scripts); right now this information is "partially" migrated,
- meaning that we need to define arches and flavours in the headers
- section of annotations (so that the annotations tool can figure out
- the list of supported arches and flavours), but arches and flavours
- are still defined elsewhere, ideally we would like to have arches and
- flavours defined only in one place: annotations.
-
-See also
-========
-
- - https://lists.ubuntu.com/archives/kernel-team/2023-June/140230.html
- - https://discourse.ubuntu.com/t/kernel-configuration-in-ubuntu/35857
projects / annotations.git / commitdiff