1 .. SPDX-License-Identifier: GPL-2.0
2 .. include:: <isonum.txt>
4 ===========================================
5 User Interface for Resource Control feature
6 ===========================================
8 :Copyright: |copy| 2016 Intel Corporation
9 :Authors: - Fenghua Yu <fenghua.yu@intel.com>
10 - Tony Luck <tony.luck@intel.com>
11 - Vikas Shivappa <vikas.shivappa@intel.com>
14 Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
15 AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).
17 This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
20 =============================================== ================================
21 RDT (Resource Director Technology) Allocation "rdt_a"
22 CAT (Cache Allocation Technology) "cat_l3", "cat_l2"
23 CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"
24 CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"
25 MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"
26 MBA (Memory Bandwidth Allocation) "mba"
27 SMBA (Slow Memory Bandwidth Allocation) ""
28 BMEC (Bandwidth Monitoring Event Configuration) ""
29 =============================================== ================================
31 Historically, new features were made visible by default in /proc/cpuinfo. This
32 resulted in the feature flags becoming hard to parse by humans. Adding a new
33 flag to /proc/cpuinfo should be avoided if user space can obtain information
34 about the feature from resctrl's info directory.
36 To use the feature mount the file system::
38 # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl
43 Enable code/data prioritization in L3 cache allocations.
45 Enable code/data prioritization in L2 cache allocations.
47 Enable the MBA Software Controller(mba_sc) to specify MBA
50 Make debug files accessible. Available debug files are annotated with
51 "Available only with debug option".
53 L2 and L3 CDP are controlled separately.
55 RDT features are orthogonal. A particular system may support only
56 monitoring, only control, or both monitoring and control. Cache
57 pseudo-locking is a unique way of using cache control to "pin" or
58 "lock" data in the cache. Details can be found in
59 "Cache Pseudo-Locking".
62 The mount succeeds if either of allocation or monitoring is present, but
63 only those files and directories supported by the system will be created.
64 For more details on the behavior of the interface during monitoring
65 and allocation, see the "Resource alloc and monitor groups" section.
70 The 'info' directory contains information about the enabled
71 resources. Each resource has its own subdirectory. The subdirectory
72 names reflect the resource names.
74 Each subdirectory contains the following files with respect to
77 Cache resource(L3/L2) subdirectory contains the following files
78 related to allocation:
81 The number of CLOSIDs which are valid for this
82 resource. The kernel uses the smallest number of
83 CLOSIDs of all enabled resources as limit.
85 The bitmask which is valid for this resource.
86 This mask is equivalent to 100%.
88 The minimum number of consecutive bits which
89 must be set when writing a mask.
92 Bitmask of shareable resource with other executing
93 entities (e.g. I/O). User can use this when
94 setting up exclusive cache partitions. Note that
95 some platforms support devices that have their
96 own settings for cache use which can over-ride
99 Annotated capacity bitmasks showing how all
100 instances of the resource are used. The legend is:
103 Corresponding region is unused. When the system's
104 resources have been allocated and a "0" is found
105 in "bit_usage" it is a sign that resources are
109 Corresponding region is used by hardware only
110 but available for software use. If a resource
111 has bits set in "shareable_bits" but not all
112 of these bits appear in the resource groups'
113 schematas then the bits appearing in
114 "shareable_bits" but no resource group will
117 Corresponding region is available for sharing and
118 used by hardware and software. These are the
119 bits that appear in "shareable_bits" as
120 well as a resource group's allocation.
122 Corresponding region is used by software
123 and available for sharing.
125 Corresponding region is used exclusively by
126 one resource group. No sharing allowed.
128 Corresponding region is pseudo-locked. No
131 Indicates if non-contiguous 1s value in CBM is supported.
134 Only contiguous 1s value in CBM is supported.
136 Non-contiguous 1s value in CBM is supported.
138 Memory bandwidth(MB) subdirectory contains the following files
139 with respect to allocation:
142 The minimum memory bandwidth percentage which
146 The granularity in which the memory bandwidth
147 percentage is allocated. The allocated
148 b/w percentage is rounded off to the next
149 control step available on the hardware. The
150 available bandwidth control steps are:
151 min_bandwidth + N * bandwidth_gran.
154 Indicates if the delay scale is linear or
155 non-linear. This field is purely informational
158 "thread_throttle_mode":
159 Indicator on Intel systems of how tasks running on threads
160 of a physical core are throttled in cases where they
161 request different memory bandwidth percentages:
164 the smallest percentage is applied
167 bandwidth percentages are directly applied to
168 the threads running on the core
170 If RDT monitoring is available there will be an "L3_MON" directory
171 with the following files:
174 The number of RMIDs available. This is the
175 upper bound for how many "CTRL_MON" + "MON"
176 groups can be created.
179 Lists the monitoring events if
180 monitoring is enabled for the resource.
183 # cat /sys/fs/resctrl/info/L3_MON/mon_features
188 If the system supports Bandwidth Monitoring Event
189 Configuration (BMEC), then the bandwidth events will
190 be configurable. The output will be::
192 # cat /sys/fs/resctrl/info/L3_MON/mon_features
195 mbm_total_bytes_config
197 mbm_local_bytes_config
199 "mbm_total_bytes_config", "mbm_local_bytes_config":
200 Read/write files containing the configuration for the mbm_total_bytes
201 and mbm_local_bytes events, respectively, when the Bandwidth
202 Monitoring Event Configuration (BMEC) feature is supported.
203 The event configuration settings are domain specific and affect
204 all the CPUs in the domain. When either event configuration is
205 changed, the bandwidth counters for all RMIDs of both events
206 (mbm_total_bytes as well as mbm_local_bytes) are cleared for that
207 domain. The next read for every RMID will report "Unavailable"
208 and subsequent reads will report the valid value.
210 Following are the types of events supported:
212 ==== ========================================================
214 ==== ========================================================
215 6 Dirty Victims from the QOS domain to all types of memory
216 5 Reads to slow memory in the non-local NUMA domain
217 4 Reads to slow memory in the local NUMA domain
218 3 Non-temporal writes to non-local NUMA domain
219 2 Non-temporal writes to local NUMA domain
220 1 Reads to memory in the non-local NUMA domain
221 0 Reads to memory in the local NUMA domain
222 ==== ========================================================
224 By default, the mbm_total_bytes configuration is set to 0x7f to count
225 all the event types and the mbm_local_bytes configuration is set to
226 0x15 to count all the local memory events.
230 * To view the current configuration::
233 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
234 0=0x7f;1=0x7f;2=0x7f;3=0x7f
236 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
237 0=0x15;1=0x15;3=0x15;4=0x15
239 * To change the mbm_total_bytes to count only reads on domain 0,
240 the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary
241 (in hexadecimal 0x33):
244 # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
246 # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
247 0=0x33;1=0x7f;2=0x7f;3=0x7f
249 * To change the mbm_local_bytes to count all the slow memory reads on
250 domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b
251 in binary (in hexadecimal 0x30):
254 # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
256 # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
257 0=0x30;1=0x30;3=0x15;4=0x15
259 "max_threshold_occupancy":
260 Read/write file provides the largest value (in
261 bytes) at which a previously used LLC_occupancy
262 counter can be considered for re-use.
264 Finally, in the top level of the "info" directory there is a file
265 named "last_cmd_status". This is reset with every "command" issued
266 via the file system (making new directories or writing to any of the
267 control files). If the command was successful, it will read as "ok".
268 If the command failed, it will provide more information that can be
269 conveyed in the error returns from file operations. E.g.
272 # echo L3:0=f7 > schemata
273 bash: echo: write error: Invalid argument
274 # cat info/last_cmd_status
275 mask f7 has non-consecutive 1-bits
277 Resource alloc and monitor groups
278 =================================
280 Resource groups are represented as directories in the resctrl file
281 system. The default group is the root directory which, immediately
282 after mounting, owns all the tasks and cpus in the system and can make
283 full use of all resources.
285 On a system with RDT control features additional directories can be
286 created in the root directory that specify different amounts of each
287 resource (see "schemata" below). The root and these additional top level
288 directories are referred to as "CTRL_MON" groups below.
290 On a system with RDT monitoring the root directory and other top level
291 directories contain a directory named "mon_groups" in which additional
292 directories can be created to monitor subsets of tasks in the CTRL_MON
293 group that is their ancestor. These are called "MON" groups in the rest
296 Removing a directory will move all tasks and cpus owned by the group it
297 represents to the parent. Removing one of the created CTRL_MON groups
298 will automatically remove all MON groups below it.
300 Moving MON group directories to a new parent CTRL_MON group is supported
301 for the purpose of changing the resource allocations of a MON group
302 without impacting its monitoring data or assigned tasks. This operation
303 is not allowed for MON groups which monitor CPUs. No other move
304 operation is currently allowed other than simply renaming a CTRL_MON or
307 All groups contain the following files:
310 Reading this file shows the list of all tasks that belong to
311 this group. Writing a task id to the file will add a task to the
312 group. Multiple tasks can be added by separating the task ids
313 with commas. Tasks will be assigned sequentially. Multiple
314 failures are not supported. A single failure encountered while
315 attempting to assign a task will cause the operation to abort and
316 already added tasks before the failure will remain in the group.
317 Failures will be logged to /sys/fs/resctrl/info/last_cmd_status.
319 If the group is a CTRL_MON group the task is removed from
320 whichever previous CTRL_MON group owned the task and also from
321 any MON group that owned the task. If the group is a MON group,
322 then the task must already belong to the CTRL_MON parent of this
323 group. The task is removed from any previous MON group.
327 Reading this file shows a bitmask of the logical CPUs owned by
328 this group. Writing a mask to this file will add and remove
329 CPUs to/from this group. As with the tasks file a hierarchy is
330 maintained where MON groups may only include CPUs owned by the
331 parent CTRL_MON group.
332 When the resource group is in pseudo-locked mode this file will
333 only be readable, reflecting the CPUs associated with the
334 pseudo-locked region.
338 Just like "cpus", only using ranges of CPUs instead of bitmasks.
341 When control is enabled all CTRL_MON groups will also contain:
344 A list of all the resources available to this group.
345 Each resource has its own line and format - see below for details.
348 Mirrors the display of the "schemata" file to display the size in
349 bytes of each allocation instead of the bits representing the
353 The "mode" of the resource group dictates the sharing of its
354 allocations. A "shareable" resource group allows sharing of its
355 allocations while an "exclusive" resource group does not. A
356 cache pseudo-locked region is created by first writing
357 "pseudo-locksetup" to the "mode" file before writing the cache
358 pseudo-locked region's schemata to the resource group's "schemata"
359 file. On successful pseudo-locked region creation the mode will
360 automatically change to "pseudo-locked".
363 Available only with debug option. The identifier used by hardware
364 for the control group. On x86 this is the CLOSID.
366 When monitoring is enabled all MON groups will also contain:
369 This contains a set of files organized by L3 domain and by
370 RDT event. E.g. on a system with two L3 domains there will
371 be subdirectories "mon_L3_00" and "mon_L3_01". Each of these
372 directories have one file per event (e.g. "llc_occupancy",
373 "mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
374 files provide a read out of the current value of the event for
375 all tasks in the group. In CTRL_MON groups these files provide
376 the sum for all tasks in the CTRL_MON group and all tasks in
377 MON groups. Please see example section for more details on usage.
380 Available only with debug option. The identifier used by hardware
381 for the monitor group. On x86 this is the RMID.
383 Resource allocation rules
384 -------------------------
386 When a task is running the following rules define which resources are
389 1) If the task is a member of a non-default group, then the schemata
390 for that group is used.
392 2) Else if the task belongs to the default group, but is running on a
393 CPU that is assigned to some specific group, then the schemata for the
396 3) Otherwise the schemata for the default group is used.
398 Resource monitoring rules
399 -------------------------
400 1) If a task is a member of a MON group, or non-default CTRL_MON group
401 then RDT events for the task will be reported in that group.
403 2) If a task is a member of the default CTRL_MON group, but is running
404 on a CPU that is assigned to some specific group, then the RDT events
405 for the task will be reported in that group.
407 3) Otherwise RDT events for the task will be reported in the root level
411 Notes on cache occupancy monitoring and control
412 ===============================================
413 When moving a task from one group to another you should remember that
414 this only affects *new* cache allocations by the task. E.g. you may have
415 a task in a monitor group showing 3 MB of cache occupancy. If you move
416 to a new group and immediately check the occupancy of the old and new
417 groups you will likely see that the old group is still showing 3 MB and
418 the new group zero. When the task accesses locations still in cache from
419 before the move, the h/w does not update any counters. On a busy system
420 you will likely see the occupancy in the old group go down as cache lines
421 are evicted and re-used while the occupancy in the new group rises as
422 the task accesses memory and loads into the cache are counted based on
423 membership in the new group.
425 The same applies to cache allocation control. Moving a task to a group
426 with a smaller cache partition will not evict any cache lines. The
427 process may continue to use them from the old partition.
429 Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
430 to identify a control group and a monitoring group respectively. Each of
431 the resource groups are mapped to these IDs based on the kind of group. The
432 number of CLOSid and RMID are limited by the hardware and hence the creation of
433 a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
434 and creation of "MON" group may fail if we run out of RMIDs.
436 max_threshold_occupancy - generic concepts
437 ------------------------------------------
439 Note that an RMID once freed may not be immediately available for use as
440 the RMID is still tagged the cache lines of the previous user of RMID.
441 Hence such RMIDs are placed on limbo list and checked back if the cache
442 occupancy has gone down. If there is a time when system has a lot of
443 limbo RMIDs but which are not ready to be used, user may see an -EBUSY
446 max_threshold_occupancy is a user configurable value to determine the
447 occupancy at which an RMID can be freed.
449 Schemata files - general concepts
450 ---------------------------------
451 Each line in the file describes one resource. The line starts with
452 the name of the resource, followed by specific values to be applied
453 in each of the instances of that resource on the system.
457 On current generation systems there is one L3 cache per socket and L2
458 caches are generally just shared by the hyperthreads on a core, but this
459 isn't an architectural requirement. We could have multiple separate L3
460 caches on a socket, multiple cores could share an L2 cache. So instead
461 of using "socket" or "core" to define the set of logical cpus sharing
462 a resource we use a "Cache ID". At a given cache level this will be a
463 unique number across the whole system (but it isn't guaranteed to be a
464 contiguous sequence, there may be gaps). To find the ID for each logical
465 CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id
467 Cache Bit Masks (CBM)
468 ---------------------
469 For cache resources we describe the portion of the cache that is available
470 for allocation using a bitmask. The maximum value of the mask is defined
471 by each cpu model (and may be different for different cache levels). It
472 is found using CPUID, but is also provided in the "info" directory of
473 the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware
474 requires that these masks have all the '1' bits in a contiguous block. So
475 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
476 and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks
477 if non-contiguous 1s value is supported. On a system with a 20-bit mask
478 each bit represents 5% of the capacity of the cache. You could partition
479 the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
481 Memory bandwidth Allocation and monitoring
482 ==========================================
484 For Memory bandwidth resource, by default the user controls the resource
485 by indicating the percentage of total memory bandwidth.
487 The minimum bandwidth percentage value for each cpu model is predefined
488 and can be looked up through "info/MB/min_bandwidth". The bandwidth
489 granularity that is allocated is also dependent on the cpu model and can
490 be looked up at "info/MB/bandwidth_gran". The available bandwidth
491 control steps are: min_bw + N * bw_gran. Intermediate values are rounded
492 to the next control step available on the hardware.
494 The bandwidth throttling is a core specific mechanism on some of Intel
495 SKUs. Using a high bandwidth and a low bandwidth setting on two threads
496 sharing a core may result in both threads being throttled to use the
497 low bandwidth (see "thread_throttle_mode").
499 The fact that Memory bandwidth allocation(MBA) may be a core
500 specific mechanism where as memory bandwidth monitoring(MBM) is done at
501 the package level may lead to confusion when users try to apply control
502 via the MBA and then monitor the bandwidth to see if the controls are
503 effective. Below are such scenarios:
505 1. User may *not* see increase in actual bandwidth when percentage
506 values are increased:
508 This can occur when aggregate L2 external bandwidth is more than L3
509 external bandwidth. Consider an SKL SKU with 24 cores on a package and
510 where L2 external is 10GBps (hence aggregate L2 external bandwidth is
511 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
512 threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
513 bandwidth of 100GBps although the percentage value specified is only 50%
514 << 100%. Hence increasing the bandwidth percentage will not yield any
515 more bandwidth. This is because although the L2 external bandwidth still
516 has capacity, the L3 external bandwidth is fully used. Also note that
517 this would be dependent on number of cores the benchmark is run on.
519 2. Same bandwidth percentage may mean different actual bandwidth
520 depending on # of threads:
522 For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
523 thread, with 10% bandwidth' can consume upto 10GBps and 40GBps although
524 they have same percentage bandwidth of 10%. This is simply because as
525 threads start using more cores in an rdtgroup, the actual bandwidth may
526 increase or vary although user specified bandwidth percentage is same.
528 In order to mitigate this and make the interface more user friendly,
529 resctrl added support for specifying the bandwidth in MBps as well. The
530 kernel underneath would use a software feedback mechanism or a "Software
531 Controller(mba_sc)" which reads the actual bandwidth using MBM counters
532 and adjust the memory bandwidth percentages to ensure::
534 "actual bandwidth < user specified bandwidth".
536 By default, the schemata would take the bandwidth percentage values
537 where as user can switch to the "MBA software controller" mode using
538 a mount option 'mba_MBps'. The schemata format is specified in the below
541 L3 schemata file details (code and data prioritization disabled)
542 ----------------------------------------------------------------
543 With CDP disabled the L3 schemata format is::
545 L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
547 L3 schemata file details (CDP enabled via mount option to resctrl)
548 ------------------------------------------------------------------
549 When CDP is enabled L3 control is split into two separate resources
550 so you can specify independent masks for code and data like this::
552 L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
553 L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
555 L2 schemata file details
556 ------------------------
557 CDP is supported at L2 using the 'cdpl2' mount option. The schemata
560 L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
564 L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
565 L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
568 Memory bandwidth Allocation (default mode)
569 ------------------------------------------
571 Memory b/w domain is L3 cache.
574 MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
576 Memory bandwidth Allocation specified in MBps
577 ---------------------------------------------
579 Memory bandwidth domain is L3 cache.
582 MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;...
584 Slow Memory Bandwidth Allocation (SMBA)
585 ---------------------------------------
586 AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
587 CXL.memory is the only supported "slow" memory device. With the
588 support of SMBA, the hardware enables bandwidth allocation on
589 the slow memory devices. If there are multiple such devices in
590 the system, the throttling logic groups all the slow sources
591 together and applies the limit on them as a whole.
593 The presence of SMBA (with CXL.memory) is independent of slow memory
594 devices presence. If there are no such devices on the system, then
595 configuring SMBA will have no impact on the performance of the system.
597 The bandwidth domain for slow memory is L3 cache. Its schemata file
601 SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
603 Reading/writing the schemata file
604 ---------------------------------
605 Reading the schemata file will show the state of all resources
606 on all domains. When writing you only need to specify those values
607 which you wish to change. E.g.
611 L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
612 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
613 # echo "L3DATA:2=3c0;" > schemata
615 L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
616 L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
618 Reading/writing the schemata file (on AMD systems)
619 --------------------------------------------------
620 Reading the schemata file will show the current bandwidth limit on all
621 domains. The allocated resources are in multiples of one eighth GB/s.
622 When writing to the file, you need to specify what cache id you wish to
623 configure the bandwidth limit.
625 For example, to allocate 2GB/s limit on the first cache id:
630 MB:0=2048;1=2048;2=2048;3=2048
631 L3:0=ffff;1=ffff;2=ffff;3=ffff
633 # echo "MB:1=16" > schemata
635 MB:0=2048;1= 16;2=2048;3=2048
636 L3:0=ffff;1=ffff;2=ffff;3=ffff
638 Reading/writing the schemata file (on AMD systems) with SMBA feature
639 --------------------------------------------------------------------
640 Reading and writing the schemata file is the same as without SMBA in
643 For example, to allocate 8GB/s limit on the first cache id:
648 SMBA:0=2048;1=2048;2=2048;3=2048
649 MB:0=2048;1=2048;2=2048;3=2048
650 L3:0=ffff;1=ffff;2=ffff;3=ffff
652 # echo "SMBA:1=64" > schemata
654 SMBA:0=2048;1= 64;2=2048;3=2048
655 MB:0=2048;1=2048;2=2048;3=2048
656 L3:0=ffff;1=ffff;2=ffff;3=ffff
660 CAT enables a user to specify the amount of cache space that an
661 application can fill. Cache pseudo-locking builds on the fact that a
662 CPU can still read and write data pre-allocated outside its current
663 allocated area on a cache hit. With cache pseudo-locking, data can be
664 preloaded into a reserved portion of cache that no application can
665 fill, and from that point on will only serve cache hits. The cache
666 pseudo-locked memory is made accessible to user space where an
667 application can map it into its virtual address space and thus have
668 a region of memory with reduced average read latency.
670 The creation of a cache pseudo-locked region is triggered by a request
671 from the user to do so that is accompanied by a schemata of the region
672 to be pseudo-locked. The cache pseudo-locked region is created as follows:
674 - Create a CAT allocation CLOSNEW with a CBM matching the schemata
675 from the user of the cache region that will contain the pseudo-locked
676 memory. This region must not overlap with any current CAT allocation/CLOS
677 on the system and no future overlap with this cache region is allowed
678 while the pseudo-locked region exists.
679 - Create a contiguous region of memory of the same size as the cache
681 - Flush the cache, disable hardware prefetchers, disable preemption.
682 - Make CLOSNEW the active CLOS and touch the allocated memory to load
684 - Set the previous CLOS as active.
685 - At this point the closid CLOSNEW can be released - the cache
686 pseudo-locked region is protected as long as its CBM does not appear in
687 any CAT allocation. Even though the cache pseudo-locked region will from
688 this point on not appear in any CBM of any CLOS an application running with
689 any CLOS will be able to access the memory in the pseudo-locked region since
690 the region continues to serve cache hits.
691 - The contiguous region of memory loaded into the cache is exposed to
692 user-space as a character device.
694 Cache pseudo-locking increases the probability that data will remain
695 in the cache via carefully configuring the CAT feature and controlling
696 application behavior. There is no guarantee that data is placed in
697 cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
698 “locked” data from cache. Power management C-states may shrink or
699 power off cache. Deeper C-states will automatically be restricted on
700 pseudo-locked region creation.
702 It is required that an application using a pseudo-locked region runs
703 with affinity to the cores (or a subset of the cores) associated
704 with the cache on which the pseudo-locked region resides. A sanity check
705 within the code will not allow an application to map pseudo-locked memory
706 unless it runs with affinity to cores associated with the cache on which the
707 pseudo-locked region resides. The sanity check is only done during the
708 initial mmap() handling, there is no enforcement afterwards and the
709 application self needs to ensure it remains affine to the correct cores.
711 Pseudo-locking is accomplished in two stages:
713 1) During the first stage the system administrator allocates a portion
714 of cache that should be dedicated to pseudo-locking. At this time an
715 equivalent portion of memory is allocated, loaded into allocated
716 cache portion, and exposed as a character device.
717 2) During the second stage a user-space application maps (mmap()) the
718 pseudo-locked memory into its address space.
720 Cache Pseudo-Locking Interface
721 ------------------------------
722 A pseudo-locked region is created using the resctrl interface as follows:
724 1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
725 2) Change the new resource group's mode to "pseudo-locksetup" by writing
726 "pseudo-locksetup" to the "mode" file.
727 3) Write the schemata of the pseudo-locked region to the "schemata" file. All
728 bits within the schemata should be "unused" according to the "bit_usage"
731 On successful pseudo-locked region creation the "mode" file will contain
732 "pseudo-locked" and a new character device with the same name as the resource
733 group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
734 by user space in order to obtain access to the pseudo-locked memory region.
736 An example of cache pseudo-locked region creation and usage can be found below.
738 Cache Pseudo-Locking Debugging Interface
739 ----------------------------------------
740 The pseudo-locking debugging interface is enabled by default (if
741 CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.
743 There is no explicit way for the kernel to test if a provided memory
744 location is present in the cache. The pseudo-locking debugging interface uses
745 the tracing infrastructure to provide two ways to measure cache residency of
746 the pseudo-locked region:
748 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
749 from these measurements are best visualized using a hist trigger (see
750 example below). In this test the pseudo-locked region is traversed at
751 a stride of 32 bytes while hardware prefetchers and preemption
752 are disabled. This also provides a substitute visualization of cache
754 2) Cache hit and miss measurements using model specific precision counters if
755 available. Depending on the levels of cache on the system the pseudo_lock_l2
756 and pseudo_lock_l3 tracepoints are available.
758 When a pseudo-locked region is created a new debugfs directory is created for
759 it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
760 write-only file, pseudo_lock_measure, is present in this directory. The
761 measurement of the pseudo-locked region depends on the number written to this
765 writing "1" to the pseudo_lock_measure file will trigger the latency
766 measurement captured in the pseudo_lock_mem_latency tracepoint. See
769 writing "2" to the pseudo_lock_measure file will trigger the L2 cache
770 residency (cache hits and misses) measurement captured in the
771 pseudo_lock_l2 tracepoint. See example below.
773 writing "3" to the pseudo_lock_measure file will trigger the L3 cache
774 residency (cache hits and misses) measurement captured in the
775 pseudo_lock_l3 tracepoint.
777 All measurements are recorded with the tracing infrastructure. This requires
778 the relevant tracepoints to be enabled before the measurement is triggered.
780 Example of latency debugging interface
781 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
782 In this example a pseudo-locked region named "newlock" was created. Here is
783 how we can measure the latency in cycles of reading from this region and
784 visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
787 # :> /sys/kernel/tracing/trace
788 # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
789 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
790 # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
791 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
792 # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist
796 # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
799 { latency: 456 } hitcount: 1
800 { latency: 50 } hitcount: 83
801 { latency: 36 } hitcount: 96
802 { latency: 44 } hitcount: 174
803 { latency: 48 } hitcount: 195
804 { latency: 46 } hitcount: 262
805 { latency: 42 } hitcount: 693
806 { latency: 40 } hitcount: 3204
807 { latency: 38 } hitcount: 3484
814 Example of cache hits/misses debugging
815 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
816 In this example a pseudo-locked region named "newlock" was created on the L2
817 cache of a platform. Here is how we can obtain details of the cache hits
818 and misses using the platform's precision counters.
821 # :> /sys/kernel/tracing/trace
822 # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
823 # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
824 # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
825 # cat /sys/kernel/tracing/trace
830 # / _----=> need-resched
831 # | / _---=> hardirq/softirq
832 # || / _--=> preempt-depth
834 # TASK-PID CPU# |||| TIMESTAMP FUNCTION
836 pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0
839 Examples for RDT allocation usage
840 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
844 On a two socket machine (one L3 cache per socket) with just four bits
845 for cache bit masks, minimum b/w of 10% with a memory bandwidth
849 # mount -t resctrl resctrl /sys/fs/resctrl
852 # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
853 # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata
855 The default resource group is unmodified, so we have access to all parts
856 of all caches (its schemata file reads "L3:0=f;1=f").
858 Tasks that are under the control of group "p0" may only allocate from the
859 "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
860 Tasks in group "p1" use the "lower" 50% of cache on both sockets.
862 Similarly, tasks that are under the control of group "p0" may use a
863 maximum memory b/w of 50% on socket0 and 50% on socket 1.
864 Tasks in group "p1" may also use 50% memory b/w on both sockets.
865 Note that unlike cache masks, memory b/w cannot specify whether these
866 allocations can overlap or not. The allocations specifies the maximum
867 b/w that the group may be able to use and the system admin can configure
870 If resctrl is using the software controller (mba_sc) then user can enter the
871 max b/w in MB rather than the percentage values.
874 # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
875 # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata
877 In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
878 of 1024MB where as on socket 1 they would use 500MB.
882 Again two sockets, but this time with a more realistic 20-bit mask.
884 Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
885 processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
886 neighbors, each of the two real-time tasks exclusively occupies one quarter
887 of L3 cache on socket 0.
890 # mount -t resctrl resctrl /sys/fs/resctrl
893 First we reset the schemata for the default group so that the "upper"
894 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
897 # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata
899 Next we make a resource group for our first real time task and give
900 it access to the "top" 25% of the cache on socket 0.
904 # echo "L3:0=f8000;1=fffff" > p0/schemata
906 Finally we move our first real time task into this resource group. We
907 also use taskset(1) to ensure the task always runs on a dedicated CPU
908 on socket 0. Most uses of resource groups will also constrain which
909 processors tasks run on.
912 # echo 1234 > p0/tasks
915 Ditto for the second real time task (with the remaining 25% of cache)::
918 # echo "L3:0=7c00;1=fffff" > p1/schemata
919 # echo 5678 > p1/tasks
922 For the same 2 socket system with memory b/w resource and CAT L3 the
923 schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
926 For our first real time task this would request 20% memory b/w on socket 0.
929 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
931 For our second real time task this would request an other 20% memory b/w
935 # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
939 A single socket system which has real-time tasks running on core 4-7 and
940 non real-time workload assigned to core 0-3. The real-time tasks share text
941 and data, so a per task association is not required and due to interaction
942 with the kernel it's desired that the kernel on these cores shares L3 with
946 # mount -t resctrl resctrl /sys/fs/resctrl
949 First we reset the schemata for the default group so that the "upper"
950 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
951 cannot be used by ordinary tasks::
953 # echo "L3:0=3ff\nMB:0=50" > schemata
955 Next we make a resource group for our real time cores and give it access
956 to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
961 # echo "L3:0=ffc00\nMB:0=50" > p0/schemata
963 Finally we move core 4-7 over to the new group and make sure that the
964 kernel and the tasks running there get 50% of the cache. They should
965 also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
966 siblings and only the real time threads are scheduled on the cores 4-7.
973 The resource groups in previous examples were all in the default "shareable"
974 mode allowing sharing of their cache allocations. If one resource group
975 configures a cache allocation then nothing prevents another resource group
976 to overlap with that allocation.
978 In this example a new exclusive resource group will be created on a L2 CAT
979 system with two L2 cache instances that can be configured with an 8-bit
980 capacity bitmask. The new exclusive resource group will be configured to use
981 25% of each cache instance.
984 # mount -t resctrl resctrl /sys/fs/resctrl/
987 First, we observe that the default group is configured to allocate to all L2
993 We could attempt to create the new resource group at this point, but it will
994 fail because of the overlap with the schemata of the default group::
997 # echo 'L2:0=0x3;1=0x3' > p0/schemata
1000 # echo exclusive > p0/mode
1001 -sh: echo: write error: Invalid argument
1002 # cat info/last_cmd_status
1005 To ensure that there is no overlap with another resource group the default
1006 resource group's schemata has to change, making it possible for the new
1007 resource group to become exclusive.
1010 # echo 'L2:0=0xfc;1=0xfc' > schemata
1011 # echo exclusive > p0/mode
1015 p0/schemata:L2:0=03;1=03
1016 p0/size:L2:0=262144;1=262144
1018 A new resource group will on creation not overlap with an exclusive resource
1025 p1/schemata:L2:0=fc;1=fc
1026 p1/size:L2:0=786432;1=786432
1028 The bit_usage will reflect how the cache is used::
1030 # cat info/L2/bit_usage
1031 0=SSSSSSEE;1=SSSSSSEE
1033 A resource group cannot be forced to overlap with an exclusive resource group::
1035 # echo 'L2:0=0x1;1=0x1' > p1/schemata
1036 -sh: echo: write error: Invalid argument
1037 # cat info/last_cmd_status
1038 overlaps with exclusive group
1040 Example of Cache Pseudo-Locking
1041 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1042 Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
1043 region is exposed at /dev/pseudo_lock/newlock that can be provided to
1044 application for argument to mmap().
1047 # mount -t resctrl resctrl /sys/fs/resctrl/
1048 # cd /sys/fs/resctrl
1050 Ensure that there are bits available that can be pseudo-locked, since only
1051 unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
1052 removed from the default resource group's schemata::
1054 # cat info/L2/bit_usage
1055 0=SSSSSSSS;1=SSSSSSSS
1056 # echo 'L2:1=0xfc' > schemata
1057 # cat info/L2/bit_usage
1058 0=SSSSSSSS;1=SSSSSS00
1060 Create a new resource group that will be associated with the pseudo-locked
1061 region, indicate that it will be used for a pseudo-locked region, and
1062 configure the requested pseudo-locked region capacity bitmask::
1065 # echo pseudo-locksetup > newlock/mode
1066 # echo 'L2:1=0x3' > newlock/schemata
1068 On success the resource group's mode will change to pseudo-locked, the
1069 bit_usage will reflect the pseudo-locked region, and the character device
1070 exposing the pseudo-locked region will exist::
1074 # cat info/L2/bit_usage
1075 0=SSSSSSSS;1=SSSSSSPP
1076 # ls -l /dev/pseudo_lock/newlock
1077 crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock
1082 * Example code to access one page of pseudo-locked cache region
1091 #include <sys/mman.h>
1094 * It is required that the application runs with affinity to only
1095 * cores associated with the pseudo-locked region. Here the cpu
1096 * is hardcoded for convenience of example.
1098 static int cpuid = 2;
1100 int main(int argc, char *argv[])
1108 page_size = sysconf(_SC_PAGESIZE);
1111 CPU_SET(cpuid, &cpuset);
1112 ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
1114 perror("sched_setaffinity");
1118 dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
1124 mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
1126 if (mapping == MAP_FAILED) {
1132 /* Application interacts with pseudo-locked memory @mapping */
1134 ret = munmap(mapping, page_size);
1145 Locking between applications
1146 ----------------------------
1148 Certain operations on the resctrl filesystem, composed of read/writes
1149 to/from multiple files, must be atomic.
1151 As an example, the allocation of an exclusive reservation of L3 cache
1154 1. Read the cbmmasks from each directory or the per-resource "bit_usage"
1155 2. Find a contiguous set of bits in the global CBM bitmask that is clear
1156 in any of the directory cbmmasks
1157 3. Create a new directory
1158 4. Set the bits found in step 2 to the new directory "schemata" file
1160 If two applications attempt to allocate space concurrently then they can
1161 end up allocating the same bits so the reservations are shared instead of
1164 To coordinate atomic operations on the resctrlfs and to avoid the problem
1165 above, the following locking procedure is recommended:
1167 Locking is based on flock, which is available in libc and also as a shell
1172 A) Take flock(LOCK_EX) on /sys/fs/resctrl
1173 B) Read/write the directory structure.
1178 A) Take flock(LOCK_SH) on /sys/fs/resctrl
1179 B) If success read the directory structure.
1184 # Atomically read directory structure
1185 $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
1187 # Read directory contents and create new subdirectory
1190 find /sys/fs/resctrl/ > output.txt
1191 mask = function-of(output.txt)
1192 mkdir /sys/fs/resctrl/newres/
1193 echo mask > /sys/fs/resctrl/newres/schemata
1195 $ flock /sys/fs/resctrl/ ./create-dir.sh
1200 * Example code do take advisory locks
1201 * before accessing resctrl filesystem
1203 #include <sys/file.h>
1206 void resctrl_take_shared_lock(int fd)
1210 /* take shared lock on resctrl filesystem */
1211 ret = flock(fd, LOCK_SH);
1218 void resctrl_take_exclusive_lock(int fd)
1222 /* release lock on resctrl filesystem */
1223 ret = flock(fd, LOCK_EX);
1230 void resctrl_release_lock(int fd)
1234 /* take shared lock on resctrl filesystem */
1235 ret = flock(fd, LOCK_UN);
1246 fd = open("/sys/fs/resctrl", O_DIRECTORY);
1251 resctrl_take_shared_lock(fd);
1252 /* code to read directory contents */
1253 resctrl_release_lock(fd);
1255 resctrl_take_exclusive_lock(fd);
1256 /* code to read and write directory contents */
1257 resctrl_release_lock(fd);
1260 Examples for RDT Monitoring along with allocation usage
1261 =======================================================
1262 Reading monitored data
1263 ----------------------
1264 Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
1265 show the current snapshot of LLC occupancy of the corresponding MON
1266 group or CTRL_MON group.
1269 Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
1270 ------------------------------------------------------------------------
1271 On a two socket machine (one L3 cache per socket) with just four bits
1272 for cache bit masks::
1274 # mount -t resctrl resctrl /sys/fs/resctrl
1275 # cd /sys/fs/resctrl
1277 # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
1278 # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
1279 # echo 5678 > p1/tasks
1280 # echo 5679 > p1/tasks
1282 The default resource group is unmodified, so we have access to all parts
1283 of all caches (its schemata file reads "L3:0=f;1=f").
1285 Tasks that are under the control of group "p0" may only allocate from the
1286 "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
1287 Tasks in group "p1" use the "lower" 50% of cache on both sockets.
1289 Create monitor groups and assign a subset of tasks to each monitor group.
1292 # cd /sys/fs/resctrl/p1/mon_groups
1294 # echo 5678 > m11/tasks
1295 # echo 5679 > m12/tasks
1297 fetch data (data shown in bytes)
1300 # cat m11/mon_data/mon_L3_00/llc_occupancy
1302 # cat m11/mon_data/mon_L3_01/llc_occupancy
1304 # cat m12/mon_data/mon_L3_00/llc_occupancy
1307 The parent ctrl_mon group shows the aggregated data.
1310 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
1313 Example 2 (Monitor a task from its creation)
1314 --------------------------------------------
1315 On a two socket machine (one L3 cache per socket)::
1317 # mount -t resctrl resctrl /sys/fs/resctrl
1318 # cd /sys/fs/resctrl
1321 An RMID is allocated to the group once its created and hence the <cmd>
1322 below is monitored from its creation.
1325 # echo $$ > /sys/fs/resctrl/p1/tasks
1330 # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
1333 Example 3 (Monitor without CAT support or before creating CAT groups)
1334 ---------------------------------------------------------------------
1336 Assume a system like HSW has only CQM and no CAT support. In this case
1337 the resctrl will still mount but cannot create CTRL_MON directories.
1338 But user can create different MON groups within the root group thereby
1339 able to monitor all tasks including kernel threads.
1341 This can also be used to profile jobs cache size footprint before being
1342 able to allocate them to different allocation groups.
1345 # mount -t resctrl resctrl /sys/fs/resctrl
1346 # cd /sys/fs/resctrl
1347 # mkdir mon_groups/m01
1348 # mkdir mon_groups/m02
1350 # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
1351 # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks
1353 Monitor the groups separately and also get per domain data. From the
1354 below its apparent that the tasks are mostly doing work on
1358 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
1360 # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
1362 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
1364 # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
1368 Example 4 (Monitor real time tasks)
1369 -----------------------------------
1371 A single socket system which has real time tasks running on cores 4-7
1372 and non real time tasks on other cpus. We want to monitor the cache
1373 occupancy of the real time threads on these cores.
1376 # mount -t resctrl resctrl /sys/fs/resctrl
1377 # cd /sys/fs/resctrl
1380 Move the cpus 4-7 over to p1::
1384 View the llc occupancy snapshot::
1386 # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
1392 Intel MBM Counters May Report System Memory Bandwidth Incorrectly
1393 -----------------------------------------------------------------
1395 Errata SKX99 for Skylake server and BDF102 for Broadwell server.
1397 Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics
1398 according to the assigned Resource Monitor ID (RMID) for that logical
1399 core. The IA32_QM_CTR register (MSR 0xC8E), used to report these
1400 metrics, may report incorrect system bandwidth for certain RMID values.
1402 Implication: Due to the errata, system memory bandwidth may not match
1405 Workaround: MBM total and local readings are corrected according to the
1406 following correction factor table:
1408 +---------------+---------------+---------------+-----------------+
1409 |core count |rmid count |rmid threshold |correction factor|
1410 +---------------+---------------+---------------+-----------------+
1411 |1 |8 |0 |1.000000 |
1412 +---------------+---------------+---------------+-----------------+
1413 |2 |16 |0 |1.000000 |
1414 +---------------+---------------+---------------+-----------------+
1415 |3 |24 |15 |0.969650 |
1416 +---------------+---------------+---------------+-----------------+
1417 |4 |32 |0 |1.000000 |
1418 +---------------+---------------+---------------+-----------------+
1419 |6 |48 |31 |0.969650 |
1420 +---------------+---------------+---------------+-----------------+
1421 |7 |56 |47 |1.142857 |
1422 +---------------+---------------+---------------+-----------------+
1423 |8 |64 |0 |1.000000 |
1424 +---------------+---------------+---------------+-----------------+
1425 |9 |72 |63 |1.185115 |
1426 +---------------+---------------+---------------+-----------------+
1427 |10 |80 |63 |1.066553 |
1428 +---------------+---------------+---------------+-----------------+
1429 |11 |88 |79 |1.454545 |
1430 +---------------+---------------+---------------+-----------------+
1431 |12 |96 |0 |1.000000 |
1432 +---------------+---------------+---------------+-----------------+
1433 |13 |104 |95 |1.230769 |
1434 +---------------+---------------+---------------+-----------------+
1435 |14 |112 |95 |1.142857 |
1436 +---------------+---------------+---------------+-----------------+
1437 |15 |120 |95 |1.066667 |
1438 +---------------+---------------+---------------+-----------------+
1439 |16 |128 |0 |1.000000 |
1440 +---------------+---------------+---------------+-----------------+
1441 |17 |136 |127 |1.254863 |
1442 +---------------+---------------+---------------+-----------------+
1443 |18 |144 |127 |1.185255 |
1444 +---------------+---------------+---------------+-----------------+
1445 |19 |152 |0 |1.000000 |
1446 +---------------+---------------+---------------+-----------------+
1447 |20 |160 |127 |1.066667 |
1448 +---------------+---------------+---------------+-----------------+
1449 |21 |168 |0 |1.000000 |
1450 +---------------+---------------+---------------+-----------------+
1451 |22 |176 |159 |1.454334 |
1452 +---------------+---------------+---------------+-----------------+
1453 |23 |184 |0 |1.000000 |
1454 +---------------+---------------+---------------+-----------------+
1455 |24 |192 |127 |0.969744 |
1456 +---------------+---------------+---------------+-----------------+
1457 |25 |200 |191 |1.280246 |
1458 +---------------+---------------+---------------+-----------------+
1459 |26 |208 |191 |1.230921 |
1460 +---------------+---------------+---------------+-----------------+
1461 |27 |216 |0 |1.000000 |
1462 +---------------+---------------+---------------+-----------------+
1463 |28 |224 |191 |1.143118 |
1464 +---------------+---------------+---------------+-----------------+
1466 If rmid > rmid threshold, MBM total and local values should be multiplied
1467 by the correction factor.
1471 1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
1472 http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
1474 2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
1475 http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
1477 3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
1478 https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html
1480 for further information.