1 ====================================================================
2 Interaction of Suspend code (S3) with the CPU hotplug infrastructure
3 ====================================================================
5 (C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
8 I. Differences between CPU hotplug and Suspend-to-RAM
9 ======================================================
11 How does the regular CPU hotplug code differ from how the Suspend-to-RAM
12 infrastructure uses it internally? And where do they share common code?
14 Well, a picture is worth a thousand words... So ASCII art follows :-)
16 [This depicts the current design in the kernel, and focusses only on the
17 interactions involving the freezer and CPU hotplug and also tries to explain
18 the locking involved. It outlines the notifications involved as well.
19 But please note that here, only the call paths are illustrated, with the aim
20 of describing where they take different paths and where they share code.
21 What happens when regular CPU hotplug and Suspend-to-RAM race with each other
22 is not depicted here.]
24 On a high level, the suspend-resume cycle goes like this::
26 |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw |
27 |tasks | | cpus | | | | cpus | |tasks|
40 Acquire system_transition_mutex lock
43 Send PM_SUSPEND_PREPARE
51 disable_nonboot_cpus()
55 Acquire cpu_add_remove_lock
58 Iterate over CURRENTLY
65 | [This takes cpuhotplug.lock |
66 Common | before taking down the CPU |
67 code | and releases it when done] | O
68 | While it is at it, notifications |
69 | are sent when notable events occur, |
70 ======> by running all registered callbacks. |
75 Note down these cpus in | P
76 frozen_cpus mask ----------
79 Disable regular cpu hotplug
80 by increasing cpu_hotplug_disabled
83 Release cpu_add_remove_lock
86 /* disable_nonboot_cpus() complete */
93 Resuming back is likewise, with the counterparts being (in the order of
94 execution during resume):
96 * enable_nonboot_cpus() which involves::
98 | Acquire cpu_add_remove_lock
99 | Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
100 | Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
101 | Release cpu_add_remove_lock
105 * send PM_POST_SUSPEND notifications
106 * Release system_transition_mutex lock.
109 It is to be noted here that the system_transition_mutex lock is acquired at the very
110 beginning, when we are just starting out to suspend, and then released only
111 after the entire cycle is complete (i.e., suspend + resume).
117 Regular CPU hotplug call path
118 -----------------------------
121 /sys/devices/system/cpu/cpu*/online
129 Acquire cpu_add_remove_lock
132 If cpu_hotplug_disabled > 0
138 | [This takes cpuhotplug.lock
139 Common | before taking down the CPU
140 code | and releases it when done]
141 | While it is at it, notifications
142 | are sent when notable events occur,
143 ======> by running all registered callbacks.
147 Release cpu_add_remove_lock
153 So, as can be seen from the two diagrams (the parts marked as "Common code"),
154 regular CPU hotplug and the suspend code path converge at the _cpu_down() and
155 _cpu_up() functions. They differ in the arguments passed to these functions,
156 in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen'
157 argument. But during suspend, since the tasks are already frozen by the time
158 the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called
159 with the 'tasks_frozen' argument set to 1.
160 [See below for some known issues regarding this.]
163 Important files and functions/entry points:
164 -------------------------------------------
166 - kernel/power/process.c : freeze_processes(), thaw_processes()
167 - kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish()
168 - kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](), [disable|enable]_nonboot_cpus()
172 II. What are the issues involved in CPU hotplug?
173 ------------------------------------------------
175 There are some interesting situations involving CPU hotplug and microcode
176 update on the CPUs, as discussed below:
178 [Please bear in mind that the kernel requests the microcode images from
179 userspace, using the request_firmware() function defined in
180 drivers/base/firmware_loader/main.c]
183 a. When all the CPUs are identical:
185 This is the most common situation and it is quite straightforward: we want
186 to apply the same microcode revision to each of the CPUs.
187 To give an example of x86, the collect_cpu_info() function defined in
188 arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU
189 and thereby in applying the correct microcode revision to it.
190 But note that the kernel does not maintain a common microcode image for the
191 all CPUs, in order to handle case 'b' described below.
194 b. When some of the CPUs are different than the rest:
196 In this case since we probably need to apply different microcode revisions
197 to different CPUs, the kernel maintains a copy of the correct microcode
198 image for each CPU (after appropriate CPU type/model discovery using
199 functions such as collect_cpu_info()).
202 c. When a CPU is physically hot-unplugged and a new (and possibly different
203 type of) CPU is hot-plugged into the system:
205 In the current design of the kernel, whenever a CPU is taken offline during
206 a regular CPU hotplug operation, upon receiving the CPU_DEAD notification
207 (which is sent by the CPU hotplug code), the microcode update driver's
208 callback for that event reacts by freeing the kernel's copy of the
209 microcode image for that CPU.
211 Hence, when a new CPU is brought online, since the kernel finds that it
212 doesn't have the microcode image, it does the CPU type/model discovery
213 afresh and then requests the userspace for the appropriate microcode image
214 for that CPU, which is subsequently applied.
216 For example, in x86, the mc_cpu_callback() function (which is the microcode
217 update driver's callback registered for CPU hotplug events) calls
218 microcode_update_cpu() which would call microcode_init_cpu() in this case,
219 instead of microcode_resume_cpu() when it finds that the kernel doesn't
220 have a valid microcode image. This ensures that the CPU type/model
221 discovery is performed and the right microcode is applied to the CPU after
222 getting it from userspace.
225 d. Handling microcode update during suspend/hibernate:
227 Strictly speaking, during a CPU hotplug operation which does not involve
228 physically removing or inserting CPUs, the CPUs are not actually powered
229 off during a CPU offline. They are just put to the lowest C-states possible.
230 Hence, in such a case, it is not really necessary to re-apply microcode
231 when the CPUs are brought back online, since they wouldn't have lost the
232 image during the CPU offline operation.
234 This is the usual scenario encountered during a resume after a suspend.
235 However, in the case of hibernation, since all the CPUs are completely
236 powered off, during restore it becomes necessary to apply the microcode
237 images to all the CPUs.
239 [Note that we don't expect someone to physically pull out nodes and insert
240 nodes with a different type of CPUs in-between a suspend-resume or a
241 hibernate/restore cycle.]
243 In the current design of the kernel however, during a CPU offline operation
244 as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set),
245 the existing copy of microcode image in the kernel is not freed up.
246 And during the CPU online operations (during resume/restore), since the
247 kernel finds that it already has copies of the microcode images for all the
248 CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU
249 type/model and the need for validating whether the microcode revisions are
250 right for the CPUs or not (due to the above assumption that physical CPU
251 hotplug will not be done in-between suspend/resume or hibernate/restore
258 Are there any known problems when regular CPU hotplug and suspend race
261 Yes, they are listed below:
263 1. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to
264 the _cpu_down() and _cpu_up() functions is *always* 0.
265 This might not reflect the true current state of the system, since the
266 tasks could have been frozen by an out-of-band event such as a suspend
267 operation in progress. Hence, the cpuhp_tasks_frozen variable will not
268 reflect the frozen state and the CPU hotplug callbacks which evaluate
269 that variable might execute the wrong code path.
271 2. If a regular CPU hotplug stress test happens to race with the freezer due
272 to a suspend operation in progress at the same time, then we could hit the
273 situation described below:
275 * A regular cpu online operation continues its journey from userspace
276 into the kernel, since the freezing has not yet begun.
277 * Then freezer gets to work and freezes userspace.
278 * If cpu online has not yet completed the microcode update stuff by now,
279 it will now start waiting on the frozen userspace in the
280 TASK_UNINTERRUPTIBLE state, in order to get the microcode image.
281 * Now the freezer continues and tries to freeze the remaining tasks. But
282 due to this wait mentioned above, the freezer won't be able to freeze
283 the cpu online hotplug task and hence freezing of tasks fails.
285 As a result of this task freezing failure, the suspend operation gets