Testing:
* Deployed ISO with changes.
* Configured kube-cpu-mgr-policy=static.
* Verified that metrics-server were running on platform CPUs.
Partial-Bug: 1964503
Signed-off-by: Thiago Miranda <ThiagoOliveira.Miranda@windriver.com>
Change-Id: I9bebe9ec27fcd70e89a4cae52bfacde993f958eb
When executing a reboot/shutdown
k8s pods are not receiving the SIGTERM
signal which leads some of them to
unexpected behaviour such as generating
huge coredumps.
There is an upstream issue regarding this:
https://github.com/kubernetes/kubernetes/issues/107158
The problem seems to be systemd related
but this commit addresses the problem
with a workaround.
This commit introduces a new script that
will cleanup all the remaing pods and will
be run after kubelet is stopped.
The script is executed successfully when
kubelet stops and the pods are stopped
before the system shuts down.
Closes-bug: 1964111
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
Change-Id: Ia0376aa510dd0dc3983e16cd89840726c15d6c92
The script will run everytime before the kubelet service is started.
It reads the reserved-cpus list for the kubelet from the service
environment file and sanitizes it on the basis of online CPUs.
If none of the reserved cpus is online, it removes the
--reserved-cpus flag from the environment file which allows
the kubelet to choose CPUs itself.
Sanitizing the reserved-cpus list everytime before the kubelet starts
assures that the kubelet will not fail to start due to unavailability
of one or more CPUs in the list.
By enabling or disabling CPU hyperthreading, available CPUs change.
This change will make sure changing CPU hyperthreading setting will
not lead to kubelet start failure after the system boots up.
Test Plan: (On AIO-SX)
PASS:
Initial Hyperthreading state: enabled
Host-lock->Reboot->Disable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Initial Hyperthreading state: disabled
Host-lock->Reboot->Enable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Manually restart the Kubelet service.
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Host-lock->Host unlock (without any config change).
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Packages built successfully on both Debian and CentOS.
Closes-Bug: 1955608
Change-Id: I699c5c36a56a50d4c48faa816edad69c17058079
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
The existing device manager code returns CPUs as devices in unsorted
order. This numerically sorts isolcpus allocations when SMT/HT is
enabled on the host. This logs SMT pairs, singletons, and algorithm
order details to make the algorithm understandable.
Example log for a 3 cpu isolcpus request:
2022-02-11T16:27:50.345 controller-0 kubelet[1531574]: info I0211
16:27:50.345529 1531574 manager.go:741] order_devices_by_sibling:
needed=3, smtpairs=[4 5 6 7 10 11], singletons=[8 12],
order=[8 4 5 6 7 10 11 12]
The specific host with SMT enabled has this topology:
LOGICAL CPU TOPOLOGY:
cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
core_id : 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7
thread_id : 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Before cpu allocation, host has Isolated_free: 4-8,10-12.
New pod gets the following isolcpus cpuset: 4-5,8.
Test Plan: (On AIO-SX, SMT enabled)
PASS: Verify cpu sort order for even needed and no singletons
PASS: Verify cpu sort order for odd needed and no singletons
PASS: Verify cpu sort order for even needed and singletons
PASS: Verify cpu sort order for odd needed and singletons
Story: 2008760
Task: 44190
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I1d743f80925b35ecee7936c12b0f4328f83b7eb2
Enhance isolcpus support in Kubernetes to allocate isolated SMT
siblings to the same container when SMT/HT is enabled on the host.
As it stands, the device manager code in Kubernetes is not SMT-aware
(since normally it doesn't deal with CPUs). However, StarlingX
exposes isolated CPUs as devices and if possible we want to allocate
all SMT siblings from a CPU core to the same container in order to
minimize cross- container interference due to resource contention
within the CPU core.
The solution is basically to take the list of isolated CPUs and
re-order it so that the SMT siblings are next to each other. That
way the existing resource selection code will allocate the siblings
together. As an optimization, if it is known that an odd number
of isolated CPUs are desired, a singleton SMT sibling will be
inserted into the list to avoid breaking up sibling pairs.
Test Plan:
Tested with AIO-SX HT enabled and disabled (disabled will not
run the code). Test results below are with HT enabled.
Platform cpu layout:
Socket 0 Socket 1
-------- --------
Core 0 [0, 16] [8, 24]
Core 1 [1, 17] [9, 25]
Core 2 [2, 18] [10, 26]
Core 3 [3, 19] [11, 27]
Core 4 [4, 20] [12, 28]
Core 5 [5, 21] [13, 29]
Core 6 [6, 22] [14, 30]
Core 7 [7, 23] [15, 31]
isolcpus=2-3,8-9,18-19,24-25
Ct: container
InCt: initcontainer
U: isolcpus
Test command in pod:cat /sys/fs/cgroup/cpuset/cpuset.cpus
Case 1: 1Ct_3U,got [2-3,19],Passed
Case 2: Keep case 1;create 1Ct_2U,got [9,25],Passed
Case 3: Keep case 1,2;create 1Ct_1U got[18],
create another 1Ct_2U got [8,24],Passed
Case 4: Reboot after case 3;Pods keep cpu as above.Passed
Case 5: Clean All;create 2Ct_3U by one replicaset,pod_1[8-9,24],
pod_2[3,18-19],Passed
Case 6: Keep case 5;create 1Ct_2U,got [2,25],[2,25] is the last
two non-siblings isocpus,as expected,Passed
Case 7: Clean All;create 2InCt_1Ct_2U,InCt_1 got[2,18],
InCt_2 got[2,18],Pod got [2,18],Passed
Case 8: Clean All;create 2InCt_2Ct_2U, Pod1_InCt_1 got[3,19],
Pod1_InCt_2 got[3,19], Pod1 got [3,19],Pod2_InCt_1 got[2,18],
Pod2_InCt_2 got[2,18], Pod2 got [2,18],Cpu in initcontainer
is tested by write file to pvc,Passed
Story: 2008760
Task: 44190
Change-Id: I8bd03352cc395bada9126fb0fce8ed268ac36456
Signed-off-by: Tao Wang <tao.wang@windriver.com>
Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
the newer version.
This has been booted in vbox and a basic pod has been started.
A full regression will be performed.
Depends-On: https://review.opendev.org/c/starlingx/compile/+/824802
Partial-Bug: 1957994
Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
Change-Id: I64e4a64c90ef7591aeee52742dfcba9fdd8e5063