The new minimum supported k8s version
will be 1.21. This commit cleans the pkg
files needed to build the old k8s versions.
The pkgs build successfully. Deployed on
AIO-SX and AIO-DX, the k8s services were running ok.
Story: 2009859
Task: 44498
Change-Id: Ib39e9d1522a49c5788240781c8edee2bdffbc97a
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
By removing:
User=docker-registry
the service now runs as user root which allows ansible to use the
registry without hitting permissions issues.
Test Plan:
Pass: Execute ansible playbook in Debian OS
Pass: Service is active (running) after ansible playbook
Story: 2009101
Task: 44419
Signed-off-by: Fabricio Henrique Ramos <fabriciohenrique.ramos@windriver.com>
Change-Id: I06f3b0ed19d60400630bd01d3ae115fe44b6582b
The script will run everytime before the kubelet service is started.
It reads the reserved-cpus list for the kubelet from the service
environment file and sanitizes it on the basis of online CPUs.
If none of the reserved cpus is online, it removes the
--reserved-cpus flag from the environment file which allows
the kubelet to choose CPUs itself.
Sanitizing the reserved-cpus list everytime before the kubelet starts
assures that the kubelet will not fail to start due to unavailability
of one or more CPUs in the list.
By enabling or disabling CPU hyperthreading, available CPUs change.
This change will make sure changing CPU hyperthreading setting will
not lead to kubelet start failure after the system boots up.
Test Plan: (On AIO-SX)
PASS:
Initial Hyperthreading state: enabled
Host-lock->Reboot->Disable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Initial Hyperthreading state: disabled
Host-lock->Reboot->Enable CPU hyperthreading and reboot->Host-unlock
Observe kubelet does not fail to start before host-unlock.
All pods states are as expected. Host-unlock succeeds.
PASS:
Manually restart the Kubelet service.
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Host-lock->Host unlock (without any config change).
Observe that the kubelet does not fail to start.
All pods states are as expected.
PASS:
Packages built successfully on both Debian and CentOS.
Closes-Bug: 1955608
Change-Id: I699c5c36a56a50d4c48faa816edad69c17058079
Signed-off-by: Kaustubh Dhokte <kaustubh.dhokte@windriver.com>
The existing device manager code returns CPUs as devices in unsorted
order. This numerically sorts isolcpus allocations when SMT/HT is
enabled on the host. This logs SMT pairs, singletons, and algorithm
order details to make the algorithm understandable.
Example log for a 3 cpu isolcpus request:
2022-02-11T16:27:50.345 controller-0 kubelet[1531574]: info I0211
16:27:50.345529 1531574 manager.go:741] order_devices_by_sibling:
needed=3, smtpairs=[4 5 6 7 10 11], singletons=[8 12],
order=[8 4 5 6 7 10 11 12]
The specific host with SMT enabled has this topology:
LOGICAL CPU TOPOLOGY:
cpu_id : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
socket_id : 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
core_id : 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7
thread_id : 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Before cpu allocation, host has Isolated_free: 4-8,10-12.
New pod gets the following isolcpus cpuset: 4-5,8.
Test Plan: (On AIO-SX, SMT enabled)
PASS: Verify cpu sort order for even needed and no singletons
PASS: Verify cpu sort order for odd needed and no singletons
PASS: Verify cpu sort order for even needed and singletons
PASS: Verify cpu sort order for odd needed and singletons
Story: 2008760
Task: 44190
Signed-off-by: Jim Gauld <james.gauld@windriver.com>
Change-Id: I1d743f80925b35ecee7936c12b0f4328f83b7eb2
Enhance isolcpus support in Kubernetes to allocate isolated SMT
siblings to the same container when SMT/HT is enabled on the host.
As it stands, the device manager code in Kubernetes is not SMT-aware
(since normally it doesn't deal with CPUs). However, StarlingX
exposes isolated CPUs as devices and if possible we want to allocate
all SMT siblings from a CPU core to the same container in order to
minimize cross- container interference due to resource contention
within the CPU core.
The solution is basically to take the list of isolated CPUs and
re-order it so that the SMT siblings are next to each other. That
way the existing resource selection code will allocate the siblings
together. As an optimization, if it is known that an odd number
of isolated CPUs are desired, a singleton SMT sibling will be
inserted into the list to avoid breaking up sibling pairs.
Test Plan:
Tested with AIO-SX HT enabled and disabled (disabled will not
run the code). Test results below are with HT enabled.
Platform cpu layout:
Socket 0 Socket 1
-------- --------
Core 0 [0, 16] [8, 24]
Core 1 [1, 17] [9, 25]
Core 2 [2, 18] [10, 26]
Core 3 [3, 19] [11, 27]
Core 4 [4, 20] [12, 28]
Core 5 [5, 21] [13, 29]
Core 6 [6, 22] [14, 30]
Core 7 [7, 23] [15, 31]
isolcpus=2-3,8-9,18-19,24-25
Ct: container
InCt: initcontainer
U: isolcpus
Test command in pod:cat /sys/fs/cgroup/cpuset/cpuset.cpus
Case 1: 1Ct_3U,got [2-3,19],Passed
Case 2: Keep case 1;create 1Ct_2U,got [9,25],Passed
Case 3: Keep case 1,2;create 1Ct_1U got[18],
create another 1Ct_2U got [8,24],Passed
Case 4: Reboot after case 3;Pods keep cpu as above.Passed
Case 5: Clean All;create 2Ct_3U by one replicaset,pod_1[8-9,24],
pod_2[3,18-19],Passed
Case 6: Keep case 5;create 1Ct_2U,got [2,25],[2,25] is the last
two non-siblings isocpus,as expected,Passed
Case 7: Clean All;create 2InCt_1Ct_2U,InCt_1 got[2,18],
InCt_2 got[2,18],Pod got [2,18],Passed
Case 8: Clean All;create 2InCt_2Ct_2U, Pod1_InCt_1 got[3,19],
Pod1_InCt_2 got[3,19], Pod1 got [3,19],Pod2_InCt_1 got[2,18],
Pod2_InCt_2 got[2,18], Pod2 got [2,18],Cpu in initcontainer
is tested by write file to pvc,Passed
Story: 2008760
Task: 44190
Change-Id: I8bd03352cc395bada9126fb0fce8ed268ac36456
Signed-off-by: Tao Wang <tao.wang@windriver.com>
Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
This commit adds the bonding CNI plugin to StarlingX.
The bonding CNI plugin allows a container to bond multiple
interfaces together to be used in a fail-over or load
balancing configuration.
https://github.com/k8snetworkplumbingwg/bond-cni
Note that this plugin (for now) resides outside of the
main containernetwork project, and is still part of the
k8s network plumbing working group project. As such,
it is required to build this plugin separately.
v1.0 of the bond-cni was released in 2018. Since then,
14 commits containing such things as doc clean-ups and
bug fixes have been committed. We pick up these additional
fixes by clamping down on the latest commit SHA.
Testing:
- Configure bond interface name (ifName)
- Configure miimon value (miimon)
- Configure and verify traffic path for modes (mode):
- balance-rr (0)
- active-backup (1)
- balance-xor (2)
- broadcast (3)
- 802.3ad (4)
- balance-tlb (5)
- balance-alb (6)
- Configure and verify behaviour for fail-over-mac
modes (failOverMac):
- none (0)
- active (1)
- follow (2)
- Configure linksInContainer:
- take lower interfaces existing on host
- take lower interfaces existing on container
- Links tested:
- virtual interfaces
- SR-IOV VF interfaces
Story: 2009800
Task: 44344
Change-Id: I7bffaa272ffe9eba85c3aa0a26b9c4f61428b640
Signed-off-by: Steven Webster <steven.webster@windriver.com>
helm-upload was located on /usr/sbin while the puppet
module was expecting it to be located in /usr/local/sbin.
Put it in the place where the puppet module was expecting it.
Test Plan:
Pass Build helm with correct paths.
Pass Check to see if helm-upload is located in /usr/local/sbin.
Story: 2009101
Task: 43715
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I0dc22b4e5888cbfd26728588d0c09a5a47bd3305
Here are the changes needed for adding k8s v1.22.5
in StarlingX alongside with the changes needed
for the build environment to find and build the package.
The package builds successfully.
Deployed an iso with k8s 1.22.5 on
AIO-SX and AIO-DX. The deployment phase
works and the pods are up and running after
the upgrade completes.
Story: 2009789
Task: 44305
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
Change-Id: Ibb9be075fa0b1491b9ab1854ebb1fddf4df53461
Fix lintian errors while building the package. Also add missing
source/format file.
Test Plan:
PASS Build package succesfully
PASS Check for Lintian errors
Story: 2009221
Task: 43917
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I4cca2497d15a8ecef56936fc4e928dec19b699be
When the node is bootstrapped via the ansible playbooks, the
client certs are stored in the /etc/etcd directory, which is
not created by the debian package. This causes the ansible
playbook to fail.
Create the /etc/etcd directory in the etc-server.postinst
script when the server is installed.
Testing:
PASS: Package installed and ISO built successfully
Story: 2009221
Task: 43521
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I10fbf0bc34a0d65ec75ef9237f12cf63ce1b8c50
In testing K8s 1.21.8 used less CPU than 1.21.3, so we are moving to
the newer version.
This has been booted in vbox and a basic pod has been started.
A full regression will be performed.
Depends-On: https://review.opendev.org/c/starlingx/compile/+/824802
Partial-Bug: 1957994
Signed-off-by: Chris Friesen <chris.friesen@windriver.com>
Change-Id: I64e4a64c90ef7591aeee52742dfcba9fdd8e5063
"dl_path" to replace
tar xfz ${ARMADA_PKG}
cp -pr ${PKG}/charts ${PKG_BUILD_ROOT}
debrepack extracts the ${ARMADA_PKG} as ${PKG_BUILD_ROOT},
so no need cp -pr ${PKG}/charts ${PKG_BUILD_ROOT}.
And move the patches into deb_folder
Test Plan:
Pass: successfully build test
Story: 2009221
Task: 43917
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: I041bbed3dff41ed7e63c99ddfef14c206628436e
Modify the debian/rules to Prevent the docker-registry from
starting when the package is installed. This causes a conflict
when keystone and docker-registry is trying to start at the same time.
Tested by building package, checked on image docker-registry
started or not.
Story: 2009221
Task: 43631
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: I8c5016dfbb8f1508ac1c0a1864977a00d9d4072d
Remove the the gecos option when running the adduser command.
This will cause the following issue when running build-image:
Setting up docker-registry (2.7.1+ds2-7.stx.1) ...
Adding system user `docker-registry' (UID 108) ...
Adding new group `docker-registry' (GID 114) ...
Adding new user `docker-registry' (UID 108) with group `docker-registry' ...
chfn: PAM: Critical error - immediate abort
adduser: `/bin/chfn -f Docker Registry docker-registry'
returned error code 1. Exiting.
dpkg: error processing package docker-registry (--configure):
installed docker-registry package post-installation script
subprocess returned error exit status 1
This is due to the fact that debootstrap does not like
to run chfn since its setuid.
Testing: Apply the patch and run the build-image. The
build-image process should not fail with the above error.
Story: 2009221
Task: 43631
Signed-off-by: Charles Short <charles.short@windriver.com>
Change-Id: Iefe9465b17543627e86fb6797eae678359541b34
"dl_files" replaces the ${HELM_SOURCE} in dl_hook
"src_path" replaces the ${SRC} in dl_hook
debrepack deploys the 2 fields to build directory,
so no longer need dl_hook
Test Plan:
Pass: successfully build test
Story: 2009101
Task: 43715
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: Icae8471dc5480f8176946143361e65f626ca5fef
We now support multiple kubernetes versions,
and the common configuration/environment
files have been moved to a new package named
kubernetes-unversioned in CentOS
https://review.opendev.org/c/starlingx/integ/+/804243.
This commit will port kubernetes-unversioned to Debian.
- Did build-pkg
- Did build-img
- Verified if the package was installed correctly
- Verified the permissions of the installed files
Story: 2009101
Task: 44008
Change-Id: I65a9817d7f09736919453c0e81221f1fa1c590f5
Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>
"dl_path" replaces the ${CM_SOURCE} in dl_hook
"dl_files" replaces the ${CM_EXEC} in dl_hook
debrepack extracts the "dl_path" automatically and
copies the "dl_files" to build directory, so no longer
need the dl_hook, and remove the untar operation in
debian/rules.
Test Plan:
Pass: successfully build test
Story: 2009101
Task: 43715
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: Id11fa62cc510808899ba79f5f2a44cd65bd97109
The "dl_path" is used to set the downloading information of the source
tar ball of a package, and debrepack module can extract the tar ball by
parsing the meta_data.yaml automatically, that is equal to "dl_hook"
scripts.
Test Plan:
Pass: successfully build test
Story: 2009221
Task: 43917
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: Ie70517814a16852a9ab20495c2fc51622acd8e2f
The script will run for every k8s upgrade as a part of the
control-plane upgrade of the first master.
It updates kubeadm-config configmap to configure kube-apiserver
manifest with RemoveSelfLink=false.
The kubelet config override file is consumed by
kubeadm upgrade apply to configure cgroupDriver=cgroupfs.
Kubernetes changed default cgroupDriver cgroupfs to systemd
from k8s 1.21. We need to configure cgroupDriver with 'cgroupfs'
in the kubelet config otherwise kubelet will fail to launch.
Kubernetes changed kube-apiserver feature-gate RemoveSelfLink
default value to true from k8s 1.20 onwards.
This caused PVCs to not bound to PVs and eventually caused some
system apps to fail. We need to configure the kube-apiserver
feature-gates with RemoveSelfLink=false to avoid it until we can
update the application code to handle the new behaviour.
Test Plan:
PASS: successful testing of the script during k8s version upgrades
from 1.18 -> 1.19 -> 1.20 -> 1.21 on AIO-DX system.
Verified that PVCs are bound to PVs after each upgrade.
Verified that pods are up and running after each upgrade.
Verified that kube-apiserver manifest and kubelet config
file is configured as expected after each upgrade.
PASS: 'build-pkgs --dep-test kubernetes-unversioned'
package built with dependencies passed
Story: 2008972
Task: 44037
Signed-off-by: kdhokte <kaustubh.dhokte@windriver.com>
Change-Id: Ie53ebb5839eb0bd843f5bb2c577a4292969c0997
Dependencies:
- some upstream debian packages:
cherry-pick https://review.opendev.org/c/starlingx/tools/+/817042
debdownloader $MY_REPO_ROOT_DIR/stx-tools/debian-mirror-tools/config/debian/common/base-bullseye.lst
- golang 1.16.6
cherry-pick https://review.opendev.org/c/starlingx/compile/+/809321
add it to 'debian_pkg_dirs' in the compile folder
build-pkgs -p golang 1.16.6
Build-image dependencies:
- etcd
- runc
- containerd
Skipped the generation of the man pages, focused
only on building the package.
Did build-pkgs
Did build-iso
Story: 2009101
Task: 43945
Change-Id: I5024e65f8c84d7b5d57e2a47b060b6ef4416053a
Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>
The src_path points to the local source directory.
Story: 2009101
Task: 43697
Signed-off-by: Yue Tao <yue.tao@windriver.com>
Change-Id: I869a5e0d9aefb88cae78856bbd076778b0b7f37f
Set SOURCE_REF to the latest release.
[Submitted on behalf of Vefa Bicakci.]
Closes-Bug: #1950513
Change-Id: Id9a0ac2e2c62fa4c829e280cd740fab1f40a2131
Signed-off-by: Joe Slater <joe.slater@windriver.com>
Create debian package structure
for armada and armada-helm-toolkit.
Story: 2009221
Task: 43917
Change-Id: Ie1177d50b1c22239be18c109129308d13d75ba28
Signed-off-by: Daniel Safta <daniel.safta@windriver.com>
Reduce Kubelet log volume by changing frequent log messages
from INFO to DEBUG by backporting Kubernetes commit
8f08db9164b9038b2a62ad3d0290d725bc860744.
Test Plan: Verify that the Kubelet log volume is reduced during
execution
Pass: Verified that the Kubelet log volume is greatly reduced by
examining the daemon.log file. Previously informational messages
relating to reconcileState no longer appear at the default log level.
Story: 2009272
Task: 43858
Change-Id: Ibfb2967f5b643269203fc91d884964f93e091d34
- Built the package
- Built the iso
- Verified if the package was installed correctly
Story: 2009101
Task: 43697
Change-Id: Id10c87f9a24329b533864411075e096f56dbcd8b
Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>