From b9e6848c5366046ee7cbd3e2ae683ecee30cc721 Mon Sep 17 00:00:00 2001 From: Ryan Hallisey <rhallise@redhat.com> Date: Mon, 11 Apr 2016 11:12:39 -0400 Subject: [PATCH] Spec: Deploy Kolla images using Kubernetes related blueprint kolla-kubernetes Change-Id: Ib1023787fca096d22a21c583c1ef647ccf925f00 --- specs/kubernetes-deployment.rst | 352 ++++++++++++++++++++++++++++++++ 1 file changed, 352 insertions(+) create mode 100644 specs/kubernetes-deployment.rst diff --git a/specs/kubernetes-deployment.rst b/specs/kubernetes-deployment.rst new file mode 100644 index 0000000000..edf5454c61 --- /dev/null +++ b/specs/kubernetes-deployment.rst @@ -0,0 +1,352 @@ +=================================== +Deploy Kolla images with Kubernetes +=================================== + +https://blueprints.launchpad.net/kolla/+spec/kolla-kubernetes + +Kubernetes was evaluated by the Kolla team in the first two months of the +project and it was found to be problematic because it did not support net=host, +pid=host, and --privileged features in docker. Since then, it has developed +these features [1]. + +The objective is to manage the lifecycle of containerized OpenStack services by +using Kubernetes container managment tools in order to obtain the self-healing +and upgrade capabilities inherent in Kubernetes. + +Problem description +=================== + +Kubernetes +- life-cycle management: service monitoring, HA, loadbalancing, and + health checks +- upgrades: rolling + +Kubernetes has services that provide kolla-kubernetes with service monitoring, +health checks, service scaling, and upgrades. The community can use the +scheduler and node affinity trait to assign work loads to appropriate nodes [2]. +Kubernetes also has built in health checks that monitors a container's status. +Finally, kolla-kubernetes can use the replication controller to scale up and +down the stack [3]. + +For upgrades, Kubernetes has an object called 'deployments'[4], which detects +when a pod needs to change. It starts to scale down the current running +pods and scale up the new pods. + +Use Cases +========= + +- Kubernetes as an underlay for OpenStack. +- Kubernetes to handle container scheduling. +- Feedback loop when using Kubernetes health checks during deployment and + upgrade. +- High Availability for individual containers + +Proposed change +=============== + +- Add a deployment specific git repo (kolla-kubernetes) under the kolla + governance that contains the Kubernetes deployment code. + +Orchestration +------------- + +OpenStack on Kubernetes will be orchestrated by outside tools in order to create +a production ready OpenStack environment. The kolla-kubernetes repo is where +any deployment tool can join the community and be a part of orchestrating a +kolla-kubernetes deployment. + +Service Config Management +------------------------- + +Config generation will be completely decoupled from the deployment. The +containers only expect a config file to land in a specific directory in +the container in order to run. With this decoupled model, any tool could be +used to generate config files. The kolla-kubernetes community will evaluate +any config generation tool, but will likely use Ansible for config generation +in order to reuse existing work from the community. This solution uses +customized Ansible and jinja2 templates to generate the config. Also, there will +be a maintained set of defaults and a global yaml file that can override the +defaults. + +The config files will be injected into the kubernetes configmap and loaded into +the containers. There will be one configmap per configuration file and there can +be multiple config maps. The containers will configure themselves using the +configuration files loaded into the appropriate directories [5][6]. + +Bootstrapping +------------- + +Bootstrapping the Kolla containers involves running a single task per service +that will initialize the databases and create the users. The bootstrapping task +will be a Kubernetes Job, which will run the task until completion then +terminate the pods [7]. + +Each service will have a bootstrap task so that when the operator upgrades, +the bootstrap tasks are reused to upgrade the database. This will allow +deployment and upgrades to follow the same pipeline. + +The Kolla containers will communicate with the Kubernetes API server to in order +to be self aware of if any bootstrapping processes are occuring. + +Alternative bootstrap approaches: + +1) Create 2 pods per OpenStack service. One pod is designed to do the +bootstrapping/db_sync while the other pod runs as the normal service. This will +require some orchestration and the bootstrap pod will need to be setup to +never restart or be replicated. + +2) Use a sidecar container in the pod to handle the database sync with proper +health checking to make sure the services are coming up healthy. The big +difference between kolla's old docker-compose solution and Kubernetes, is that +docker-compose would only restart the containers. Kubernetes will completely +reschedule them. Which means, removing the pod and restarting it. The reason +this would fix that race condition failure kolla saw from docker-compose is +because glance would be rescheduled on failure allowing keystone to get a +chance to sync with the database and become active instead of constantly being +piled with glance requests. There can also be health checks around this to help +determine order. + +If kolla-kubernetes used this sidecar approach, it would regain the use of +native Kubernetes upgrades [16]. + +Dependencies +------------ + +- Kubernetes >= 1.3.0 +- Docker >= 1.10.0 +- Jinja2 >= 2.8.0 + +Kubernetes does not support dependencies between pods. The operator will launch +all the services and use kubernetes health checks to bring the deployment to an +operational state. + +With orchestration around Kubernetes, the operator can determine what tasks are +run and when the tasks are run. This way, dependencies are handled at the +orchestration level, but they are not required because proper health checking +will bring up the cluster in a healthy state. + +Upgrades +-------- + +Kubernetes has an object called a Deployment, where the operator defines a +desired state for the pods and the deployment will move the cluster to the +desired state when a change is detected. + +Kolla-kubernetes will provide Jobs that will provide the operator with the +flexibility needed to under go a step wise upgrade. In future releases, +kolla-kubernetes will look to Kubernetes to provide a means for operators to +plugin these jobs into a Deployment. + +Reconfigure +----------- + +The operator generates a new config and loads it into the Kubernetes configmap +by changing the configmap version in the service yaml file. Then, the operator +will trigger a rolling upgrade, which will scale down old pods and bring up new +ones that will run with the updated configuration files. + +There's an open issue upstream in Kubernetes where the plan is to add support +around detecting if a pod has a changed in the configmap [6]. Depending on what +the solution is, kolla-kubernetes may or may not use it. The rolling +upgrade feature will provide kolla-kubernetes with an elegant way to handle +restarting the services. + +HA Architecture +--------------- + +Kubernetes uses health checks to bring up the services. Therefore, +kolla-kubernetes will use the same checks when monitoring if a service is +healthy. When a service fails, the replication controller will be responsible +for bringing up a new container in its place [8][9]. + +However, Kubernetes does not cover all the HA corner cases, for instance, +fencing. But, there are some operator known practices that can be used to get +around this [10]. For example, to implement storage fencing, the operator can +use ceph backed storage [11][12]. This is an option that the community can +document in order to provide kolla-kubernetes with a production ready solution +if Kubernetes cannot. + +.. note:: There is a known issue in Kubernetes with releasing volumes from a +node that disappeared from the cluster. This is expected to be fixed in the 1.3 +release [13]. + +Persistent Storage +------------------ + +Kubernetes has many types of persistent storage [14]. Since Kubernetes doesn't +guarantee a pod will always be scheduled to a host, it makes node based +persistent storage unlikely, unless the community uses labels for every pod. + +Persistent storage in kolla-kubernetes will come from volumes backed by +different storage offerings to provide persistent storage. Kolla-kubernetes +will provide a default solution using Ceph RBD, that the community will use to +deploy multinode with. From there, kolla-kubernetes can add any additional +persistent storage options as well as support options for the operator to +reference an existing storage solution. + +To deploy Ceph, the community will use the Ansible playbooks from Kolla to +deploy a containerized Ceph at least for the 1.0 release. After Kubernetes +deployment matures, the community can evaluate building its own Ceph deployment +solution. + +Existing external Ceph deployments will require additional documentation +to describe how to integrate them with a Kubernetes deployment. + +Service Roles +------------- + +At the broadest level, OpenStack can split up into two main roles, Controller +and Compute. With Kubernetes, the role definition layer changes. +Kolla-kubernetes will still need to define Compute nodes, but not Controller +nodes. Compute nodes hold the libvirt container and the running vms. That +service cannont migrate because the vms associated with it exist on the node. +However, the Controller role is more flexible. The Kubernetes layer provides IP +persistence so that APIs will remain active and abstracted from the operator's +view [15]. kolla-kubernetes can direct Controller services away from the Compute +node using labels, while managing Compute services more strictly. + +The Kubernetes Label field will be configurable to allow the operator to +define roles and direct where services will land. + +Security impact +--------------- + +Kolla-Kubernetes will run the containers as non root wherever possible. +SELinux or AppArmor will be in place to limit the damage from container +breakouts. + +Kubernetes is planning to adding capabilities to the pod level that will enable +the community to restrict container privileges even further [16]. + +Performance Impact +------------------ + +Since kolla-kubernetes is not using dependencies for the service deployment, the +services will take a different amount of time to start up for each deployment +because the order will always vary when the services become active. +As such, it's hard to quantify the exact performance impact other than it is +small. + +Networking +---------- + +Kolla-kubernetes will initially use 'net=host' everywhere to get the project +going. As the project matures, 'net=host' needs to be reevaluated as to which +services will run without 'net=host' in order to gain additional functionality. +For example, controller services will float between nodes potentially landing +two of the same pods on the same node. Those pods will be listening on the same +ports in the hosts network stack, which could prevent the pods from working. + +Logging & Monitoring +-------------------- + +To reuse Kolla's containers, kolla-kubernetes will use elastic search, heka, and +kibana as the default logging mechanism. + +The community will implement centralized logging by using a 'side car' container +in the Kubernetes pod [17]. The logging service will trace the logs from the +shared volume of the running serivce and send the data to elastic search. This +solution is ideal because volumes are shared amoung the containers in a pod. + +Implementation +============== + +Primary Assignee(s) +----------- + Ryan Hallisey (rhallisey) + +Other contributor(s): + kolla-core team [18] + Alex Polvi (polvi) + Andrew Battye + Brandon Jozsa (v1k0d3n) + Britt Houser (britthouser) + Davanum Srinivas (dims) + David Wang (dcwangmit01) + Egor Guz (eghobo) + Greg Herlein (gherlein) + Hui Kang (huikang) + Ian Main (Slower) + Jinay Vora (jvora) + Keith Byrne (kbyrne) + Ken Wronkiewicz (wirehead) + Kevin Fox (kfox1111) + Marga Millet (fragatina) + Marian Schwarz + Mark Casey (mark-casey) + Mauricio Lima (mlima) + Md Nadeem (mail2nadeem92) + Michael Schmidt + Michal Rostecki (mrostecki) + Qiu Yu (unicell) + Rajath Agasthya (rajathagasthya) + Rob Mason + Sean Mooney (sean-k-mooney) + Serguei Bezverkhi (sbezverk) + Sidharth Surana (ssurana) + Zdenek Janda (xdeu) + <Please add your name here if you are getting involved in kolla-kubernetes> + +Milestones +---------- + +Target Milestone for tech-preview code: + Newton + +Work Items +---------- +1. Create kolla-kubernetes repo +2. Build yaml files for each service +3. Build a CLI to handle templated yaml files +4. Build an all in one environment +5. Drop net=host on a set of services +6. Write per service health checks +7. Write startup docs +8. Add orchestration tools around the pods +9. All in one gating +10. Convert each service to a 'Deployment' +11. Build multinode environment +12. Config generation tools +13. Multinode docs +14. Implement reconfigure by templating configmaps +15. Centralized logging +16. Implement upgrades +17. Advanced deployment docs +<Please add new work items that are worth mentioning in the spec> + +Testing +======= + +Functional tests will be implemented in the OpenStack check/gating system to +automatically test that the Kubernetes deployment works for an AIO +environment [19]. + +Documentation Impact +==================== +Add a quick start guide, which explains how to deploy kolla-kubernetes. +Add a developer guide on how to contribute which also explains how the +deployment works. + +References +========== + +- [1] https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0 +- [2] http://kubernetes.io/docs/user-guide/node-selection/ +- [3] http://kubernetes.io/v1.0/docs/user-guide/managing-deployments.html +- [4] https://cloud.google.com/container-engine/docs/replicationcontrollers/ +- [5] https://github.com/kubernetes/kubernetes/blob/master/docs/design/configmap.md +- [6] https://github.com/kubernetes/kubernetes/issues/24957 +- [7] http://kubernetes.io/docs/user-guide/jobs/ +- [8] http://kubernetes.io/docs/user-guide/replication-controller/ +- [9] http://kubernetes.io/docs/user-guide/replicasets/ +- [10] http://kubernetes.io/docs/admin/high-availability/#master-elected-components +- [11] http://kubernetes.io/docs/user-guide/volumes/#rbd +- [12] http://docs.ceph.com/docs/master/cephfs/eviction/ +- [13] https://github.com/kubernetes/kubernetes/issues/20262 +- [14] http://kubernetes.io/docs/user-guide/volumes/ +- [15] http://kubernetes.io/docs/user-guide/node-selection/ +- [16] https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-security-context.md +- [17] http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html +- [18] https://review.openstack.org/#/admin/groups/460,members +- [19] https://etherpad.openstack.org/p/kolla-newton-summit-kolla-gate-walkthru +- https://github.com/kubernetes/kubernetes