案例一 Traefik Operator开发
1.kubebuilder 创建项目
2.Crontroller开发与部署
开发环境准备
kubebuilder 介绍
CRD的开发与部署
Crontroller开发与部署
Operator功能设计
借助operator完成 和企业内部注册中心打通
这里以Traefik+etcd的模式为例进行演示说明
在这里etcd provider注册中心可以换成eruka
流程
1. watch Pods
2.动态注册Pods的Endponit到Traefik
Operator 开发工具选择
Operator 开发SDK有2个选择:
kubebuilder
operator sdk
本质上他们都是在k8s控制器运行时上进行的封装,主要都是脚手架的生成,使用体验相差不大
kubebuilder
维护方: kubernetes-sigs
基于 k8s控制器运行时 封装
包含CRD和Controller开发
operator sdk
维护方: coreOS
基于 k8s控制器运行时封装
早期不包含CRD开发,在往kubebuilder方向进行融合
环境
go v1.18.5
kubernetes v1.24.2
kubebuilder v3.6.0
安装kubebuilder
https://objects.githubusercontent.com/github-production-release-asset-2e65be/125275487/35d09c97-2992
https://github.com/kubernetes-sigs/kubebuilder/releases/download/v3.6.0/kubebuilder_linux_amd64
[root@k8s-worker02 ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.6.0", KubernetesVendor:"1.24.1", GitCommit:"f20414648f1851ae97997f4a5f8eb4329f450f6d", BuildDate:"2022-08-03T11:47:17Z", GoOs:"linux", GoArch:"amd64"}
可以自己编译出windows的包
kubebuilder介绍
[root@k8s-worker02 ~]# kubebuilder -h
CLI tool for building Kubernetes extensions and tools.Usage:kubebuilder [flags]kubebuilder [command]Examples:
The first step is to initialize your project:kubebuilder init [--plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]]<PLUGIN KEYS> is a comma-separated list of plugin keys from the following table
and <PROJECT VERSION> a supported project version for these plugins.Plugin keys | Supported project versions
------------------------------------------+----------------------------base.go.kubebuilder.io/v3 | 3declarative.go.kubebuilder.io/v1 | 2, 3deploy-image.go.kubebuilder.io/v1-alpha | 3go.kubebuilder.io/v2 | 2, 3go.kubebuilder.io/v3 | 3go.kubebuilder.io/v4-alpha | 3grafana.kubebuilder.io/v1-alpha | 3kustomize.common.kubebuilder.io/v1 | 3kustomize.common.kubebuilder.io/v2-alpha | 3For more specific help for the init command of a certain plugins and project version
configuration please run:kubebuilder init --help --plugins=<PLUGIN KEYS> [--project-version=<PROJECT VERSION>]Default plugin keys: "go.kubebuilder.io/v3"
Default project version: "3"Available Commands:alpha Alpha-stage subcommandscompletion Load completions for the specified shellcreate Scaffold a Kubernetes API or webhookedit Update the project configurationhelp Help about any commandinit Initialize a new projectversion Print the kubebuilder versionFlags:-h, --help help for kubebuilder--plugins strings plugin keys to be used for this subcommand execution--project-version string project version (default "3")Use "kubebuilder [command] --help" for more information about a command.
创建一个Operator项目
kubebuilder提供了一个init命令用来初始化一个新的Operator工程目录,具体用法如下:
[root@k8s-worker02 ~]# kubebuilder init -h
Initialize a new project including the following files:- a "go.mod" with project dependencies- a "PROJECT" file that stores project configuration- a "Makefile" with several useful make targets for the project- several YAML files for project deployment under the "config" directory- a "main.go" file that creates the manager that will run the project controllersUsage:kubebuilder init [flags]Examples:# Initialize a new project with your domain and name in copyrightkubebuilder init --plugins go/v3 --domain example.org --owner "Your name"# Initialize a new project defining a specific project versionkubebuilder init --plugins go/v3 --project-version 3Flags:--component-config create a versioned ComponentConfig file, may be 'true' or 'false'--domain string domain for groups (default "my.domain")--fetch-deps ensure dependencies are downloaded (default true)-h, --help help for init--license string license to use to boilerplate, may be one of 'apache2', 'none' (default "apache2")--owner string owner to add to the copyright--project-name string name of this project--project-version string project version (default "3")--repo string name to use for go module (e.g., github.com/user/repo), defaults to the go package of the current working directory.--skip-go-version-check if specified, skip checking the Go versionGlobal Flags:--plugins strings plugin keys to be used for this subcommand execution
关键参数说明:
--plugins,指定生成代码的插件, 默认使用“go.kubebuilder.io/v3”。
--project-version 支持的项目版本有2,3 默认3
--repo module path,就是 go mod init 指定的go module的名称·
--domain,组织名称 用于API Group等
--owner,operater所有者,一般填写开发者邮箱
创建API
通过脚手架为我们提供的create api来创建 CRD 相关的Resource 和控制器:
[root@k8s-worker02 ~]# kubebuilder create api -h
Scaffold a Kubernetes API by writing a Resource definition and/or a Controller.If information about whether the resource and controller should be scaffolded
was not explicitly provided, it will prompt the user if they should be.After the scaffold is written, the dependencies will be updated and
make generate will be run.Usage:kubebuilder create api [flags]Examples:# Create a frigates API with Group: ship, Version: v1beta1 and Kind: Frigatekubebuilder create api --group ship --version v1beta1 --kind Frigate# Edit the API Schemenano api/v1beta1/frigate_types.go# Edit the Controllernano controllers/frigate/frigate_controller.go# Edit the Controller Testnano controllers/frigate/frigate_controller_test.go# Generate the manifestsmake manifests# Install CRDs into the Kubernetes cluster using kubectl applymake install# Regenerate code and run against the Kubernetes cluster configured by ~/.kube/configmake runFlags:--controller if set, generate the controller without prompting the user (default true)--force attempt to create resource even if it already exists--group string resource Group-h, --help help for api--kind string resource Kind--make make generate if true, run make generate after generating files (default true)--namespaced resource is namespaced (default true)--plural string resource irregular plural form--resource if set, generate the resource without prompting the user (default true)--version string resource VersionGlobal Flags:--plugins strings plugin keys to be used for this subcommand execution
[root@k8s-worker02 ~]#
项目生成文件解读
kubebuilder create api --group traefik --version v1 --kind TraefikService
- api/v1 目录下主要存放是我们API Object, 就是我们的Resource对象相关信息
- config/crd 目录下是我们crd的描述文件,我们需要把自定义资源(CRD)的描述信息注册给k8s时需要的
- rbac 目录下 存放着 关于CRD资源的 role定义的样例文件(editor/viewer)
- samples 目录下 存放着CRD的一个样例文件,后面部署完成后可以 直接编辑下 apply到k8s集群中去
- controllers 目录下 存放着 我们所有的 Object 的Controller 代码
cd /home/gopath/src/
mkdir k8s-operator
cd k8s-operator
kubebuilder init --plugins go/v3 --domain wu123.com --owner "wu123"
kubebuilder create api --group traefik --version v1 --kind TraefikService
make install
CRD开发
CRD设计
我们需要定义Traefik Service, 我们来看看Traefik Service一个service 实例定义:
<etcd_prefix>/<entry_point>/services/loadBalancer/servers/<index>/url <url_value>
traefik etcd配置的前缀,provider配置时 有设置
services: 表示 web entrypoint的 services配置
loadBalancer: cmdb 服务loadBalancer配置
servers: loadBalancer 下的实例配置
0(变量): index
因此我们定义的Service需要有如下属性
entrypoint name
service name
service url
由于 Name已经在 ObjectMeta 有声明了,因此我们只需要添加entrypoint 和 url
修改资源定义: api/v1/traefikservice_types.go
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required, Any new fields you add must have json tags for the fields to be serialized.
// TraefikServiceSpec defines the desired state of TraefikService
type TraefikServiceSpec struct {// INSERT ADDITIONAL SPEC FIELDS desired state of cluster// Important: Run "make" to regenerate code after modifying this file// Foo is an example field of TraefikService, Edit traefikservice types.go to remove/updateEntrypoint string `json:"entrypoint"` URL string `json:"url"`
}// TraefikServiceStatus defines the observed state of TraefikService
type TraefikServiceStatus struct {// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster// Important: Run "make" to regenerate code after modifying this fileActive bool `json:"active"`LastUpdateTime *metav1.Time `json:"lastupdatetime"`
}
生成CRD相关描述与代码
CRD代码生成
我们通过make 提供
。manifests 重新生成修改后的 CRD定义描述
。generate 重新生成代码
安装CRD
安装CRD
脚手架使用
。 install: 安装CRD
。 uninstall:卸载CRD
Deployment
install Install CRDs into the K8s cluster specified in */.kube/config.
uninstall Uninstall CRDs from the K8s cluster specified in */.kube/config.
当然你也可以使用SDK, List Resource 查看当前集群的所有支持的Resource
$ kubectl get crd
[root@k8s-worker02 k8s-operator]# kubectl get crd
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2023-02-19T12:58:01Z
bgppeers.crd.projectcalico.org 2023-02-19T12:58:01Z
blockaffinities.crd.projectcalico.org 2023-02-19T12:58:01Z
caliconodestatuses.crd.projectcalico.org 2023-02-19T12:58:01Z
clusterinformations.crd.projectcalico.org 2023-02-19T12:58:01Z
felixconfigurations.crd.projectcalico.org 2023-02-19T12:58:01Z
globalnetworkpolicies.crd.projectcalico.org 2023-02-19T12:58:01Z
globalnetworksets.crd.projectcalico.org 2023-02-19T12:58:01Z
hostendpoints.crd.projectcalico.org 2023-02-19T12:58:01Z
ipamblocks.crd.projectcalico.org 2023-02-19T12:58:01Z
ipamconfigs.crd.projectcalico.org 2023-02-19T12:58:01Z
ipamhandles.crd.projectcalico.org 2023-02-19T12:58:02Z
ippools.crd.projectcalico.org 2023-02-19T12:58:02Z
ipreservations.crd.projectcalico.org 2023-02-19T12:58:02Z
kubecontrollersconfigurations.crd.projectcalico.org 2023-02-19T12:58:02Z
networkpolicies.crd.projectcalico.org 2023-02-19T12:58:02Z
networksets.crd.projectcalico.org 2023-02-19T12:58:02Z
students.stable.example.com 2023-03-08T03:23:57Z
traefikservices.traefik.wu123.com 2023-04-19T14:14:50Z
验证CRD
1.准备CRD的YAML定义
我们参考samples下的样例,补充spec相关参数
```yaml
apiVersion: traefik.wu123.com/v1
kind: TraefikService
metadata:name: traefikservice-sample
spec:# TODO(user): Add fields hereentrypoint: weburl: https://www.baidu.com
```
# 创建资源
[root@k8s-worker02 k8s-operator]# kubectl apply -f config/samples/traefik_v1_traefikservice.yaml
traefikservice.traefik.wu123.com/traefikservice-sample created
#查看资源
[root@k8s-worker02 k8s-operator]# kubectl get TraefikService
NAME AGE
traefikservice-sample 59s# 查看资源yaml
[root@k8s-worker02 k8s-operator]# kubectl describe TraefikService
[root@k8s-worker02 k8s-operator]# kubectl get TraefikService -o yaml
最后我们删除我们定义的资源
root@k8s-worker02 k8s-operator]# kubectl delete -f config/samples/traefik_v1_traefikservice.yaml
traefikservice.traefik.wu123.com "traefikservice-sample" deleted
Crontroller工作原理
因此Crontroller 是个面向期望的编程模型,我们声明的这个期望对象,就是API Object(K8s Runtime Object)
Crontroller业务逻辑编写
// 每当traefikv1.TraefikService对象有变化时,我们就会收到一个请求
func (r *TraefikServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {// 获取日志对象l := log.FromContext(ctx, "namespace", req.Namespace)// TODO(user): your logic here// 1.通过名称获取TraefikService对象,并打印var obj traefikv1.TraefikServiceif err := r.Get(ctx, req.NamespacedName, &obj); err != nil {// 如果Not Found则表示该资源已经删除,需要做删除处理if apierrors.IsNotFound(err) {l.Info("delete service ...")err = nil} else {l.Error(err, "unable to fetch TraefikService")}return ctrl.Result{}, err}l.Info("get TraefikService object","name", obj.Name,"ur1", obj.Spec.URL,"entrypoint", obj.Spec.Entrypoint)// 2.更新状态 排除已经同步完成的对象if obj.Status.Active {l.Info("traefik service is active, skip ...")return ctrl.Result{}, nil}// 3.注册服务到Traefik service中,比如写入到etcd provider中l.Info("set traefik service ...")// 4,修改成功调整对象的状态obj.Status.Active = trueobj.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}if err := r.Status().Update(ctx, &obj); err != nil {l.Info("unable to update status", "reason", err.Error())}return ctrl.Result{}, nil
}
controllers/traefikservice_controller.go
/*
Copyright 2023 wu123.Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/package controllersimport ("context""time"apierrors "k8s.io/apimachinery/pkg/api/errors"metav1 "k8s.io/apimachinery/pkg/apis/meta/v1""k8s.io/apimachinery/pkg/runtime"ctrl "sigs.k8s.io/controller-runtime""sigs.k8s.io/controller-runtime/pkg/client""sigs.k8s.io/controller-runtime/pkg/log"traefikv1 "k8s-operator/api/v1"
)// TraefikServiceReconciler reconciles a TraefikService object
type TraefikServiceReconciler struct {client.ClientScheme *runtime.Scheme
}//+kubebuilder:rbac:groups=traefik.wu123.com,resources=traefikservices,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=traefik.wu123.com,resources=traefikservices/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=traefik.wu123.com,resources=traefikservices/finalizers,verbs=update// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the TraefikService object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.2/pkg/reconcile
// 每当traefikv1.TraefikService对象有变化时,我们就会收到一个请求
func (r *TraefikServiceReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {// 获取日志对象l := log.FromContext(ctx, "namespace", req.Namespace)// TODO(user): your logic here// 1.通过名称获取TraefikService对象,并打印var obj traefikv1.TraefikServiceif err := r.Get(ctx, req.NamespacedName, &obj); err != nil {// 如果Not Found则表示该资源已经删除,需要做删除处理if apierrors.IsNotFound(err) {l.Info("delete service ...")err = nil} else {l.Error(err, "unable to fetch TraefikService")}return ctrl.Result{}, err}l.Info("get TraefikService object","name", obj.Name,"ur1", obj.Spec.URL,"entrypoint", obj.Spec.Entrypoint)// 2.更新状态 排除已经同步完成的对象if obj.Status.Active {l.Info("traefik service is active, skip ...")return ctrl.Result{}, nil}// 3.注册服务到Traefik service中,比如写入到etcd provider中l.Info("set traefik service ...")// 4,修改成功调整对象的状态obj.Status.Active = trueobj.Status.LastUpdateTime = &metav1.Time{Time: time.Now()}if err := r.Status().Update(ctx, &obj); err != nil {l.Info("unable to update status", "reason", err.Error())}return ctrl.Result{}, nil
}// SetupWithManager sets up the controller with the Manager.
func (r *TraefikServiceReconciler) SetupWithManager(mgr ctrl.Manager) error {return ctrl.NewControllerManagedBy(mgr).For(&traefikv1.TraefikService{}).Complete(r)
}
Crontroller 测试
部署 Controller为了确保我们的CRD描述是最新的. 我们重新安装下
[root@k8s-worker02 k8s-operator]# make manifests && make install
test -s /home/gopath/src/k8s-operator/bin/controller-gen || GOBIN=/home/gopath/src/k8s-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2
/home/gopath/src/k8s-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /home/gopath/src/k8s-operator/bin/controller-gen || GOBIN=/home/gopath/src/k8s-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2
/home/gopath/src/k8s-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/gopath/src/k8s-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/traefikservices.traefik.wu123.com configured
[root@k8s-worker02 k8s-operator]#
为了方便起见,我们将在本地运行 controller,当然您也可以将其部署到 Kubernetes 上运行
[root@k8s-worker02 k8s-operator]# make run
test -s /home/gopath/src/k8s-operator/bin/controller-gen || GOBIN=/home/gopath/src/k8s-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.9.2
/home/gopath/src/k8s-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/gopath/src/k8s-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
I0420 00:13:46.160952 32048 request.go:601] Waited for 1.048392806s due to client-side throttling, not priority and fairness, request: GET:https://192.168.204.129:6443/apis/storage.k8s.io/v1beta1?timeout=32s
1.681920826812908e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.6819208268136308e+09 INFO setup starting manager
1.6819208268141122e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6819208268141956e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6819208268142333e+09 INFO Starting EventSource {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "source": "kind source: *v1.TraefikService"}
1.6819208268142862e+09 INFO Starting Controller {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService"}
1.6819208269164274e+09 INFO Starting workers {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "worker count": 1}
验证 Controller
我们继续使用之前的样例, 看看Controller能否感觉对象的变化 正常工作
#创建资源
[root@k8s-worker02 k8s-operator]# kubectl apply -f config/samples/traefik_v1_traefikservice.yaml
traefikservice.traefik.wu123.com/traefikservice-sample created
我们通过controller的日志,来确认Controller是否正常工作
1.681921012500303e+09 INFO get TraefikService object {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "traefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "c9b7be99-3190-48d9-bafe-0d9194dc26af", "namespace": "default", "name": "traefikservice-sample", "ur1": "https://www.baiwu.com", "entrypoint": "web"}
1.68192101250033e+09 INFO set traefik service ... {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "traefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "c9b7be99-3190-48d9-bafe-0d9194dc26af", "namespace": "default"}
1.6819210125653844e+09 INFO get TraefikService object {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "traefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "ec735952-7de0-4bff-b9b4-2cfefd03a822", "namespace": "default", "name": "traefikservice-sample", "ur1": "https://www.baiwu.com", "entrypoint": "web"}
1.6819210125654137e+09 INFO traefik service is active, skip ... {"controller": "traefikservice", "controllerGroup": "traefik.wu123.com", "controllerKind": "TraefikService", "traefikService": {"name":"traefikservice-sample","namespace":"default"}, "namespace": "default", "name": "traefikservice-sample", "reconcileID": "ec735952-7de0-4bff-b9b4-2cfefd03a822", "namespace": "default"}
案例二 Pod Crontroller 实战
复制controllers/traefikservice_controller.go,修改为controllers/pod_controller.go
// SetupWithManager sets up the controller with the Manager.
func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error {return ctrl.NewControllerManagedBy(mgr).For(&v1.Pod{}).Complete(r)
}// 每当v1.Pod对象有变化时,我们就会收到一个请求
func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {// 获取日志对象l := log.FromContext(ctx, "namespace", req.Namespace)// TODO(user): your logic here// 1.通过名称获取Pod对象,并打印var obj v1.Podif err := r.Get(ctx, req.NamespacedName, &obj); err != nil {// 如果Not Found则表示该资源已经删除,需要做删除处理if apierrors.IsNotFound(err) {l.Info("delete Pods ...","namespace", req.Namespace,"name", req.Name)err = nil} else {l.Error(err, "unable to fetch Pod")}}l.Info(obj.Name,"namespace", obj.Namespace,"labels", obj.Labels,)l.Info(obj.Status.PodIP)for _, c := range obj.Spec.Containers {l.Info(c.Name,"pod_id", c.Ports)}return ctrl.Result{}, nil
}
pod_controller.go没法直接make run,还要在main.go添加
if err = (&controllers.TraefikServiceReconciler{Client: mgr.GetClient(),Scheme: mgr.GetScheme(),}).SetupWithManager(mgr); err != nil {setupLog.Error(err, "unable to create controller", "controller", "TraefikService")os.Exit(1)}//+kubebuilder:scaffold:builder// Endpoint Controllerif err = (&controllers.PodReconciler{Client: mgr.GetClient(),Scheme: mgr.GetScheme(),}).SetupWithManager(mgr); err != nil {setupLog.Error(err, "unable to create controller", "controller", "podReconciler")os.Exit(1)}if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {setupLog.Error(err, "unable to set up health check")os.Exit(1)}if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {setupLog.Error(err, "unable to set up ready check")os.Exit(1)}
测试,新建一个deployment nginx-test, service
查看controller的日志
删除deployment
代码仓库https://github.com/yunixiangfeng/k8s-exercise
k8s-exercise/k8s-operator at main · yunixiangfeng/k8s-exercise · GitHub