K8s Operator之controller Tools学习笔记

2024-09-12 约 2542 字预计阅读 6 分钟

controller-tools是一个由 Kubernetes 社区维护的项目，用于简化 Kubernetes 控制器的开发。其中提供了一组工具来生成和更新 Kubernetes API 对象的代码，以及构建自定义控制器所需的代码框架。
仓库地址：GitHub - kubernetes-sigs/controller-tools: Tools to use with the controller-runtime libraries

controller-tools包含工具与安装

包含工具

查看controller-tools源码的cmd目录可以发现，有以下三个cli工具：

controller-gen：用于生成 zz_xxx.deepcopy.go 文件以及 crd 文件【kubebuilder也是通过这个工具生成crd的相关框架的】
type-scaffold：用于生成所需的 types.go 文件
helpgen：用于生成针对 Kubernetes API 对象的代码文档，可以包括 API 对象的字段、标签和注释等信息

安装

从仓库release中下载

Releases · kubernetes-sigs/controller-tools 但看着只有controller-gen这个工具，没有看到另外两个。

从源码编译安装

1
2
3
4
5
6
git clone https://github.com/kubernetes-sigs/controller-tools.git
cd controller-tools
go mod tidy

# 直接安装到GOPATH bin目录下，需要提前把GOPATH bin添加进系统PATH，添加步骤在此不再赘述。
go install ./cmd/{controller-gen,type-scaffold,helpgen}

查看是否安装成功：

1
2
3
controller-gen -h
type-scaffold -h
helpgen -h

快速开始及示例

初始化示例项目：

1
2
3
4
mkdir controller-tools-study 
cd controller-tools-study 
go mod init controller-tools-study
mkdir -pv pkg/apis/appcontroller/v1alpha1

使用type-scaffold工具生成types.go：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
type-scaffold --kind=Application > pkg/apis/appcontroller/v1alpha1/types.go

cat pkg/apis/appcontroller/v1alpha1/types.go
# 输出内容如下：
// ApplicationSpec defines the desired state of Application
type ApplicationSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
}

// ApplicationStatus defines the observed state of Application.
// It should always be reconstructable from the state of the cluster and/or outside world.
type ApplicationStatus struct {
        // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Application is the Schema for the applications API
// +k8s:openapi-gen=true
type Application struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`

        Spec   ApplicationSpec   `json:"spec,omitempty"`
        Status ApplicationStatus `json:"status,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// ApplicationList contains a list of Application
type ApplicationList struct {
        metav1.TypeMeta `json:",inline"`
        metav1.ListMeta `json:"metadata,omitempty"`
        Items           []Application `json:"items"`
}

可以发现生成之后的代码，是没有包名和引用依赖包的，所以需要手动在文件头添加一下，编辑pkg/apis/appcontroller/v1alpha1/types.go文件：

1
2
3
4
5
package v1alpha1 

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

... 后面省略。。。

导入完成之后运行go mod tidy获取一下依赖。 3. 使用controller-gen生成deepcopy和crd文件在此之前，首先在pkg/apis/appcontroller/v1alpha1 下创建一个 doc.go 文件，并使用注释标记groupName，不然生成crd的时候文件名会变成下划线，空的.

1
2
3
4
vim pkg/apis/appcontroller/v1alpha1/doc.go

// +groupName=appcontroller.k8s.io
package v1alpha1

使用controller-gen生成deepcopy相关代码：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
controller-gen object paths=pkg/apis/appcontroller/v1alpha1/types.go

# 执行完成之后，发现多了个deepcopy文件。
❯ tree
.
├── go.mod
├── go.sum
└── pkg
    └── apis
        └── appcontroller
            └── v1alpha1
                ├── doc.go
                ├── types.go
                └── zz_generated.deepcopy.go

5 directories, 5 files

使用controller-gen生成crd：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
controller-gen crd paths=./... output:crd:dir=config/crd

# 再次查看目录结构
❯ tree
.
├── config
│   └── crd
│       └── appcontroller.k8s.io_applications.yaml
├── go.mod
├── go.sum
└── pkg
    └── apis
        └── appcontroller
            └── v1alpha1
                ├── doc.go
                ├── types.go
                └── zz_generated.deepcopy.go

7 directories, 6 files

查看生成之后的crd资源清单文件：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
cat config/crd/appcontroller.k8s.io_applications.yaml

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  name: applications.appcontroller.k8s.io
spec:
  group: appcontroller.k8s.io
  names:
    kind: Application
    listKind: ApplicationList
    plural: applications
    singular: application
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Application is the Schema for the applications API
        properties:
          apiVersion:
            description: |-
              APIVersion defines the versioned schema of this representation of an object.
              Servers should convert recognized schemas to the latest internal value, and
              may reject unrecognized values.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
            type: string
          kind:
            description: |-
              Kind is a string value representing the REST resource this object represents.
              Servers may infer this from the endpoint the client submits requests to.
              Cannot be updated.
              In CamelCase.
              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
            type: string
          metadata:
            type: object
          spec:
            description: ApplicationSpec defines the desired state of Application
            type: object
          status:
            description: |-
              ApplicationStatus defines the observed state of Application.
              It should always be reconstructable from the state of the cluster and/or outside world.
            type: object
        type: object
    served: true
    storage: true

可以看到properties.spec下面除了type没有任何字段，所以接下来我们为其添加两个字段，然后重新生成一下crd：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
vim pkg/apis/appcontroller/v1alpha1/types.go

...
type ApplicationSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
    // 添加两个测试字段，分别是名字和副本数
	Name string `json:"name"`
    Replicas int32 `json:"replicas"`
}
...

使用controller-gen重新生成crd：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
controller-gen crd paths=./... output:crd:dir=config/crd

# 再次查看crd文件内容，可以发现spec里面已经有上面代码中指定的字段了
...
          spec:
            description: ApplicationSpec defines the desired state of Application
            properties:
              name:
                description: INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
                type: string
              replicas:
                format: int32
                type: integer
            required:
            - name
            - replicas
            type: object
...

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

const GroupName = "appcontroller.k8s.io"

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1alpha1"}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	// SchemeBuilder initializes a scheme builder
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme is a global function that registers this API group & version to a scheme
	AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Application{},
		&ApplicationList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

应用crd文件

1
2
3
4
kubectl apply -f config/crd/appcontroller.k8s.io_applications.yaml

# 报错
The CustomResourceDefinition "applications.appcontroller.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

可以发现直接报错了，crd也没有成功创建，根据报错信息，protect kubernetes community owned API groups in CRDs by deads2k · Pull Request #1111 · kubernetes/enhancements · GitHub，简单来说就是crd的一个保护机制，不允许外部随意创建crd，要解决这个问题，添加一个annotations即可：编辑config/crd/appcontroller.k8s.io_applications.yaml添加annotations：

1
2
3
4
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
    # 添加以下内容
    api-approved.kubernetes.io: https://github.com/kubernetes/kubernetes/pull/78458

重新应用crd文件：

1
2
kubectl apply -f config/crd/appcontroller.k8s.io_applications.yaml
customresourcedefinition.apiextensions.k8s.io/applications.appcontroller.k8s.io created

成功创建。 5. 编写crd测试的资源：创建config/examples文件夹，用于测试，并在改目录添加以下测试文件test.yaml：

1
2
3
4
5
6
7
apiVersion: appcontroller.k8s.io/v1alpha1
kind: Application
metadata:
  name: test-crd
spec:
  name: test-crd
  replicas: 3

应用资源清单文件：

1
2
$ kubectl apply -f config/examples/test.yaml
application.appcontroller.k8s.io/test-crd created

查看创建结果：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
$ kubectl get applications.appcontroller.k8s.io
NAME       AGE
test-crd   43s

$ kubectl describe applications.appcontroller.k8s.io test-crd
Name:         test-crd
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  appcontroller.k8s.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2024-09-13T08:37:49Z
  Generation:          1
  Resource Version:    34589
  UID:                 f5d18812-b6bd-476f-b794-85f33a9e6fb5
Spec:
  Name:      test-crd
  Replicas:  3
Events:      <none>

测试cr的获取在项目根目录，创建一个cmd目录，并在该目录中创建程序入口文件main.go，内容如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
package main

import (
	"context"
	"controller-tools-study/pkg/apis/appcontroller/v1alpha1"
	"fmt"
	"log"
	"strings"

	"k8s.io/client-go/kubernetes/scheme"

	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
)

func main() {
	config, err := clientcmd.BuildConfigFromFlags("", clientcmd.RecommendedHomeFile)
	if err != nil {
		log.Fatalln(err)
	}

	config.APIPath = "/apis/"
	config.GroupVersion = &v1alpha1.SchemeGroupVersion
	config.NegotiatedSerializer = scheme.Codecs

	client, err := rest.RESTClientFor(config)
	if err != nil {
		log.Fatalln(err)
	}

	app := v1alpha1.Application{}
	err = client.Get().Namespace("default").Resource("applications").Name("test-crd").Do(context.TODO()).Into(&app)
	if err != nil {
		log.Fatalln(err)
	}

	// 测试深拷贝
	newObj := app.DeepCopy()
	newObj.Spec.Name = "test-crd"

	fmt.Println("ori：", app.Spec)
	fmt.Println(strings.Repeat("=", 50))
	fmt.Println("deepcopy：", newObj.Spec)
}

运行：

1
2
3
4
5
6
go run cmd/main.go
# 输出
ori： {test-crd 3}
==================================================
deepcopy： {test-crd 3}

目录