• author

第31节 kubernetes 的编译和调试

⬆️上一节🔗 ⬇️下一节🔗

❤️💕💕新时代拥抱云原生，云原生具有环境统一、按需付费、即开即用、稳定性强特点。Myblog:http://nsddd.top

---

[TOC]

编译启动本地单节点集群：

cd $GOPATH/src/k8s.io/kubernetes
# 编译单个组建：sudo make WHAT="cmd/kube-apiserver"  
# 编译所有组件：sudo make all  
# 启动本地单节点集群： sudo ./hack/local-up-cluster.sh  

local-up-cluster 脚本

./hack/local-up-cluster.sh 脚本是 kubernetes 社区构建的方便开发者的脚本。

它的作用是编译出所有需要编译的应用程序，并且在当前的机器上跑一个单节点的集群。

• 编译单个组建：sudo make WHAT="cmd/kube-apiserver"

• 编译所有组件：sudo make all

• 启动本地单节点集群： sudo ./hack/local-up-cluster.sh

第一次启动单节点集群分析

记录第一次启动但集群的状态分析：

time：大概等了 20 - 30 分钟（可以先喝两杯咖啡☕）

安装细节

开始时间：

root@cubmaster01:~/go/src/k8s.io/kubernetes# date
Thu 01 Dec 2022 06:45:33 AM UTC

details 展开🔽

root@cubmaster01:~/go/src/k8s.io/kubernetes# sudo ./hack/local-up-cluster.sh
make: Entering directory '/root/go/src/k8s.io/kubernetes'
make[1]: Entering directory '/root/go/src/k8s.io/kubernetes'
+++ [1201 06:45:46] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
+++ [1201 06:45:50] Generating openapi code for KUBE
+++ [1201 06:46:29] Generating openapi code for AGGREGATOR
+++ [1201 06:46:31] Generating openapi code for APIEXTENSIONS
+++ [1201 06:46:34] Generating openapi code for CODEGEN
+++ [1201 06:46:36] Generating openapi code for SAMPLEAPISERVER
make[1]: Leaving directory '/root/go/src/k8s.io/kubernetes'
+++ [1201 06:46:39] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)

make: Leaving directory '/root/go/src/k8s.io/kubernetes'
API SERVER secure port is free, proceeding...
Detected host and ready to start services.  Doing some housekeeping first...
Using GO_OUT /root/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64
Starting services now!
Starting etcd
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.ZA6EXyPPss --listen-client-urls http://127.0.0.1:2379 --log-level=warn 2> "/tmp/etcd.log" >/dev/null
Waiting for etcd to come up.
+++ [1201 06:58:54] On try 2, etcd: : {"health":"true","reason":""}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}Generating a RSA private key
...............................................+++++
.....+++++
writing new private key to '/var/run/kubernetes/server-ca.key'
-----
Generating a RSA private key
..............+++++
.......+++++
writing new private key to '/var/run/kubernetes/client-ca.key'
-----
Generating a RSA private key
.................+++++
......................................+++++
writing new private key to '/var/run/kubernetes/request-header-ca.key'
-----
2022/12/01 06:58:54 [INFO] generate received request
2022/12/01 06:58:54 [INFO] received CSR
2022/12/01 06:58:54 [INFO] generating key: rsa-2048
2022/12/01 06:58:54 [INFO] encoded CSR
2022/12/01 06:58:54 [INFO] signed certificate with serial number 189366430448716360665414597641985438063348873322
2022/12/01 06:58:54 [INFO] generate received request
2022/12/01 06:58:54 [INFO] received CSR
2022/12/01 06:58:54 [INFO] generating key: rsa-2048
2022/12/01 06:58:54 [INFO] encoded CSR
2022/12/01 06:58:54 [INFO] signed certificate with serial number 190125437621142035036292880957413679793726301630
2022/12/01 06:58:54 [INFO] generate received request
2022/12/01 06:58:54 [INFO] received CSR
2022/12/01 06:58:54 [INFO] generating key: rsa-2048
2022/12/01 06:58:54 [INFO] encoded CSR
2022/12/01 06:58:54 [INFO] signed certificate with serial number 606668605759100598551725540945577297218224811054
2022/12/01 06:58:54 [INFO] generate received request
2022/12/01 06:58:54 [INFO] received CSR
2022/12/01 06:58:54 [INFO] generating key: rsa-2048
2022/12/01 06:58:54 [INFO] encoded CSR
2022/12/01 06:58:54 [INFO] signed certificate with serial number 635150309675323248743719255558829775631435343029
2022/12/01 06:58:55 [INFO] generate received request
2022/12/01 06:58:55 [INFO] received CSR
2022/12/01 06:58:55 [INFO] generating key: rsa-2048
2022/12/01 06:58:55 [INFO] encoded CSR
2022/12/01 06:58:55 [INFO] signed certificate with serial number 502583423540832867136826540974772275871393658467
2022/12/01 06:58:55 [INFO] generate received request
2022/12/01 06:58:55 [INFO] received CSR
2022/12/01 06:58:55 [INFO] generating key: rsa-2048
2022/12/01 06:58:55 [INFO] encoded CSR
2022/12/01 06:58:55 [INFO] signed certificate with serial number 712098371673773073639809258966059598612266254784
2022/12/01 06:58:55 [INFO] generate received request
2022/12/01 06:58:55 [INFO] received CSR
2022/12/01 06:58:55 [INFO] generating key: rsa-2048
2022/12/01 06:58:55 [INFO] encoded CSR
2022/12/01 06:58:55 [INFO] signed certificate with serial number 254427577071095149538770795047596258056763227355
2022/12/01 06:58:55 [INFO] generate received request
2022/12/01 06:58:55 [INFO] received CSR
2022/12/01 06:58:55 [INFO] generating key: rsa-2048
2022/12/01 06:58:55 [INFO] encoded CSR
2022/12/01 06:58:55 [INFO] signed certificate with serial number 583900352045971716923710917507647330423197506650
Waiting for apiserver to come up
+++ [1201 06:59:03] On try 6, apiserver: : ok
clusterrolebinding.rbac.authorization.k8s.io/kube-apiserver-kubelet-admin created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-csr created
Cluster "local-up-cluster" set.
use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated API server
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
coredns addon successfully deployed.
Checking CNI Installation at /opt/cni/bin
WARNING : The kubelet is configured to not fail even if swap is enabled; production deployments should disable swap unless testing NodeSwap feature.
2022/12/01 06:59:07 [INFO] generate received request
2022/12/01 06:59:07 [INFO] received CSR
2022/12/01 06:59:07 [INFO] generating key: rsa-2048
2022/12/01 06:59:07 [INFO] encoded CSR
2022/12/01 06:59:07 [INFO] signed certificate with serial number 396503892652732852810951929324414744717912185576
kubelet ( 27489 ) is running.
wait kubelet ready
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
127.0.0.1   NotReady   <none>   1s    v1.24.0-dirty
2022/12/01 06:59:22 [INFO] generate received request
2022/12/01 06:59:22 [INFO] received CSR
2022/12/01 06:59:22 [INFO] generating key: rsa-2048
2022/12/01 06:59:22 [INFO] encoded CSR
2022/12/01 06:59:22 [INFO] signed certificate with serial number 270579443091576203614576860806582064911709216666
Create default storage class for 
storageclass.storage.k8s.io/standard created
Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

Logs:
  /tmp/kube-apiserver.log
  /tmp/kube-controller-manager.log
  
  /tmp/kube-proxy.log
  /tmp/kube-scheduler.log
  /tmp/kubelet.log

To start using your cluster, you can open up another terminal/tab and run:

  export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
  cluster/kubectl.sh

Alternatively, you can write to the default kubeconfig:

  export KUBERNETES_PROVIDER=local

  cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
  cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
  cluster/kubectl.sh config set-context local --cluster=local --user=myself
  cluster/kubectl.sh config use-context local
  cluster/kubectl.sh

结束时间：

root@cubmaster01:~/go/src/k8s.io/kubernetes# date
Thu 01 Dec 2022 07:04:00 AM UTC

停止

stop：ctrl + c

日志

Logs：

• /tmp/kube-apiserver.log
• /tmp/kube-controller-manager.log
• /tmp/kube-proxy.log
• /tmp/kube-scheduler.log
• /tmp/kubelet.log

test：编写 pod.yaml

创建文件并编写内容：

mkdir ~/go-yaml; cd ~/go-yaml
vim pod.yaml

输入以下：

apiVersion: v1
kind: Pod
metadata: 
  name: busybox
  namespaces: default
spec:
  containers: 
  - name: busybox
    image: busybox
    command: 
  	  - sleep
  	  - "3600"
  	imagePullPolicy: IfNotPresent
  restartPolicy: Always

创建它：

cluster/kubectl.sh create -f ~/go-yaml/pod.yaml

获取：

cluster/kubectl.sh get pod

细节：

cluster/kubectl.sh describe pod busybox
cluster/kubectl.sh describe node

再次运行脚本

注意

我们一般再次运行的时候，编译成 exe 的动作也是被缓存了，不会再生成，所以修改后 debug 删除：

sudo make clean

开启本地debug功能

需要开启 debug

一般编译单个组建，或者是编译所有的组件，都需要开启 debug 调试。

我们为了方便，在任何时候都把 debug 能力 build 过去。

cd $GOPATH/src/k8s.io/kubernetes
# kubernetes go编译文件
sudo vim ./hack/lib/golang.sh
# 查找build_binaries()函数 vi语法
?build_binaries()

找到一下bebug判断，注释，一直开启debug能力

	gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}"
    if [[ "${DBG:-}" == 1 ]]; then
        # Debugging - disable optimizations and inlining.
        gogcflags="${gogcflags} -N -l"
    fi

    goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}"
    if [[ "${DBG:-}" != 1 ]]; then
        # Not debugging - disable symbols and DWARF.
        goldflags="${goldflags} -s -w"
    fi

注释判断，将debug直接放在下面， 再保存即可：

	gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}"
    # if [[ "${DBG:-}" == 1 ]]; then
    #     # Debugging - disable optimizations and inlining.
    #     gogcflags="${gogcflags} -N -l"
    # fi
	gogcflags="${gogcflags} -N -l"
    goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}"
    # if [[ "${DBG:-}" != 1 ]]; then
    #     # Not debugging - disable symbols and DWARF.
    #     goldflags="${goldflags} -s -w"
    # fi

观察此时的进程：

delve 调试

或许你可以选择 GDB，或许也有一个更适合的方式：go-delve:

提示

• delve

特性

• 本地调试 和 远程调试
• 简单易用，开源

启动本地集群从而 Debug - 以 API Server 为例

提示

正好之前的 API server 证书出现了问题，这个问题的地址 ISSUE：

• 【An error occurred attempting to install kubernetes learning】 #114222

编译参数

• 修改 hack/lib/golang.sh 文件，从而使得编译器不去优化掉支持的 debug 信息

• 禁止 -w 、-s 保留文件名，行号

• 加上 -gcflags= "all=-N -l" ，禁止优化和内联。

在k8s.io/kubernetes/hack/lib/golang.sh中设置了-s -w选项来禁用符号表以及debug信息，因此在编译 Kubernetes 组件进行远程调试时需要去掉这两个限制，如下：

-    goldflags="${GOLDFLAGS=-s -w} $(kube::version::ldflags)"
+    #goldflags="${GOLDFLAGS=-s -w} $(kube::version::ldflags)"
+    goldflags="${GOLDFLAGS:-} $(kube::version::ldflags)"

启动本地集群

如果需要，用 make clean 清除未编译的可执行程序。通过 hack/local-up-cluster.sh 脚本启动本地集群。

重启 API Server

如果要调试 API Server ，先关闭其进程，再以 dlv 启动。

⚠️ 为什么要重启，因为 go 的 debug 依靠 dlv，所以需要用 dlv 包袱再启动下。

连接 Debug Server

通过 delve 连接 debug server 并且开始调试。

杀死 Api Server 调试

查看 Api Server 的应用程序和后面的参数，这个很重要：

root@cubmaster01:~/go/src/k8s.io/kubernetes#  ps -ef | grep kube-apiserver
root       27052    8091 10 06:58 pts/0    00:00:43 /root/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins=/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$
root       33017    9251  0 07:05 pts/1    00:00:00 grep --color=auto kube-apiserver

杀死

kill -9 {api-server PID}

启动：

创建日志信息保存目录：mkdir -p /root/kubelog/delve-log

sudo dlv  --headless exec /root/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --listen=127.0.0.1:2345 --api-version=2 --log --log-output=debugger,gdbwire,lidbout,debuglineerr,rpc,dap,fncall,minidump --log-dest=/root/kubelog/delve-log/log -- --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins="/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$"

另一台机器连接：

dlv connect localhost:2345

vscode 远程调试方案

{
    // 使用 IntelliSense 了解相关属性。 
    // 悬停以查看现有属性的描述。
    // 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to server",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            // "remotePath": "${workspaceFolder}",
            "port": 2345,
            "host": "192.168.71.130"
        }
    ]
}

在远程启动 API Server：

和上面的启动一样：

END 链接

• ⬆️上一节🔗 ️下一节🔗

• Ⓜ️回到目录🏠

• 🫵参与贡献💞❤️‍🔥💖)

• ✴️版权声明 © ：本书所有内容遵循CC-BY-SA 3.0协议（署名-相同方式共享）©