运维开发网

k8s中部署prometheus常用exporter

运维开发网 https://www.qedev.com 2020-04-14 14:49 出处:51CTO 作者:大牙啊
k8s中常用的几个exporter部署

部署kube-state-metrics, kube-state-metrics用来获取k8s集群所有资源的状态:

准备镜像:

[root@hdss7-200 ~]# docker pull quay.io/coreos/kube-state-metrics:v1.5.0
v1.5.0: Pulling from coreos/kube-state-metrics
cd784148e348: Pull complete
f622528a393e: Pull complete
Digest: sha256:b7a3143bd1eb7130759c9259073b9f239d0eeda09f5210f1cd31f1a530599ea1
Status: Downloaded newer image for quay.io/coreos/kube-state-metrics:v1.5.0
quay.io/coreos/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker images|grep kube-state-metrics
quay.io/coreos/kube-state-metrics          v1.5.0                     91599517197a        15 months ago       31.8MB
[root@hdss7-200 ~]# docker tag  91599517197a harbor.od.com/public/kube-state-metrics:v1.5.0
[root@hdss7-200 ~]# docker push harbor.od.com/public/kube-state-metrics:v1.5.0
The push refers to repository [harbor.od.com/public/kube-state-metrics]
5b3c36501a0a: Pushed
7bff100f35cb: Pushed
v1.5.0: digest: sha256:16e9a1d63e80c19859fc1e2727ab7819f89aeae5f8ab5c3380860c2f88fe0a58 size: 739

准备资源配置清单:

[root@hdss7-200 kube-state-metrics]# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system
[root@hdss7-200 kube-state-metrics]# cat dp.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    grafanak8sapp: "true"
    app: kube-state-metrics
  name: kube-state-metrics
  namespace: kube-system
spec:
  selector:
    matchLabels:
      grafanak8sapp: "true"
      app: kube-state-metrics
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        grafanak8sapp: "true"
        app: kube-state-metrics
    spec:
      containers:
      - name: kube-state-metrics
        image: harbor.od.com/public/kube-state-metrics:v1.5.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http-metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
      serviceAccountName: kube-state-metrics

应用资源配置清单:

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/rbac.yaml
serviceaccount/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/kube-state-metrics/dp.yaml
deployment.extensions/kube-state-metrics created
[root@hdss7-21 ~]# kubectl get pod -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-6b6c4f9648-rrgfx                1/1     Running   0          49m
kube-state-metrics-8669f776c6-gb6nd     0/1     Running   0          15s
kubernetes-dashboard-76dcdb4677-c847s   1/1     Running   0          38m
traefik-ingress-h2jpc                   1/1     Running   0          2d
traefik-ingress-k5hgk                   1/1     Running   0          2d
[root@hdss7-21 ~]# curl 172.7.21.9:8080/healthz
ok

部署node-exporter,node-exporter是帮我们监控宿主机的资源:

准备镜像:

[root@hdss7-200 ~]# docker pull prom/node-exporter:v0.15.0
v0.15.0: Pulling from prom/node-exporter
Image docker.io/prom/node-exporter:v0.15.0 uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
aa3e9481fcae: Pull complete
a3ed95caeb02: Pull complete
afc308b02dc6: Pull complete
4cafbffc9d4f: Pull complete
Digest: sha256:a59d1f22610da43490532d5398b3911c90bfa915951d3b3e5c12d3c0bf8771c3
Status: Downloaded newer image for prom/node-exporter:v0.15.0
docker.io/prom/node-exporter:v0.15.0
[root@hdss7-200 ~]# docker images|grep node-exporter:v0.15.0
[root@hdss7-200 ~]# docker images|grep node-exporter
prom/node-exporter                         v0.15.0                    12d51ffa2b22        2 years ago         22.8MB
[root@hdss7-200 ~]# docker tag 12d51ffa2b22 harbor.od.com/public/node-exporter:v0.15.0
[root@hdss7-200 ~]# docker push harbor.od.com/public/node-exporter:v0.15.0
The push refers to repository [harbor.od.com/public/node-exporter]
5f70bf18a086: Mounted from public/pause
1c7f6350717e: Pushed
a349adf62fe1: Pushed
c7300f623e77: Pushed
v0.15.0: digest: sha256:57d9b335b593e4d0da1477d7c5c05f23d9c3dc6023b3e733deb627076d4596ed size: 1979

准备资源配置清单:

[root@hdss7-200 node-exporter]# cat ds.yaml
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    daemon: "node-exporter"
    grafanak8sapp: "true"
spec:
  selector:
    matchLabels:
      daemon: "node-exporter"
      grafanak8sapp: "true"
  template:
    metadata:
      name: node-exporter
      labels:
        daemon: "node-exporter"
        grafanak8sapp: "true"
    spec:
      volumes:
      - name: proc
        hostPath:
          path: /proc
          type: ""
      - name: sys
        hostPath:
          path: /sys
          type: ""
      containers:
      - name: node-exporter
        image: harbor.od.com/public/node-exporter:v0.15.0
        imagePullPolicy: IfNotPresent
        args:
        - --path.procfs=/host_proc
        - --path.sysfs=/host_sys
        ports:
        - name: node-exporter
          hostPort: 9100
          containerPort: 9100
          protocol: TCP
        volumeMounts:
        - name: sys
          readOnly: true
          mountPath: /host_sys
        - name: proc
          readOnly: true
          mountPath: /host_proc
      hostNetwork: true
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/node-exporter/ds.yaml
daemonset.extensions/node-exporter created

因为node-exporter部署的方式是daemonset,在每个运算节点都会运行一个POD

[root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP            NODE                NOMINATED NODE   READINESS GATES
coredns-6b6c4f9648-rrgfx                1/1     Running   0          63m    172.7.22.7    hdss7-22.host.com   <none>           <none>
kube-state-metrics-8669f776c6-gb6nd     1/1     Running   0          14m    172.7.21.9    hdss7-21.host.com   <none>           <none>
kubernetes-dashboard-76dcdb4677-c847s   1/1     Running   0          52m    172.7.22.10   hdss7-22.host.com   <none>           <none>
node-exporter-gn2hb                     1/1     Running   0          3m     10.4.7.22     hdss7-22.host.com   <none>           <none>
node-exporter-nzww6                     1/1     Running   0          3m     10.4.7.21     hdss7-21.host.com   <none>           <none>
traefik-ingress-h2jpc                   1/1     Running   0          2d1h   172.7.21.3    hdss7-21.host.com   <none>           <none>
traefik-ingress-k5hgk                   1/1     Running   0          2d     172.7.22.2    hdss7-22.host.com   <none>           <none>

部署cadvisor,cadvisor是帮我们去向kubelet要每个容器所消耗的资源:

准备镜像:

[root@hdss7-200 node-exporter]# docker pull google/cadvisor:v0.28.3
v0.28.3: Pulling from google/cadvisor
ab7e51e37a18: Pull complete
a2dc2f1bce51: Pull complete
3b017de60d4f: Pull complete
Digest: sha256:9e347affc725efd3bfe95aa69362cf833aa810f84e6cb9eed1cb65c35216632a
Status: Downloaded newer image for google/cadvisor:v0.28.3
docker.io/google/cadvisor:v0.28.3
[root@hdss7-200 node-exporter]#
[root@hdss7-200 node-exporter]#
[root@hdss7-200 node-exporter]# docker images|grep google/cadvisor
google/cadvisor                            v0.28.3                    75f88e3ec333        2 years ago         62.2MB
[root@hdss7-200 node-exporter]# docker tag 75f88e3ec333 harbor.od.com/public/cadvisor:v0.28.3
[root@hdss7-200 node-exporter]# docker push harbor.od.com/public/cadvisor:v0.28.3
The push refers to repository [harbor.od.com/public/cadvisor]
f60e27acaccf: Pushed
f04a25da66bf: Pushed
52a5560f4ca0: Pushed
v0.28.3: digest: sha256:34d9d683086d7f3b9bbdab0d1df4518b230448896fa823f7a6cf75f66d64ebe1 size: 951

修改所有运算节点软连接:

mount -o remount,rw /sys/fs/cgroup/
ln -s /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct,cpu

应用资源配置清单:

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/cadvisor/ds.yaml
daemonset.apps/cadvisor created
[root@hdss7-22 ~]# kubectl -n kube-system get pod
NAME                                    READY   STATUS    RESTARTS   AGE
cadvisor-cdtzg                          1/1     Running   0          12s
cadvisor-pbmb7                          1/1     Running   0          12s
coredns-6b6c4f9648-rrgfx                1/1     Running   0          82m
kube-state-metrics-8669f776c6-gb6nd     1/1     Running   0          33m
kubernetes-dashboard-76dcdb4677-c847s   1/1     Running   0          71m
node-exporter-gn2hb                     1/1     Running   0          22m
node-exporter-nzww6                     1/1     Running   0          22m
traefik-ingress-h2jpc                   1/1     Running   0          2d1h
traefik-ingress-k5hgk                   1/1     Running   0          2d1h

部署blackbox-exporter:

[root@hdss7-200 cadvisor]# docker pull prom/blackbox-exporter:v0.15.1
v0.15.1: Pulling from prom/blackbox-exporter
8e674ad76dce: Pull complete
e77d2419d1c2: Pull complete
969c24328c68: Pull complete
d9df4d63dd8a: Pull complete
Digest: sha256:0ccbb0bb08bbc00f1c765572545e9372a4e4e4dc9bafffb1a962024f61d6d996
Status: Downloaded newer image for prom/blackbox-exporter:v0.15.1
docker.io/prom/blackbox-exporter:v0.15.1
[root@hdss7-200 cadvisor]# docker images|grep black
prom/blackbox-exporter                     v0.15.1                    81b70b6158be        6 months ago        19.7MB
[root@hdss7-200 cadvisor]# docker tag 81b70b6158be harbor.od.com/public/blackbox-exporter:v0.15.1
[root@hdss7-200 cadvisor]# docker push harbor.od.com/public/blackbox-exporter:v0.15.1
The push refers to repository [harbor.od.com/public/blackbox-exporter]
2e93bab0c159: Pushed
4f2b5ab68d7f: Pushed
3163e6173fcc: Pushed
6194458b07fc: Pushed
v0.15.1: digest: sha256:f7c335cc7898c6023346a0d5fba8566aca4703b69d63be8dc5367476c77cf2c4 size: 1155

准备资源配置清单:

[root@hdss7-200 blackbox-exporter]# cat cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: blackbox-exporter
  name: blackbox-exporter
  namespace: kube-system
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
        timeout: 2s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          valid_status_codes: [200,301,302]
          method: GET
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 2s
[root@hdss7-200 blackbox-exporter]# cat dp.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: blackbox-exporter
  namespace: kube-system
  labels:
    app: blackbox-exporter
  annotations:
    deployment.kubernetes.io/revision: 1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blackbox-exporter
  template:
    metadata:
      labels:
        app: blackbox-exporter
    spec:
      volumes:
      - name: config
        configMap:
          name: blackbox-exporter
          defaultMode: 420
      containers:
      - name: blackbox-exporter
        image: harbor.od.com/public/blackbox-exporter:v0.15.1
        imagePullPolicy: IfNotPresent
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --log.level=info
        - --web.listen-address=:9115
        ports:
        - name: blackbox-port
          containerPort: 9115
          protocol: TCP
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 50Mi
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
[root@hdss7-200 blackbox-exporter]# cat svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
  selector:
    app: blackbox-exporter
  ports:
    - name: blackbox-port
      protocol: TCP
      port: 9115
[root@hdss7-200 blackbox-exporter]# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: blackbox-exporter
  namespace: kube-system
spec:
  rules:
  - host: blackbox.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: blackbox-exporter
          servicePort: blackbox-port

应用资源配置清单:

[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/cm.yaml
configmap/blackbox-exporter created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/dp.yaml
deployment.extensions/blackbox-exporter created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/svc.yaml
service/blackbox-exporter created
[root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/blackbox-exporter/ingress.yaml
ingress.extensions/blackbox-exporter created
[root@hdss7-21 ~]# kubectl get pod -n kube-system -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP            NODE                NOMINATED NODE   READINESS GATES
blackbox-exporter-659fc46b55-p8wxp      0/1     Running   0          19s    172.7.22.11   hdss7-22.host.com   <none>           <none>
cadvisor-cdtzg                          1/1     Running   0          15m    10.4.7.21     hdss7-21.host.com   <none>           <none>
cadvisor-pbmb7                          1/1     Running   0          15m    10.4.7.22     hdss7-22.host.com   <none>           <none>
coredns-6b6c4f9648-rrgfx                1/1     Running   0          97m    172.7.22.7    hdss7-22.host.com   <none>           <none>

k8s中部署prometheus常用exporter

0

精彩评论

暂无评论...
验证码 换一张
取 消