最近研究OpenShift virtulization, 各种Pod对KVM进程的封装，引发了Pod中进程到底在Node中是什么表现形势的好奇，因为对基础知识的不扎实，还是希望找个环境能仔细看看，建立起openshift 4.12的环境后，首先列出某个节点上的所有的Pod

[lab-user@bastion ~]$ oc get pods -A --field-selector=spec.host=ip-10-0-148-57.us-east-2.compute.internal
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-cluster-csi-drivers aws-ebs-csi-driver-node-c4ddr 3/3 Running 0 46m
openshift-cluster-node-tuning-operator tuned-5lrjg 1/1 Running 0 46m
openshift-console downloads-595667555c-477b2 1/1 Running 0 39m
openshift-debug-b7f7z ip-10-0-148-57.us-east-2.compute.internal-debug 1/1 Running 0 22m
openshift-dns dns-default-cxr55 2/2 Running 0 45m
openshift-dns node-resolver-szmvw 1/1 Running 0 46m
openshift-image-registry image-registry-6945b97db4-gnjfl 1/1 Running 0 44m
openshift-image-registry node-ca-56rgw 1/1 Running 0 44m
openshift-ingress-canary ingress-canary-7twkk 1/1 Running 0 44m
openshift-ingress router-default-74f6d94f48-cwczb 1/1 Running 0 30m
openshift-machine-config-operator machine-config-daemon-fgchf 2/2 Running 0 46m
openshift-monitoring alertmanager-main-1 6/6 Running 1 (39m ago) 39m
openshift-monitoring node-exporter-w8f5k 2/2 Running 0 44m
openshift-monitoring prometheus-adapter-b87c6d546-7jlzw 1/1 Running 0 39m
openshift-monitoring prometheus-k8s-1 6/6 Running 0 39m
openshift-monitoring prometheus-operator-admission-webhook-566d9cc57c-mfwc8 1/1 Running 0 52m
openshift-monitoring thanos-querier-7dcb8856d-d8g8t 6/6 Running 0 39m
openshift-multus multus-8cgln 1/1 Running 0 46m
openshift-multus multus-additional-cni-plugins-cpdbf 1/1 Running 0 46m
openshift-multus network-metrics-daemon-5rjqw 2/2 Running 0 46m
openshift-network-diagnostics network-check-source-6655898df9-p9qjz 1/1 Running 0 54m
openshift-network-diagnostics network-check-target-5qlvj 1/1 Running 0 46m
openshift-operator-lifecycle-manager collect-profiles-28103550-nrvlz 0/1 Completed 0 40m
openshift-operator-lifecycle-manager collect-profiles-28103565-lt2ct 0/1 Completed 0 28m
openshift-operator-lifecycle-manager collect-profiles-28103580-nwh94 0/1 Completed 0 13m
openshift-ovn-kubernetes ovnkube-node-p4dld 5/5 Running 0 46m

以prometheus-adapter-xxxx为例，登录进去查看一下Pod里面运行的进程

[lab-user@bastion ~]$ oc project openshift-monitoring
Already on project "openshift-monitoring" on server "https://api.cluster-gmwqp.gmwqp.sandbox1496.opentlc.com:6443".
[lab-user@bastion ~]$ oc rsh prometheus-adapter-b87c6d546-7jlzw
sh-4.4$ ps -ef
UID PID PPID C STIME TTY TIME CMD
1000430+ 1 0 0 08:34 ? 00:00:13 /usr/bin/adapter --prometheus-auth-config=/etc/prometheus-config/prometheus-config.yaml --config=/etc/adapter/config.yaml --logtostderr=
1000430+ 37 0 1 09:15 pts/0 00:00:00 /bin/sh
1000430+ 43 37 0 09:15 pts/0 00:00:00 ps -ef

比较简单，只有一个，然后登陆到节点查看相关进程

sh-4.4# ps -ef | grep prometheus-adapter
root 11677 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966/userdata -c 1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-adapter-b87c6d546-7jlzw_18617999-d875-40e2-b29f-c40743f8314b/prometheus-adapter/0.log --log-level info -n k8s_prometheus-adapter_prometheus-adapter-b87c6d546-7jlzw_openshift-monitoring_18617999-d875-40e2-b29f-c40743f8314b_0 -P /run/containers/storage/overlay-containers/1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 1961af83ec2bf15ae89371edd5ebb534886c9db6bfaa385ac4fb53b4232df966 -s
root 64634 35507 0 09:16 pts/1 00:00:00 grep prometheus-adapter

可见Pod里面的进程和Nodes上的进程是一对一的关系，我们再找一个多进程的，比如prometheus-k8s-1

因为rsh进入pod以后发现没有ps,那么我们登录节点看看

sh-4.4# crictl ps | grep prometheus-k8s-1
1b1d29d044d4f 8dd80e25afda5c9466a26ef90346b419a8cc9840bd8b553257a0c6df2c5bac6c 42 minutes ago Running kube-rbac-proxy-thanos 0 9d74de5a22175 prometheus-k8s-1
bacbcddd17d2c 8dd80e25afda5c9466a26ef90346b419a8cc9840bd8b553257a0c6df2c5bac6c 42 minutes ago Running kube-rbac-proxy 0 9d74de5a22175 prometheus-k8s-1
22d9f341cb2b9 e1f27569f2e6745ac23c33e89b8e569069970ab56a54114e93f6a31570b86334 42 minutes ago Running prometheus-proxy 0 9d74de5a22175 prometheus-k8s-1
a36c49e125c9a 4efdbf6a731d2feb217ecd0fb0d792dcadf854e826d4db235cafc0268b2621d4 42 minutes ago Running thanos-sidecar 0 9d74de5a22175 prometheus-k8s-1
76140a26677b1 c9868e04bec41eda58502d70bacc3f2cb5084f5d643b76fb018e3076f7d38014 42 minutes ago Running config-reloader 0 9d74de5a22175 prometheus-k8s-1
2b0aa6f218f55 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97ffae63353c694fe66615679429351a96711cbdeb400555181dea8688a388ea 42 minutes ago Running prometheus

sh-4.4# crictl ps | grep prometheus-k8s-1 | wc -l
6

然后通过ps查看

sh-4.4# ps -ef | grep prometheus-k8s-1
root 12918 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316/userdata -c 2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/prometheus/0.log --log-level info -n k8s_prometheus_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 2b0aa6f218f552f0d1009b07a80d23d9d47c8b63c23777fb51dcb1aa648d0316 -s
root 12978 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66/userdata -c 76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/config-reloader/0.log --log-level info -n k8s_config-reloader_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 76140a26677b1512e18864b5c1bfb31a2847d581721a64cd7f7633fb2a62ab66 -s
root 13037 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730/userdata -c a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/thanos-sidecar/0.log --log-level info -n k8s_thanos-sidecar_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u a36c49e125c9a385e8e32e904e61c404c40135a1c5040d01244692e1a930f730 -s
root 13087 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec/userdata -c 22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/prometheus-proxy/0.log --log-level info -n k8s_prometheus-proxy_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 22d9f341cb2b9f8748852a5b83146844cebb8ec2e4e00ce17016d523f5edb1ec -s
root 13142 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369/userdata -c bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/kube-rbac-proxy/0.log --log-level info -n k8s_kube-rbac-proxy_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u bacbcddd17d2cfd032c0e890783018bec8d98a06b373918576b0de533332b369 -s
root 13194 1 0 08:34 ? 00:00:00 /usr/bin/conmon -b /run/containers/storage/overlay-containers/1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6/userdata -c 1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6 --exit-dir /var/run/crio/exits -l /var/log/pods/openshift-monitoring_prometheus-k8s-1_0a15cd78-757d-4322-9f35-94f7c42860b4/kube-rbac-proxy-thanos/0.log --log-level info -n k8s_kube-rbac-proxy-thanos_prometheus-k8s-1_openshift-monitoring_0a15cd78-757d-4322-9f35-94f7c42860b4_0 -P /run/containers/storage/overlay-containers/1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6/userdata/conmon-pidfile -p /run/containers/storage/overlay-containers/1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6/userdata/pidfile --persist-dir /var/lib/containers/storage/overlay-containers/1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6/userdata -r /usr/bin/runc --runtime-arg --root=/run/runc --socket-dir-path /var/run/crio --syslog -u 1b1d29d044d4fadd61fdc8b3e040c4aaa9a8fb2690cffb562d170d3952dbc9d6 -s
root 69497 35507 0 09:20 pts/1 00:00:00 grep prometheus-k8s-1

sh-4.4# ps -ef | grep prometheus-k8s-1 | wc -l
7

所以基本是对映关系。

关于为什么是通过conmon把容器运行起来，以及整体的架构大家可以去参考cri-o的官网和整个过程

https://cri-o.io/