kubernetes
Istio CNI 플러그인과 Pod Security Admission
misankim
2024. 6. 15. 20:51
https://istio.io/latest/docs/setup/additional-setup/cni/
Install Istio with the Istio CNI plugin
Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.
istio.io
https://istio.io/latest/docs/setup/additional-setup/pod-security-admission/
Install Istio with Pod Security Admission
Install and use Istio with the Pod Security admission controller.
istio.io
Istio CNI 플러그인 활성화하여 설치
kubernetes 의 pod security admission 을 baseline 정책으로 enforce 할 경우(enforce=baseline) istio-init 컨테이너의 NET_ADMIN, NET_RAW 에 대한 capabilities 옵션 때문에 파드 생성 자체가 불가하다.
Istio CNI 플러그인을 활성화하면 istio-init 컨테이너와 istio-proxy 컨테이너는 주입되지만 istio-init 컨테이너가 실제로 iptables 조작을 하지 않고, 그 역할을 istio-cni-node 데몬셋이 대신하기 때문에 capabilities 관련 정책을 위반하지 않기 때문에 pod security admission 을 적용할 수 있다.
pod security 에 의해 파드 생성이 차단된 로그
Warning FailedCreate 17s (x9 over 56s) replicaset-controller (combined from similar events): Error creating: pods "bootcamp-96886dc5b-8r4tj" is forbidden: violates PodSecurity "baseline:latest": non-default capabilities (container "istio-init" must not include "NET_ADMIN", "NET_RAW" in securityContext.capabilities.add)
istio operator 에 Istio CNI 플러그인 활성화하도록 설정
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
...
components:
cni:
enabled: true
...
GKE(Google Kubernetes Engine) 환경에서 Istio CNI 플러그인 활성화
https://istio.io/latest/docs/setup/additional-setup/cni/#hosted-kubernetes-settings
Install Istio with the Istio CNI plugin
Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.
istio.io
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
...
components:
cni:
enabled: true
namespace: kube-system
values:
cni:
cniBinDir: /home/kubernetes/bin
...
Istio CNI 플러그인 활성화 전/후의 istio-init 컨테이너의 스펙
활성화 전
initContainers:
- args:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- '*'
- -d
- 15090,15021,15020
- --log_output_level=default:info
image: docker.io/istio/proxyv2:1.21.2
imagePullPolicy: IfNotPresent
name: istio-init
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 10m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_ADMIN
- NET_RAW
drop:
- ALL
privileged: false
readOnlyRootFilesystem: false
runAsGroup: 0
runAsNonRoot: false
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-6r4n8
readOnly: true
활성화 후
initContainers:
- args:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- '*'
- -x
- ""
- -b
- '*'
- -d
- 15090,15021,15020
- --log_output_level=default:info
- --run-validation
- --skip-rule-apply
image: docker.io/istio/proxyv2:1.21.2
imagePullPolicy: IfNotPresent
name: istio-validation
resources:
limits:
cpu: "2"
memory: 1Gi
requests:
cpu: 10m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
runAsGroup: 1337
runAsNonRoot: true
runAsUser: 1337
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-25b2p
readOnly: true
pod security 활성화
-> istio-init 컨테이너로 인해 파드 생성이 차단되지 않는지 확인(enforce=baseline)
https://kubernetes.io/docs/concepts/security/pod-security-standards/
Pod Security Standards
A detailed look at the different policy levels defined in the Pod Security Standards.
kubernetes.io
https://kubernetes.io/docs/concepts/security/pod-security-admission/
Pod Security Admission
An overview of the Pod Security Admission Controller, which can enforce the Pod Security Standards.
kubernetes.io
kubectl label ns bootcamp pod-security.kubernetes.io/warn=baseline
kubectl label ns bootcamp pod-security.kubernetes.io/audit=baseline
kubectl label ns bootcamp pod-security.kubernetes.io/enforce=baseline
(참고) 생성된 Istio CNI 플러그인 데몬셋
➜ [istio] k get ds -n istio-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
istio-cni-node 1 1 1 1 1 kubernetes.io/os=linux 8m54s
(참고) docker desktop 에서 실행한 로컬 kubernetes 클러스터에서의 cni 는 istio cni 적용 불가
https://istio.io/latest/docs/setup/additional-setup/cni/#prerequisites
Install Istio with the Istio CNI plugin
Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.
istio.io
Warning FailedCreatePodSandBox 4m59s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "5783eafdc481c5c1925334fe2514c938a77be91c6c8f436f8c9c5db225a3c1e3" network for pod "bootcamp-5b67dc74bb-8smzk": networkPlugin cni failed to set up pod "bootcamp-5b67dc74bb-8smzk_bootcamp" network: plugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "5783eafdc481c5c1925334fe2514c938a77be91c6c8f436f8c9c5db225a3c1e3" network for pod "bootcamp-5b67dc74bb-8smzk": networkPlugin cni failed to teardown pod "bootcamp-5b67dc74bb-8smzk_bootcamp" network: plugin type="loopback" failed (delete): missing network name]
(참고) istio-cni-node 파드 로그
-> 노드에 istio-proxy 컨테이너 주입이 활성화된 파드 배치 시 자동으로 iptables 세팅
2024-05-10T01:52:31.053848Z info cni ============= Start iptables configuration for bootcamp-764dbb76f4-7688l =============
2024-05-10T01:52:31.055613Z info cni Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=
2024-05-10T01:52:31.055706Z info cni Istio iptables variables:
IPTABLES_VERSION=
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020,15021,15090
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=15020
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
NETWORK_NAMESPACE=/var/run/netns/cni-2ac62102-d38c-f082-047c-828feb67b467
CNI_MODE=true
EXCLUDE_INTERFACES=
2024-05-10T01:52:31.055774Z info cni Running iptables-restore with the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2024-05-10T01:52:31.055821Z info cni Running command (without lock by mount and nss): iptables-restore --noflush
2024-05-10T01:52:31.060019Z info cni Running ip6tables-restore with the following input:
2024-05-10T01:52:31.060107Z info cni Running command (without lock by mount and nss): ip6tables-restore --noflush
2024-05-10T01:52:31.062294Z info cni Running command (without nss): iptables-save
2024-05-10T01:52:31.065981Z info cni Command output:
# Generated by iptables-save v1.8.5 on Fri May 10 01:52:31 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Fri May 10 01:52:31 2024
2024-05-10T01:52:31.066106Z info cni ============= End iptables configuration for bootcamp-764dbb76f4-7688l =============
만약 노드에 배치된 파드가 istio-proxy 컨테이너가 없는 경우 iptables 룰 주입을 하지 않음
2024-05-10T01:56:07.895173Z info cni excluded because it does not have istio-proxy container (have [bootcamp])
샘플 매니페스트 작성
apiVersion: apps/v1
kind: Deployment
metadata:
name: bootcamp
spec:
selector:
matchLabels:
app: bootcamp
replicas: 5
template:
metadata:
labels:
app: bootcamp
spec:
containers:
- image: gcr.io/google-samples/kubernetes-bootcamp:v1
imagePullPolicy: Always
name: bootcamp
ports:
- containerPort: 8080
resources:
limits:
cpu: 200m
memory: 256Mi
ephemeral-storage: 1Gi
requests:
cpu: 200m
memory: 256Mi
ephemeral-storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: bootcamp
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
type: LoadBalancer
selector:
app: bootcamp
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: bootcamp
spec:
gateways:
- istio-system/istio-ingressgateway
hosts:
- www.example.com
http:
- match:
- uri:
prefix: /
route:
- destination:
host: bootcamp.bootcamp.svc.cluster.local
port:
number: 80
호출 테스트
kubectl exec -it bootcamp-84bb8b8cd-5r5rm -- curl -v -H 'Host: www.example.com' http://istio-ingressgateway.istio-system.svc.cluster.local/