본문 바로가기
kubernetes

Istio CNI 플러그인과 Pod Security Admission

by misankim 2024. 6. 15.

https://istio.io/latest/docs/setup/additional-setup/cni/

 

Install Istio with the Istio CNI plugin

Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.

istio.io

 

https://istio.io/latest/docs/setup/additional-setup/pod-security-admission/

 

Install Istio with Pod Security Admission

Install and use Istio with the Pod Security admission controller.

istio.io

 

Istio CNI 플러그인 활성화하여 설치

 

kubernetes 의 pod security admission 을 baseline 정책으로 enforce 할 경우(enforce=baseline) istio-init 컨테이너의 NET_ADMIN, NET_RAW 에 대한 capabilities 옵션 때문에 파드 생성 자체가 불가하다.

 

Istio CNI 플러그인을 활성화하면 istio-init 컨테이너와 istio-proxy 컨테이너는 주입되지만 istio-init 컨테이너가 실제로 iptables 조작을 하지 않고, 그 역할을 istio-cni-node 데몬셋이 대신하기 때문에 capabilities 관련 정책을 위반하지 않기 때문에 pod security admission 을 적용할 수 있다.

 

 

pod security 에 의해 파드 생성이 차단된 로그

 

Warning  FailedCreate      17s (x9 over 56s)  replicaset-controller  (combined from similar events): Error creating: pods "bootcamp-96886dc5b-8r4tj" is forbidden: violates PodSecurity "baseline:latest": non-default capabilities (container "istio-init" must not include "NET_ADMIN", "NET_RAW" in securityContext.capabilities.add)

 

 

istio operator 에 Istio CNI 플러그인 활성화하도록 설정

 

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
...
  components:
    cni:
      enabled: true
...

 

 

GKE(Google Kubernetes Engine) 환경에서 Istio CNI 플러그인 활성화

 

https://istio.io/latest/docs/setup/additional-setup/cni/#hosted-kubernetes-settings

 

Install Istio with the Istio CNI plugin

Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.

istio.io

 

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
...
  components:
    cni:
      enabled: true
      namespace: kube-system
  values:
    cni:
      cniBinDir: /home/kubernetes/bin
...

 

 

Istio CNI 플러그인 활성화 전/후의 istio-init 컨테이너의 스펙

 

활성화 전

  initContainers:
  - args:
    - istio-iptables
    - -p
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - '*'
    - -d
    - 15090,15021,15020
    - --log_output_level=default:info
    image: docker.io/istio/proxyv2:1.21.2
    imagePullPolicy: IfNotPresent
    name: istio-init
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-6r4n8
      readOnly: true

 

 

활성화 후

  initContainers:
  - args:
    - istio-iptables
    - -p
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - -m
    - REDIRECT
    - -i
    - '*'
    - -x
    - ""
    - -b
    - '*'
    - -d
    - 15090,15021,15020
    - --log_output_level=default:info
    - --run-validation
    - --skip-rule-apply
    image: docker.io/istio/proxyv2:1.21.2
    imagePullPolicy: IfNotPresent
    name: istio-validation
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1337
      runAsNonRoot: true
      runAsUser: 1337
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-25b2p
      readOnly: true

 

 

pod security 활성화

-> istio-init 컨테이너로 인해 파드 생성이 차단되지 않는지 확인(enforce=baseline)

https://kubernetes.io/docs/concepts/security/pod-security-standards/

 

Pod Security Standards

A detailed look at the different policy levels defined in the Pod Security Standards.

kubernetes.io

 

https://kubernetes.io/docs/concepts/security/pod-security-admission/

 

Pod Security Admission

An overview of the Pod Security Admission Controller, which can enforce the Pod Security Standards.

kubernetes.io

 

kubectl label ns bootcamp pod-security.kubernetes.io/warn=baseline
kubectl label ns bootcamp pod-security.kubernetes.io/audit=baseline
kubectl label ns bootcamp pod-security.kubernetes.io/enforce=baseline

 

 

(참고) 생성된 Istio CNI 플러그인 데몬셋

 

➜  [istio] k get ds -n istio-system
NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
istio-cni-node   1         1         1       1            1           kubernetes.io/os=linux   8m54s

 

 

(참고) docker desktop 에서 실행한 로컬 kubernetes 클러스터에서의 cni 는 istio cni 적용 불가

 

https://istio.io/latest/docs/setup/additional-setup/cni/#prerequisites

 

Install Istio with the Istio CNI plugin

Install and use Istio with the Istio CNI plugin, allowing operators to deploy services with lower privilege.

istio.io

 

Warning  FailedCreatePodSandBox  4m59s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "5783eafdc481c5c1925334fe2514c938a77be91c6c8f436f8c9c5db225a3c1e3" network for pod "bootcamp-5b67dc74bb-8smzk": networkPlugin cni failed to set up pod "bootcamp-5b67dc74bb-8smzk_bootcamp" network: plugin type="loopback" failed (add): missing network name:, failed to clean up sandbox container "5783eafdc481c5c1925334fe2514c938a77be91c6c8f436f8c9c5db225a3c1e3" network for pod "bootcamp-5b67dc74bb-8smzk": networkPlugin cni failed to teardown pod "bootcamp-5b67dc74bb-8smzk_bootcamp" network: plugin type="loopback" failed (delete): missing network name]

 

 

(참고) istio-cni-node 파드 로그

-> 노드에 istio-proxy 컨테이너 주입이 활성화된 파드 배치 시 자동으로 iptables 세팅

 

2024-05-10T01:52:31.053848Z    info    cni    ============= Start iptables configuration for bootcamp-764dbb76f4-7688l =============
2024-05-10T01:52:31.055613Z    info    cni    Istio iptables environment:
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=
2024-05-10T01:52:31.055706Z    info    cni    Istio iptables variables:
IPTABLES_VERSION=
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020,15021,15090
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=15020
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DUAL_STACK=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
NETWORK_NAMESPACE=/var/run/netns/cni-2ac62102-d38c-f082-047c-828feb67b467
CNI_MODE=true
EXCLUDE_INTERFACES=
2024-05-10T01:52:31.055774Z    info    cni    Running iptables-restore with the following input:
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -p tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2024-05-10T01:52:31.055821Z    info    cni    Running command (without lock by mount and nss): iptables-restore --noflush
2024-05-10T01:52:31.060019Z    info    cni    Running ip6tables-restore with the following input:
2024-05-10T01:52:31.060107Z    info    cni    Running command (without lock by mount and nss): ip6tables-restore --noflush
2024-05-10T01:52:31.062294Z    info    cni    Running command (without nss): iptables-save
2024-05-10T01:52:31.065981Z    info    cni    Command output:
# Generated by iptables-save v1.8.5 on Fri May 10 01:52:31 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 15008 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Fri May 10 01:52:31 2024
2024-05-10T01:52:31.066106Z    info    cni    ============= End iptables configuration for bootcamp-764dbb76f4-7688l =============

 

만약 노드에 배치된 파드가 istio-proxy 컨테이너가 없는 경우 iptables 룰 주입을 하지 않음

 

2024-05-10T01:56:07.895173Z    info    cni    excluded because it does not have istio-proxy container (have [bootcamp])

 

 

샘플 매니페스트 작성

 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bootcamp
spec:
  selector:
    matchLabels:
      app: bootcamp
  replicas: 5
  template:
    metadata:
      labels:
        app: bootcamp
    spec:
      containers:
      - image: gcr.io/google-samples/kubernetes-bootcamp:v1
        imagePullPolicy: Always
        name: bootcamp
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
            ephemeral-storage: 1Gi
          requests:
            cpu: 200m
            memory: 256Mi
            ephemeral-storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: bootcamp
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  selector:
    app: bootcamp
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: bootcamp
spec:
  gateways:
  - istio-system/istio-ingressgateway
  hosts:
  - www.example.com
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: bootcamp.bootcamp.svc.cluster.local
        port:
          number: 80

 

 

호출 테스트

 

kubectl exec -it bootcamp-84bb8b8cd-5r5rm -- curl -v -H 'Host: www.example.com' http://istio-ingressgateway.istio-system.svc.cluster.local/

 

 

 

저작자표시 비영리 변경금지

'kubernetes' 카테고리의 다른 글

ksniff 로 kubernetes 컨테이너 패킷 캡쳐  (1) 2024.04.06
how to install knative and create service  (0) 2023.10.30
[cdk8s] Define k8s yaml file in programming language  (0) 2023.09.08
kube API server and kubernetes python client  (0) 2023.08.13
kops k8s 클러스터 관리도구  (0) 2023.04.10

관련글

티스토리툴바