k8s cert-manager

k8s cert-manager

https://cert-manager.io/docs/tutorials/getting-started-with-cert-manager-on-google-kubernetes-engine-using-lets-encrypt-for-ingress-ssl/

# 도메인 레코드 세팅

example.com 도메인의 레코드를 gcp lb 의 퍼블릭 아이피(34.111.226.147)로 세팅

# ingress 생성

vim ingress-http.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

kubectl apply -f ingress-http.yaml

kubectl describe ingress istio-ingressgateway -n istio-system

접속 확인

export DOMAIN_NAME=example.com
curl http://$DOMAIN_NAME

# cert-manager 배포

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml

kubectl get all -n cert-manager

crd 상세 내용 확인

kubectl explain Certificate
kubectl explain CertificateRequest
kubectl explain Issuer

# letsencrypt staging issuer 생성

vim issuer-lets-encrypt-staging.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: istio-system
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: my-email@gmail.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          name: istio-ingressgateway

kubectl apply -f issuer-lets-encrypt-staging.yaml

확인

kubectl describe issuers.cert-manager.io letsencrypt-staging

# ingress 에 ssl 적용

-> 자동으로 gcp lb 에 /.well-known/ 경로에 대한 경로 규칙 생성되며, 인증서 발급을 위한 메타태그 인증 진행됨

-> 인증서 발급 완료 후 /.well-known/ 경로에 대한 경로 규칙은 자동으로 제거됨

-> 인증서 발급 완료 후 k8s secret 리소스(example-com)에 인증서 정보가 자동으로 업데이트되며, gcp lb 에도 "자체 관리형" 인증서 개체가 생성되어 lb 에 적용됨

vim secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: example-com
  namespace: istio-system
type: kubernetes.io/tls
stringData:
  tls.key: ""
  tls.crt: ""

kubectl apply -f secret.yaml

vim ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  annotations:
    cert-manager.io/issuer: letsencrypt-staging
    #cert-manager.io/issuer: letsencrypt-production
spec:
  tls:
    - secretName: example-com
      hosts:
        - example.com
        - www.example.com
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

kubectl apply -f ingress.yaml

접속 테스트

-> staging issuer 는 인증서 오류 발생하는 것이 정상(production issuer 생성하여 최종 적용)

curl -v --insecure https://$DOMAIN_NAME

# letsencrypt production issuer 생성

vim issuer-lets-encrypt-production.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-production
  namespace: istio-system
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my-email@gmail.com
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          name: istio-ingressgateway

kubectl apply -f issuer-lets-encrypt-production.yaml

기존 ingress 에 annotate 수정

-> 위의 ingress.yaml 파일 수정하여 "cert-manager.io/issuer: letsencrypt-production" 주석 해제해도 됨

kubectl annotate ingress istio-ingressgateway cert-manager.io/issuer=letsencrypt-production --overwrite

확인

curl -v https://$DOMAIN_NAME

인증서 개체 확인

kubectl get certificate example-com -n istio-system

kubectl describe certificate example-com -n istio-system

 

# dns 인증을 통해 와일드카드 인증서 발급

-> 자동으로 clouddns에 인증 레코드 생성하여 인증 진행

-> 레코드 예시(아래)

호스트 - _acme-challenge.example.com.
레코드 값 - rc71w6x4vNp7VsEJckb83WLBnoDL0tHw7XzyqC1hjtA

-> 인증 완료 후 레코드는 자동으로 삭제됨

https://cert-manager.io/docs/configuration/acme/dns01/google/

## cert-manager 파드의 파라미터 추가

-> cert-manager 파드에 파리미터를 추가하지 않으면 워크로드 아이덴티티를 통해 부여한 인증 정보를 인식하지 못함

https://github.com/cert-manager/cert-manager/issues/3009#issuecomment-770345196

kubectl edit deployment.apps/cert-manager -n cert-manager

    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
-->
    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        - --issuer-ambient-credentials=true
        - --cluster-issuer-ambient-credentials=true

## 워크로드 아이덴티티 설정

gcloud iam service-accounts create dns01-solver --display-name "dns01-solver" --project my-project-id

gcloud projects add-iam-policy-binding my-project-id \
   --member serviceAccount:dns01-solver@my-project-id.iam.gserviceaccount.com \
   --role roles/dns.admin

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:my-project-id.svc.id.goog[cert-manager/cert-manager]" \
    dns01-solver@my-project-id.iam.gserviceaccount.com

kubectl annotate serviceaccount --namespace=cert-manager cert-manager \
    "iam.gke.io/gcp-service-account=dns01-solver@my-project-id.iam.gserviceaccount.com"

## dns 인증을 사용하는 issuer 생성

vim issuer-lets-encrypt-production-dns.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-production-dns
  namespace: istio-system
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: my-email@gmail.com
    privateKeySecretRef:
      name: letsencrypt-production-dns
    solvers:
    - dns01:
        cloudDNS:
          project: my-project-id

kubectl apply -f issuer-lets-encrypt-production-dns.yaml

## 인증서 secret 생성

vim secret-star.yaml

apiVersion: v1
kind: Secret
metadata:
  name: star-example-com
  namespace: istio-system
type: kubernetes.io/tls
stringData:
  tls.key: ""
  tls.crt: ""

## 인증서 적용

vim ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  annotations:
    cert-manager.io/issuer: letsencrypt-production-dns
spec:
  tls:
    - secretName: star-example-com
      hosts:
        - example.com
        - '*.example.com'
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

## ingress 로 internal lb 생성하여 인증서 적용

https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balance-ingress?hl=ko#deploy-ingress

vim ingress-int.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingressgateway-int
  namespace: istio-system
  annotations:
    kubernetes.io/ingress.class: "gce-internal"
    kubernetes.io/ingress.allow-http: "false"
    cert-manager.io/issuer: letsencrypt-production-dns
spec:
  tls:
    - secretName: star-example-com
      hosts:
        - example.com
        - '*.example.com'
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

# 인증서를 수동으로 생성

-> ingress 생성 시 tls 설정을 통해 certificate 리소스가 자동으로 생성됨

-> ingress 없이 certificate 리소스만 따로 만들어주고 싶은 경우 아래와 같이 설정

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: star-example-com
  namespace: istio-system
spec:
  secretName: star-example-com
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: letsencrypt-production-dns
  dnsNames:
  - example.com
  - '*.example.com'

kubectl apply -f cert.yaml

kubectl get certificate -n istio-system