workload identity federation with k8s cluster
This post covers how to use Kubernetes as an OIDC provider role to use GCP's workload identity federation service by delegating an IAM role without a json key for your GCP service account.
It can be used when an application running on a Kubernetes cluster in a non-GCP environment wants to access GCP's services, or when an application running on a server in a non-GCP environment wants to access GCP's services.
Workload identity federation
https://cloud.google.com/iam/docs/workload-identity-federation
Integrate Cloud Run and workload identity federation
https://cloud.google.com/iam/docs/tutorial-cloud-run-workload-id-federation
Configure workload identity federation with other identity providers
https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers
# workload identity federation
-> A way to grant authenticated users permission to impersonate gcp's mapped sa through a trusted external certification authority.
-> Since it is a means of authenticating outside of gcp without a json key, there is no concern about key management or leakage.
# create workload identity pool
gcloud iam workload-identity-pools create k8s-oidc-pool \
--location="global" \
--description="workload identity pool for k8s" \
--display-name="k8s" \
--project=my-test-project
# create workload identity provider
gcloud iam workload-identity-pools providers create-oidc k8s-oidc-provider \
--location="global" \
--workload-identity-pool="k8s-oidc-pool" \
--issuer-uri="https://example.34.107.196.145.nip.io/" \
--attribute-mapping='google.subject=assertion.sub' \
--project=my-test-project
(Note) Google must be able to publicly call the OIDC provider's metadata URL.
https://cloud.google.com/iam/docs/workload-identity-federation-with-other-providers?hl=ko#prepare
https://cloud.google.com/iam/docs/troubleshooting-workload-identity-federation?hl=ko#error-connecting-issuer
https://cloud.google.com/iam/docs/configuring-workload-identity-federation?hl=ko#oidc
The IdP's OIDC metadata and JWK endpoints are publicly accessible over the Internet.
Google Cloud uses these endpoints to download the IdP's key set and use this key set to validate tokens.
If cloud armor is set on the top of K8s
-> User agent google-thirdparty-credentials, source IP 107.178.192.0/18 must be allowed.
Path where the call occurs during actual authentication
https://example.34.107.196.145.nip.io/.well-known/openid-configuration
https://example.34.107.196.145.nip.io/openid/v1/jwks
(Note) If the k8s cluster, which is the oidc provider, is in a private network, google cannot access the kube api server endpoint.
-> Set the "--issuer-uri" parameter to the default issuer url of the current K8S, and create a provider by including the jwk value of the cluster through the "--jwk-json-path" parameter.
-> Even without publicly accessing OIDC metadata and JWK endpoints, the user can obtain the sts federation token after JWT validation through the JWK entered when creating the provider.
https://cloud.google.com/iam/docs/workload-identity-federation?hl=ko#oidc-credential-security
ISSUER="$(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer)"
kubectl get --raw /openid/v1/jwks > jwk.json
gcloud iam workload-identity-pools providers create-oidc k8s-oidc-provider \
--location="global" \
--workload-identity-pool="k8s-oidc-pool" \
--issuer-uri=${ISSUER} \
--allowed-audiences=${ISSUER} \
--attribute-mapping='google.subject=assertion.sub' \
--jwk-json-path=/k8s/jwk.json \
--project=my-test-project# check list of providers
gcloud iam workload-identity-pools providers list \
--workload-identity-pool="k8s-oidc-pool" \
--location="global" \
--project=my-test-project
# output
---
attributeMapping:
google.subject: assertion.sub
name: projects/123123123123/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider
oidc:
allowedAudiences:
- https://kubernetes.default.svc.cluster.local
issuerUri: https://kubernetes.default.svc.cluster.local/
jwksJson: |
{"keys":[{"use":"sig","kty":"RSA","kid":"0WGsJ1dI1epkGfo1B9xpygpfT0ddyu7P2pldHvjbbIw","alg":"RS256","n":"o6Za..(skip)..vDjXw","e":"AQAB"}]}
state: ACTIVE
# Grant the iam.workloadIdentityUser role so that sa in k8s can impersonate sa in gcp.
create sa
gcloud iam service-accounts create gcs-gsa \
--project=my-test-project
Grant gcs permission to sa
gcloud projects add-iam-policy-binding my-test-project \
--member "serviceAccount:gcs-gsa@my-test-project.iam.gserviceaccount.com" \
--role "roles/storage.objectAdmin"
check project number
gcloud projects describe my-test-project
To allow only specific sa
gcloud iam service-accounts add-iam-policy-binding gcs-gsa@my-test-project.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="principal://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/subject/system:serviceaccount:default:testsa" \
--project=my-test-project
To allow all sa in k8s
gcloud iam service-accounts add-iam-policy-binding gcs-gsa@my-test-project.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="principalSet://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/*" \
--project=my-test-project
# k8s kube api server settings
-> Add option when running kube api server to operate oidc issuer through kube api server of k8s
-> Below are the options when running a cluster in the minikube environment.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
minikube start \
--extra-config=apiserver.service-account-issuer=https://example.34.107.196.145.nip.io \
--extra-config=apiserver.service-account-jwks-uri=https://example.34.107.196.145.nip.io/openid/v1/jwks \
--extra-config=apiserver.api-audiences=https://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider
# Bind a role to allow unauthenticated users to access the oidc metadata url endpoint
https://developer.hashicorp.com/vault/docs/auth/jwt/oidc-providers/kubernetes
https://techblog.cisco.com/blog/kubernetes-oidc
kubectl create clusterrolebinding oidc-reviewer --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated
check
https://example.34.107.196.145.nip.io/.well-known/openid-configuration
https://example.34.107.196.145.nip.io/openid/v1/jwks
output
{"issuer":"https://example.34.107.196.145.nip.io","jwks_uri":"https://example.34.107.196.145.nip.io/openid/v1/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"]}
{"keys":[{"use":"sig","kty":"RSA","kid":"0WGsJ1dI1epkGfo1B9xpygpfT0ddyu7P2pldHvjbbIw","alg":"RS256","n":"o6Za..(중략)..vDjXw","e":"AQAB"}]}
The information displayed is the same as the result of the command below.
kubectl get --raw /.well-known/openid-configuration
kubectl get --raw /openid/v1/jwks
oidc metadata url related k8s cluster role
kubectl get clusterrole system:service-account-issuer-discovery -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: "2023-06-02T23:09:01Z"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:service-account-issuer-discovery
resourceVersion: "100"
uid: 22cba2f0-0ebb-45a8-a7f7-627315170933
rules:
- nonResourceURLs:
- /.well-known/openid-configuration
- /openid/v1/jwks
verbs:
- get
# create k8s sa
kubectl create sa testsa
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: testsa.service-account-token
annotations:
kubernetes.io/service-account.name: testsa
type: kubernetes.io/service-account-token
EOF
# Load oidc jwt token for k8s sa
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#service-account-tokens
AUTH_TOKEN="$(kubectl create token testsa)"
echo $AUTH_TOKEN > token.txt
Below is the payload of the jwt token confirmed on the https://jwt.io/ site.
{
"aud": [
"https://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider"
],
"exp": 1688041864,
"iat": 1688038264,
"iss": "https://example.34.107.196.145.nip.io",
"kubernetes.io": {
"namespace": "default",
"serviceaccount": {
"name": "testsa",
"uid": "b2d963aa-ea13-48d7-aa28-c754a3999789"
}
},
"nbf": 1688038264,
"sub": "system:serviceaccount:default:testsa"
}
(Note) When creating a sa token, you can set the token validity period with the --duration option (if there is no option, the default is 1 hour)
kubectl create token testsa --duration 24h
# Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to authenticate using workload identity federation
-> The GOOGLE_APPLICATION_CREDENTIALS authentication file contains the path to a file containing k8s token information, and authenticates through the token.
Create authentication file
gcloud iam workload-identity-pools create-cred-config \
projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider \
--service-account="gcs-gsa@my-test-project.iam.gserviceaccount.com" \
--credential-source-type="text" \
--output-file=sts-creds.json \
--credential-source-file=/k8s/token.txt
Set as environment variable
export GOOGLE_APPLICATION_CREDENTIALS=/k8s/sts-creds.json
Contents of the generated sts-creds.json file
{
"type": "external_account",
"audience": "//iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": {
"file": "/k8s/token.txt",
"format": {
"type": "text"
}
},
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/gcs-gsa@my-test-project.iam.gserviceaccount.com:generateAccessToken"
}
# Write sample code
vim gcs-list.py
from google.cloud import storage
project_id = "my-test-project"
bucket_name = "my-test-bucket"
storage_client = storage.Client(project=project_id)
blobs = storage_client.list_blobs(bucket_name)
for blob in blobs:
print(blob.name)
Install gcs python client library
pip install google-cloud-storage
Run sample code
python gcs-list.py
# Check log
## workload identity federation authentication log
log query
resource.type="audited_resource" resource.labels.service="sts.googleapis.com" protoPayload.resourceName:"workloadIdentityPools/k8s-oidc-pool"
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalSubject": "system:serviceaccount:default:testsa"
},
"requestMetadata": {
"callerIp": "123.123.123.123",
"callerSuppliedUserAgent": "python-requests/2.28.2,gzip(gfe)",
"requestAttributes": {
"time": "2023-06-29T11:41:12.006985157Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "sts.googleapis.com",
"methodName": "google.identity.sts.v1.SecurityTokenService.ExchangeToken",
"authorizationInfo": [
{
"permission": "sts.identityProviders.checkLogging"
}
],
"resourceName": "projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider",
"request": {
"grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
"subjectTokenType": "urn:ietf:params:oauth:token-type:jwt",
"audience": "//iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/providers/k8s-oidc-provider",
"@type": "type.googleapis.com/google.identity.sts.v1.ExchangeTokenRequest",
"requestedTokenType": "urn:ietf:params:oauth:token-type:access_token"
},
"metadata": {
"mapped_principal": "principal://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/subject/system:serviceaccount:default:testsa",
"@type": "type.googleapis.com/google.identity.sts.v1.AuditData"
}
},
"insertId": "1kw8xp1cb6o",
"resource": {
"type": "audited_resource",
"labels": {
"service": "sts.googleapis.com",
"method": "google.identity.sts.v1.SecurityTokenService.ExchangeToken",
"project_id": "my-test-project"
}
},
"timestamp": "2023-06-29T11:41:11.999378987Z",
"severity": "INFO",
"logName": "projects/my-test-project/logs/cloudaudit.googleapis.com%2Fdata_access",
"receiveTimestamp": "2023-06-29T11:41:12.680063691Z"
}
## gcs bucket object list call log
-> Check if the call was made to the sa account (gcs-gsa@my-test-project.iam.gserviceaccount.com) set by impersonate.
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {},
"authenticationInfo": {
"principalEmail": "gcs-gsa@my-test-project.iam.gserviceaccount.com",
"serviceAccountDelegationInfo": [
{
"principalSubject": "principal://iam.googleapis.com/projects/123412341234/locations/global/workloadIdentityPools/k8s-oidc-pool/subject/system:serviceaccount:default:testsa"
}
]
},
"requestMetadata": {
"callerIp": "123.123.123.123",
"callerSuppliedUserAgent": "gcloud-python/2.9.0 gl-python/3.11.2 grpc/1.51.3 gax/2.11.0 gccl/2.9.0,gzip(gfe)",
"requestAttributes": {
"time": "2023-06-29T11:41:13.350663952Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "storage.googleapis.com",
"methodName": "storage.objects.list",
"authorizationInfo": [
{
"resource": "projects/_/buckets/my-test-bucket",
"permission": "storage.objects.list",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/_/buckets/my-test-bucket",
"resourceLocation": {
"currentLocations": [
"asia-northeast3"
]
}
},
"insertId": "qn7rt1e5m940",
"resource": {
"type": "gcs_bucket",
"labels": {
"bucket_name": "my-test-bucket",
"location": "asia-northeast3",
"project_id": "my-test-project"
}
},
"timestamp": "2023-06-29T11:41:13.343141622Z",
"severity": "INFO",
"logName": "projects/my-test-project/logs/cloudaudit.googleapis.com%2Fdata_access",
"receiveTimestamp": "2023-06-29T11:41:14.440941103Z"
}
# Issuance of temporary tokens through permanent tokens
-> Since the token created through the "kubectl create token SA_NAME" command is a temporary token, you can include a permanent token in the application and request a temporary token from the kube api server when needed.
-> In the case of persistent tokens, the token specifications are different and cannot be used for workload identity federation.
-> sa(token-creator) for permanent tokens and sa(testsa) to create temporary tokens can be used separately (define sa to allow token creation in the rules.[0].resourceNames path of Role)
## Create/grant sa and role for token creation
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: token-creator
namespace: default
rules:
- apiGroups:
- ""
resources:
- serviceaccounts/token
resourceNames:
- testsa
verbs:
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: token-creator
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: token-creator
subjects:
- kind: ServiceAccount
name: token-creator
namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: token-creator
namespace: default
---
apiVersion: v1
kind: Secret
metadata:
name: token-creator.service-account-token
namespace: default
annotations:
kubernetes.io/service-account.name: token-creator
type: kubernetes.io/service-account-token
## Define token creation request
https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-request-v1/
vim token_request.json
{
"kind": "TokenRequest",
"apiVersion": "authentication.k8s.io/v1",
"metadata": {
"creationTimestamp": null
},
"spec": {
"audiences": null,
"expirationSeconds": null,
"boundObjectRef": null
},
"status": {
"token": "",
"expirationTimestamp": null
}
}
## Token creation request
APISERVER="https://example.34.107.196.145.nip.io"
SERVICEACCOUNT="testsa"
NAMESPACE="default"
TOKEN="$(kubectl get secret token-creator.service-account-token -o=jsonpath='{.data.token}' | base64 --decode)"
curl -vk -H "content-type: application/json" -H "Authorization: Bearer ${TOKEN}" \
-d @token_request.json -X POST "${APISERVER}/api/v1/namespaces/${NAMESPACE}/serviceaccounts/${SERVICEACCOUNT}/token"
Create a token and save it as a token.txt file.
curl -sk -H "content-type: application/json" -H "Authorization: Bearer ${TOKEN}" \
-d @token_request.json -X POST "${APISERVER}/api/v1/namespaces/${NAMESPACE}/serviceaccounts/${SERVICEACCOUNT}/token" | jq -r .status.token > token.txt