Kubernetes Authentication, Authorization & Admission Control

Posted on Tuesday February 12, 2019

Introduction

This sample chapter extracted from the book, Kubernetes for DevOps .

Get this book on Just $9 or Ask Author for Discount

Multiple steps involved to by the Kubernetes API server, before granting/revoking access for the managed kubernetes resources. it's start with

TLS Security

Kubernetes cluster, the API serves on port 443. The API server presents a certificate. This certificate is often self-signed, so $USER/.kube/config on the user's machine typically contains the root certificate for the API server's certificate, which when specified is used in place of the system default root certificate

As shown in the below example the API server is running on the port 6443

root@kube-master:# kubectl cluster-info
Kubernetes master is running at https://172.16.2.13:6443
KubeDNS is running at https://172.16.2.13:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

by default the kubernetes admin X.509 CA certificate, Private Key and Certificate revocation list is present in the in the KUBE_HOME=/etc/kubernetes/ directory

root@kube-master:# ls -l /etc/kubernetes/pki/ca.*
-rw-r--r-- 1 root root 1025 Jan 23 11:14 /etc/kubernetes/pki/ca.crt
-rw------- 1 root root 1679 Jan 23 11:14 /etc/kubernetes/pki/ca.key
-rw-r--r-- 1 root root   17 Feb  7 13:15 /etc/kubernetes/pki/ca.srl

Authentication

Once you have located the API server, next step is to perform the Authentication, there are two kind of resources which access the kubernetes API server

Service Account
Normal User

User and serviceAccount can belongs to one or more groups, groups are designed to grant permission to several users at once, there are reserved built-in group in the kube-system namespace

The system:unauthenticated group is used for requests where none of the authentication plugins could authenticate the client.
The system:authenticated group is automatically assigned to a user who was authenticated successfully.
The system:serviceaccounts group encompasses all ServiceAccounts in the system.
The system:serviceaccounts:<namespace> includes all ServiceAccounts in a specific namespace.

ServiceAccounts

ServiceAccounts are kubernetes managed resources and are scope to individual namespaces. for example let's GET all the sa available in the kube-system.

root@kube-master:# kubectl get sa -n kube-system
NAME                                 SECRETS   AGE
attachdetach-controller              1         20d
bootstrap-signer                     1         20d
certificate-controller               1         20d
clusterrole-aggregation-controller   1         20d
coredns                              1         20d
cronjob-controller                   1         20d
daemon-set-controller                1         20d
default                              1         20d
..............
..............

Creating ServiceAccounts

Let's create a sa under the namespace dev

root@kube-master:# kubectl create sa test -n dev
serviceaccount/test created

List out the sa under the namespace dev

root@kube-master:# kubectl get sa -n dev 
NAME      SECRETS   AGE
default   1         4d
test      1         17m

you might notice a default ServiceAccounts is already exists. Each namespaces have a default servicesccount

Inspect ServiceAccount

kubectl describe the sa test under the namespace dev

root@kube-master:# kubectl describe sa test -n dev 
Name:                test
Namespace:           dev
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   test-token-hvbtq
Tokens:              test-token-hvbtq
Events:              <none>

Mountable secrets: Pods using this ServiceAccount can only mount these Secrets i.e test-token-hvbtq
Tokens: : JWT Authentication token i.e test-token-hvbtq
Image pull secrets : Credentials for pulling container images from a private image repository not set.

3 . Debugging Secrets

View the secrets of the test-token-hvbtq

root@kube-master:# kubectl describe secrets test-token-hvbtq -n dev 
Name:         test-token-hvbtq
Namespace:    dev
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=test
              kubernetes.io/service-account.uid=8d0fcb37-2e8a-11e9-8c68-fa163e589bc0

Type:  kubernetes.io/service-account-token

Data
====
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoidGVzdC10b2tlbi1odmJ0cSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0ZXN0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOGQwZmNiMzctMmU4YS0xMWU5LThjNjgtZmExNjNlNTg5YmMwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRldjp0ZXN0In0.caDSrXwkrIGXboIQ5Ag9G-2lp0Ltli5bb8V9O8a0rbVvDJChukQbAXko1pVqKA7eNlT-qOBRl1K6CKAZHiYvDEfhhCv68GF6YncDlY-1eaUdNNO-CT6d2DHXEWb6gSGK-3P1dxRqqJjVE4FBsMfLDfnky203DMuToz6BsnxgUJY8aQOl-3z8AFJJOV2-c4i5da2wKQfb2meGVQkTI_bDSdp-aq0PP9si5UXKKsCzeOBbpuEO1Xk66ggf9cIoREbJujy9zJN_QMcdYS3o1lOuIThAIRpkDid4y1VH_ulKMnt7JBrTyOxeFK1UobC2kZbP2LFCzoNIBSXvypEVlrR-hw
ca.crt:     1025 bytes
namespace:  3 bytes

The token value is JSON Web Token, the decoded format.

//Header
{
  "alg": "RS256",
  "kid": ""
}
//Payload
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "dev",
  "kubernetes.io/serviceaccount/secret.name": "test-token-hvbtq",
  "kubernetes.io/serviceaccount/service-account.name": "test",
  "kubernetes.io/serviceaccount/service-account.uid": "8d0fcb37-2e8a-11e9-8c68-fa163e589bc0",
  "sub": "system:serviceaccount:dev:test"
}

Assigning ServiceAccount to Pod

The YAML file for assigning serviceAccountName to test and launching a alpine pod with curl installed

root@kube-master:# cat alpine-curl.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: alpine
spec:
  serviceAccountName: test
  containers:
  - image: byrnedo/alpine-curl
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
  restartPolicy: Always

Creating POD from the given YAML definition

root@kube-master:# kubectl create -f alpine-curl.yaml -n dev 
pod/alpine created

Verifying pods are in the RUNNING state

root@kube-master:# kubectl get pods -n dev
NAME      READY     STATUS    RESTARTS   AGE
alpine    1/1       Running   0          48s

Exec to the alpine pod

root@kube-master:/home/ansible# kubectl exec alpine  -it sh -n dev 
/ #

Locate the Mountable secretes which is present in the /var/run/secrets/kubernetes.io/serviceaccount/token

# cat /var/run/secrets/kubernetes.io/serviceaccount/token 
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoidGVzdC10b2tlbi1odmJ0cSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0ZXN0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOGQwZmNiMzctMmU4YS0xMWU5LThjNjgtZmExNjNlNTg5YmMwIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRldjp0ZXN0In0.caDSrXwkrIGXboIQ5Ag9G-2lp0Ltli5bb8V9O8a0rbVvDJChukQbAXko1pVqKA7eNlT-qOBRl1K6CKAZHiYvDEfhhCv68GF6YncDlY-1eaUdNNO-CT6d2DHXEWb6gSGK-3P1dxRqqJjVE4FBsMfLDfnky203DMuToz6BsnxgUJY8aQOl-3z8AFJJOV2-c4i5da2wKQfb2meGVQkTI_bDSdp-aq0PP9si5UXKKsCzeOBbpuEO1Xk66ggf9cIoREbJujy9zJN_QMcdYS3o1lOuIThAIRpkDid4y1VH_ulKMnt7JBrTyOxeFK1UobC2kZbP2LFCzoNIBSXvypEVlrR-hw/

Query API server with the --header "Authorization: Bearer $TOKEN"

# export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# curl https://172.16.2.13:6443/api --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "172.16.2.13:6443"
    }
  ]
}/

Alternatively to locate Mountable secrete in the pods by describing the pods itself

kubectl describe pods alpine -n dev 
.....

Containers:
  busybox:
    Container ID:  docker://f330b20da3b95211774f53167e1f2e9149cbe144627f6f24608948cce3641767
    Image:         byrnedo/alpine-curl
    Image ID:      docker-pullable://byrnedo/alpine-curl@sha256:e8cf497b3005c2f66c8411f814f3818ecd683dfea45267ebfb4918088a26a18c
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      3600
    State:          Running
      Started:      Tue, 12 Feb 2019 12:00:30 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from test-token-hvbtq (ro)

Deleting ServiceAccount

Deleting the sa test under namespace dev

root@kube-master:# kubectl delete sa test -n dev 
serviceaccount "test" deleted

Any Authentication performed using the old token will be Unauthorized

# export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# curl https://172.16.2.13:6443/api --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

ServiceAccount Practical Use-Case

Some of the practical use case of setting sa

Jenkins: for Managing CI/CD pipeline in kubernetes cluster

kubectl -n kube-system create sa jenkins
kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=<namespace>:jenkins

Helm: Kubernetes Repository Manager

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

User Accounts

The Next form of Authentication in kubernetes API is with normal user, this is usually done with TLS authentication and typical steps involved

User generate a strong RSA or EC keys
User create CSR and bind their Common Name attribute for example [email protected]
User submit this CSR to kubernetes admin.
Kuberntetes admin verify the content of CSR before Issuing the certificate.
Kubernetes admin will sign the CSR with rootCA and rootCA private key and generate the x.509 Certificate. the kubernetes PKI information is usually located in the /etc/kubernetes/pki/

We will bet into this Authentication using normal account in detail in the upcoming topic of Setting up Role-Based Access Control

Authorization

After the request is authenticated as coming from a specific user or sa , the request must be authorized.

For example exec to the alpine kubectl exec alpine -it sh -n dev and list all pods in the dev name space the request will be forbidden

# export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# curl https://172.16.2.13:6443/api/v1/namespaces/dev/pods --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:dev:test\" cannot list pods in the namespace \"dev\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

In order to grant Authorization proper policy or RBAC control needs to be in placed, for an instance in order to give pod permission to query all pods in the given namespace, a role and corresponding rolebinding needs to be defined.

The below YAML file has setup necessary authorization for role and rolebinding under the namespace dev

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-manager
  namespace: dev
rules:
- apiGroups: ["", "batch", "extensions", "apps"]
  resources: ["pods"]
  verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-binding
  namespace: dev
subjects:
- kind: ServiceAccount
  name: test
  namespace: dev
roleRef:
  kind: Role
  name: test-manager
  apiGroup: rbac.authorization.k8s.io

root@kube-master:# kubectl delete  -f rbac.yaml
role.rbac.authorization.k8s.io "test-manager" deleted
rolebinding.rbac.authorization.k8s.io "test-binding" deleted

The role is set to query the resource pods with all known HTTP verbs

# curl https://172.16.2.13:6443/api/v1/namespaces/dev/pods --header "Authorization: Bearer $TOKEN" --insecure
"kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/dev/pods",
    "resourceVersion": "2302485"
  },
  "items": [
    {
      "metadata": {
        "name": "alpine",
        "namespace": "dev",
        "selfLink": "/api/v1/namespaces/dev/pods/alpine",
        "uid": "0abb388b-2e97-11e9-8c68-fa163e589bc0",
        "resourceVersion": "2300673",
        "creationTimestamp": "2019-02-12T07:23:06Z"

....

We will cover RBAC in much detail in upcoming topic let's stick with next important concept Admission Control

Admission Control

In Kubernetes, Admission Controllers enforce semantic validation of objects during create, update, and delete operations.

First check if the admission registration API is enabled in your cluster by running:

root@kube-master:# kubectl api-versions | grep admission
admissionregistration.k8s.io/v1beta1

Some example of semantic validation can be done through the Admission Controllers

Validate that image tags are are having certain pre-configured prefix or postfix.
Reject an image if it is being pulled from dockerhub directly.
Reject an image that has high or critical CVEs that have a fix available, but allow high-severity if no fix is available yet
Never reject images from a specific registry/repository
Others

There are two types of Webhook Admission controllers in Kubernetes 1.9.

ValidatingAdmissionWebhook
MutatingAdmissionWebhook

These webhooks is that they can be dynamically configured after the start of the api-server, this enables anyone with the correct Role-based access control (RBAC) to extend the concept of Admission Control.

Next Reading RBAC

Thanku for reading !!! Give a Share for Support

Asking for donation sound bad to me, so i'm raising fund from by offering all my Nine book for just $9