Goglides Dev đŸŒ±

Balkrishna Pandey
Balkrishna Pandey

Posted on • Originally published at goglides.io on

Exploring kubectl HTTP request verbose option and custom kube-config based on kubernetes RBAC

#kubernetes #kubectl

We can use -v or — v flags followed by an integer to check how kubectl is sending HTTP request. For integer value you can use from 0–8, but only 6–8 value is used for HTTP request.

 (adsbygoogle = window.adsbygoogle || []).push({});
Enter fullscreen mode Exit fullscreen mode

Verbosity level 6

This level will display all requested resources with HTTP response code and time take to complete each request. For example:


kubectl get pods -v=6
I0927 16:59:16.117531 85026 loader.go:357] Config loaded from file /Users/username/.kube/yala
I0927 16:59:17.090324 85026 round_trippers.go:436] GET https://api-k8s/api 200 OK in 971 milliseconds
I0927 16:59:17.262158 85026 round_trippers.go:436] GET https://api-k8s/apis 200 OK in 170 milliseconds
No resources found.
Enter fullscreen mode Exit fullscreen mode

Verbosity level 7

This level will display HTTP request with headers. For example:


kubectl get pods -v=7
ggg:ghost bkpandey$ 18kubectl get pods -v=7
I0927 17:07:33.435101 85152 loader.go:357] Config loaded from file /Users/bkpandey/.kube/yala
I0927 17:07:33.437478 85152 round_trippers.go:414] GET https://api-k8s/api
I0927 17:07:33.437498 85152 round_trippers.go:421] Request Headers:
I0927 17:07:33.437506 85152 round_trippers.go:424] Accept: application/json, */*
I0927 17:07:33.437512 85152 round_trippers.go:424] User-Agent: 18kubectl/v1.8.11 (darwin/amd64) kubernetes/1df6a83
I0927 17:07:33.437518 85152 round_trippers.go:424] Authorization: Basic YWRtaW46cGVtbjRVd0d0b0loNGY5TzRocE9sY0tiVWZGaF=
I0927 17:07:34.119013 85152 round_trippers.go:439] Response Status: 200 OK in 681 milliseconds
Enter fullscreen mode Exit fullscreen mode

Verbosity level 8

This level will display HTTP request with contents. For example:


kubectl get pods -v=8
ggg:ghost bkpandey$ 18kubectl get pods -v=8
I0927 17:09:52.315477 85237 loader.go:357] Config loaded from file /Users/bkpandey/.kube/yala
I0927 17:09:52.317166 85237 round_trippers.go:414] GET https://api-k8s/api
I0927 17:09:52.317177 85237 round_trippers.go:421] Request Headers:
I0927 17:09:52.317181 85237 round_trippers.go:424] Authorization: Basic YWRtaW46cGVtbjRVd0d0b0loNGY5TzRocE9sY0tiVWZGaFg=
I0927 17:09:52.317185 85237 round_trippers.go:424] Accept: application/json, */*
I0927 17:09:52.317189 85237 round_trippers.go:424] User-Agent: 18kubectl/v1.8.11 (darwin/amd64) kubernetes/1df6a83
I0927 17:09:53.194604 85237 round_trippers.go:439] Response Status: 200 OK in 877 milliseconds
I0927 17:09:53.194626 85237 round_trippers.go:442] Response Headers:
I0927 17:09:53.194646 85237 round_trippers.go:445] Content-Type: application/json
I0927 17:09:53.194659 85237 round_trippers.go:445] Content-Length: 132
I0927 17:09:53.194662 85237 round_trippers.go:445] Date: Thu, 27 Sep 2018 23:09:53 GMT
I0927 17:09:53.195738 85237 request.go:836] Response Body: {“kind”:”APIVersions”,”versions”:[“v1”],”serverAddressByClientCIDRs”:[{“clientCIDR”:”0.0.0.0/0",”serverAddress”:”54.21.21.10"}]}
I0927 17:09:53.196596 85237 round_trippers.go:414] GET https://api-k8s/apis
I0927 17:09:53.196608 85237 round_trippers.go:421] Request Headers:
I0927 17:09:53.196613 85237 round_trippers.go:424] Accept: application/json, */*
I0927 17:09:53.196617 85237 round_trippers.go:424] User-Agent: 18kubectl/v1.8.11 (darwin/amd64) kubernetes/1df6a83
I0927 17:09:53.196621 85237 round_trippers.go:424] Authorization: Basic YWRtaW46cGVtbjRVd0d0b0loNGY5TzRocE9sY0tiVWZGaFg=
I0927 17:09:53.340432 85237 round_trippers.go:439] Response Status: 200 OK in 143 milliseconds
I0927 17:09:53.340454 85237 round_trippers.go:442] Response Headers:
I0927 17:09:53.340459 85237 round_trippers.go:445] Content-Type: application/json
I0927 17:09:53.340463 85237 round_trippers.go:445] Content-Length: 3268
I0927 17:09:53.340466 85237 round_trippers.go:445] Date: Thu, 27 Sep 2018 23:09:53 GMT
I0927 17:09:53.341608 85237 request.go:836] Response Body: {“kind”:”APIGroupList”,”apiVersion”:”v1",”groups”:[{“name”:”apiregistration.k8s.io”,”versions”:[{“groupVersion”:”apiregistration.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”apiregistration.k8s.io/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null},{“name”:”extensions”,”versions”:[{“groupVersion”:”extensions/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”extensions/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null},{“name”:”apps”,”versions”:[{“groupVersion”:”apps/v1beta1",”version”:”v1beta1"},{“groupVersion”:”apps/v1beta2",”version”:”v1beta2"}],”preferredVersion”:{“groupVersion”:”apps/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null},{“name”:”authentication.k8s.io”,”versions”:[{“groupVersion”:”authentication.k8s.io/v1",”version”:”v1"},{“groupVersion”:”authentication.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”authentication.k8s.io/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”authorization.k8s.io”,”versions”:[{“groupVersion”:”authorization.k8s.io/v1",”version”:”v1"},{“groupVersion”:”authorization.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”authorization.k8s.io/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”autoscaling”,”versions”:[{“groupVersion”:”autoscaling/v1",”version”:”v1"},{“groupVersion”:”autoscaling/v2beta1",”version”:”v2beta1"}],”preferredVersion”:{“groupVersion”:”autoscaling/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”batch”,”versions”:[{“groupVersion”:”batch/v1",”version”:”v1"},{“groupVersion”:”batch/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”batch/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”certificates.k8s.io”,”versions”:[{“groupVersion”:”certificates.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”certificates.k8s.io/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null},{“name”:”networking.k8s.io”,”versions”:[{“groupVersion”:”networking.k8s.io/v1",”version”:”v1"}],”preferredVersion”:{“groupVersion”:”networking.k8s.io/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”policy”,”versions”:[{“groupVersion”:”policy/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”policy/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null},{“name”:”rbac.authorization.k8s.io”,”versions”:[{“groupVersion”:”rbac.authorization.k8s.io/v1",”version”:”v1"},{“groupVersion”:”rbac.authorization.k8s.io/v1beta1",”version”:”v1beta1"},{“groupVersion”:”rbac.authorization.k8s.io/v1alpha1",”version”:”v1alpha1"}],”preferredVersion”:{“groupVersion”:”rbac.authorization.k8s.io/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”storage.k8s.io”,”versions”:[{“groupVersion”:”storage.k8s.io/v1",”version”:”v1"},{“groupVersion”:”storage.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”storage.k8s.io/v1",”version”:”v1"},”serverAddressByClientCIDRs”:null},{“name”:”apiextensions.k8s.io”,”versions”:[{“groupVersion”:”apiextensions.k8s.io/v1beta1",”version”:”v1beta1"}],”preferredVersion”:{“groupVersion”:”apiextensions.k8s.io/v1beta1",”version”:”v1beta1"},”serverAddressByClientCIDRs”:null}]}
Enter fullscreen mode Exit fullscreen mode 
 (adsbygoogle = window.adsbygoogle || []).push({});
Enter fullscreen mode Exit fullscreen mode

Figuring out RBAC rule

We can actually use verbosity level to figure out all the RBAC rules kubectl would required to access particular resources. For example, lets say you are running following kubectl command:


kubectl get node
Enter fullscreen mode Exit fullscreen mode

and to figure out which RBAC rules are needed for this, You can run:


kubectl -v=8 get node 2>&1 | grep ‘GET\|POST\|DELETE\|PATCH\|PUT’
Enter fullscreen mode Exit fullscreen mode

This will list all the API requests the code is making.


I0928 10:12:29.122226 86839 round_trippers.go:383] GET https://api-k8s/api?timeout=32s
I0928 10:12:31.291086 86839 round_trippers.go:383] GET https://api-k8s/apis?timeout=32s
I0928 10:12:31.444976 86839 round_trippers.go:383] GET https://api-k8s/apis/rbac.authorization.k8s.io/v1alpha1?timeout=32s
I0928 10:12:31.445057 86839 round_trippers.go:383] GET https://api-k8s/apis/policy/v1beta1?timeout=32s
I0928 10:12:31.445073 86839 round_trippers.go:383] GET https://api-k8s/apis/authentication.k8s.io/v1?timeout=32s
I0928 10:12:31.445194 86839 round_trippers.go:383] GET https://api-k8s/apis/apps/v1beta2?timeout=32s
I0928 10:12:31.445251 86839 round_trippers.go:383] GET https://api-k8s/apis/rbac.authorization.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.445304 86839 round_trippers.go:383] GET https://api-k8s/apis/storage.k8s.io/v1?timeout=32s
I0928 10:12:31.445345 86839 round_trippers.go:383] GET https://api-k8s/apis/authorization.k8s.io/v1?timeout=32s
I0928 10:12:31.445535 86839 round_trippers.go:383] GET https://api-k8s/apis/batch/v1beta1?timeout=32s
I0928 10:12:31.444975 86839 round_trippers.go:383] GET https://api-k8s/apis/apiextensions.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.445086 86839 round_trippers.go:383] GET https://api-k8s/apis/storage.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.446762 86839 round_trippers.go:383] GET https://api-k8s/apis/autoscaling/v1?timeout=32s
I0928 10:12:31.446965 86839 round_trippers.go:383] GET https://api-k8s/api/v1?timeout=32s
I0928 10:12:31.447051 86839 round_trippers.go:383] GET https://api-k8s/apis/certificates.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.447386 86839 round_trippers.go:383] GET https://api-k8s/apis/rbac.authorization.k8s.io/v1?timeout=32s
I0928 10:12:31.444976 86839 round_trippers.go:383] GET https://api-k8s/apis/autoscaling/v2beta1?timeout=32s
I0928 10:12:31.447785 86839 round_trippers.go:383] GET https://api-k8s/apis/batch/v1?timeout=32s
I0928 10:12:31.448087 86839 round_trippers.go:383] GET https://api-k8s/apis/authorization.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.448458 86839 round_trippers.go:383] GET https://api-k8s/apis/extensions/v1beta1?timeout=32s
I0928 10:12:31.448710 86839 round_trippers.go:383] GET https://api-k8s/apis/networking.k8s.io/v1?timeout=32s
I0928 10:12:31.449004 86839 round_trippers.go:383] GET https://api-k8s/apis/apps/v1beta1?timeout=32s
I0928 10:12:31.445173 86839 round_trippers.go:383] GET https://api-k8s/apis/authentication.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.449757 86839 round_trippers.go:383] GET https://api-k8s/apis/apiregistration.k8s.io/v1beta1?timeout=32s
I0928 10:12:31.819105 86839 round_trippers.go:383] GET https://api-k8s/api/v1/nodes?limit=500
Enter fullscreen mode Exit fullscreen mode

If you noticed kubectl caches API auto-discovery in ~/.kube/cache folder, you may need to delete this folder to get the full API request list. You can run rm -rf ~/.kube/cache to clear this folder.

 (adsbygoogle = window.adsbygoogle || []).push({});
Enter fullscreen mode Exit fullscreen mode

Based on above output its trying to hit lots of api endpoints. Ideally we should able to list individual apiGroups in our Cluster Role policy. But somehow I keep getting permission denied issue when its trying to hit /v1/nodes. Because of that in following rule I am saying apiGroups as “*” and restricting resources to only “nodes”.


kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: kubectl-get-nodes
rules:
- apiGroups:
  - '*'
  resources:
  - 'nodes'
  verbs:
  - 'get'
  - 'list'
Enter fullscreen mode Exit fullscreen mode

Now we can use this cluster role in one of the service account and create kubeconfig file based on service account.

 (adsbygoogle = window.adsbygoogle || []).push({});
Enter fullscreen mode Exit fullscreen mode

To create Service account


apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: kubectl
name: kubectl-get-nodes
namespace: default
Enter fullscreen mode Exit fullscreen mode

To create Cluster Role Binding


apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubectl-get-nodes-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl-get-nodes
subjects:
- kind: ServiceAccount
  name: kubectl-get-nodes
  namespace: default
Enter fullscreen mode Exit fullscreen mode

Finally we can execute following bash script to generate kubeconfig file:


#!/usr/bin/env bash
# Service account created using above manifest file
serviceaccount=kubectl-get-nodes
namespace=default
# Get related Secrets for this Service Account
secret=$(kubectl get sa $serviceaccount -n $namespace -o json | jq -r .secrets[].name)
# Get ca.crt from secret (using OSX base64 with -D flag for decode)
kubectl get secret $secret -n $namespace -o json | jq -r ‘.data[“ca.crt”]’ | base64 -D > ca.crt
# Get service account token from secret
user_token=$(kubectl get secret $secret -n $namespace -o json | jq -r ‘.data[“token”]’ | base64 -D)
# Get information from your kubectl config, this will use current context. Your kubeconfig file may have multiple context.
context=`kubectl config current-context`
# get cluster name of context
name=`kubectl config get-contexts $context | awk ‘{print $3}’ | tail -n 1`
# get endpoint of current context
endpoint=`kubectl config view -o jsonpath=”{.clusters[?(@.name == \”$name\”)].cluster.server}”`
# Set cluster (run in directory where ca.crt is stored)
kubectl config set-cluster $serviceaccount-$context \
— embed-certs=true \
— server=$endpoint \
— certificate-authority=./ca.crt
# Set user credentials
kubectl config set-credentials $serviceaccount-$context — token=$user_token
# Define the combination of user with the cluster
kubectl config set-context $serviceaccount-$context \
— cluster=$serviceaccount-$context \
— user=$serviceaccount-$context \
— namespace=$namespace
Enter fullscreen mode Exit fullscreen mode

When you list cluster-context you will see new entry in your config file. I am seeing following


kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
api-k8s api-k8s api-k8s
* kubectl-get-nodes-api-k8s kubectl-get-nodes-api-k8s kubectl-get-nodes-api-k8s default
Enter fullscreen mode Exit fullscreen mode

Validation

Switch the context using kubectl use context kubectl-get-nodes-api-k8s and try accessing nodes resources using kubectl get nodes. You can access node resource.


kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172–31–41–106.us-west-2.compute.internal Ready node 74d v1.15.11
ip-172–31–47–109.us-west-2.compute.internal Ready node 75d v1.15.11
ip-172–31–47–20.us-west-2.compute.internal Ready master 75d v1.15.11
Enter fullscreen mode Exit fullscreen mode

Now try accessing other resources, you will see access denied.


kubectl get pods
Error from server (Forbidden): pods is forbidden: User “system:serviceaccount:default:kubectl-get-nodes” ca
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

roshan_thapa profile image

The Ultimate Docker Tutorial

Roshan Thapa -

devleader profile image

Activator.CreateInstance vs Type.InvokeMember – A Clear Winner?

Dev Leader -

devleader profile image

Async Event Handlers in C#: What You Need to Know

Dev Leader -

devleader profile image

Plugin Architecture in C# for Improved Software Design

Dev Leader -