Balkrishna Pandey

Posted on May 16, 2022 • Originally published at goglides.io on Mar 28, 2020

Running Kaniko in (Docker/ Kubernetes/ using docker hub and AWS ECR

#docker #kaniko #kubernetes

Introduction

Anyone who doesn’t know about Kaniko (official docker image gcr.io/kaniko-project/executor), directly from official repo, its a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

This tool doesn’t depend on a Docker daemon.
Executes each command within a Dockerfile in userspace.
Suitable for the environment like standard Kubernetes Cluster where it is hard to run Docker daemon securely.

How Kaniko works?

Kaniko takes 3 arguments:

A Dockerfile,
A build context and
The name of the registry to which it should push the final image.

Note: Image reconstructed based on google blog

kaniko how it works?

The above diagram shows how Kaniko works:

Kaniko executor image is responsible for building an image from a Dockerfile and pushing it to a registry.
Kaniko extract the filesystem of the base image (the FROM image in the Dockerfile).
Kaniko then executes the commands in the Dockerfile.
For each command execution, snapshot the filesystem in userspace.
After each command, kaniko append a layer of changed files to the base image (if there are any) and update image metadata.
Finally save stage and publish to the image repository.

In this process, docker daemon or CLI is not involved.

(adsbygoogle = window.adsbygoogle || []).push({});

Kaniko itself is running as root!

I was going throw this blog, quick look at google kaniko, Kaniko actually run as root (uid=0). Run the following command to see this.


$ docker run -it --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug
/ # id
uid=0 gid=0

While Kaniko needs to run as root it can run as an unprivileged container.

Jessfraz’ take on Kaniko’s security model

 (adsbygoogle = window.adsbygoogle || []).push({});

She drew a picture showing how Kaniko works.

Jessie Frazelle : How Kaniko Works

Based on this conversation Kaniko is not as secure as I thought.

Also, Kaniko official page mentioned following about security,

If you have a minimal base image (SCRATCH or similar) that doesn’t require permissions to unpack, and your Dockerfile doesn’t execute any commands as the root user, you can run Kaniko without root permissions. It should be noted that Docker runs as root by default, so you still require (in a sense) privileges to use Kaniko.

Confusing??? I think so.

Let’s move further.

Using Kaniko

Let’s build something so that we can test Kaniko from scratch. I am following alexellis.io blog (link in section reference below) to build OpenFaaS Go function. Let’s install faas-cli (not mandatory to complete this tutorial but never tried this before, so I want to learn.)

I am using macOS so we can install this tool using brew


$ brew install faas-cli

or use below


$ curl -SLs cli.openfaas.com | sudo sh

Let’s verify


$ faas-cli version
  ________  ____
 / _ \ _ _____ _ __|___ |_ _ ___/___ |
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
 \ ___/| .__ / \ ___|_| |_|_| \__ ,_|\ __,_|____ /
      |_|
CLI:
 commit: ea687659ecf14931a29be46c4d2866899d36c282
 version: 0.11.8

Generate a Go function


$ mkdir -p tutorial 
$ cd tutorial
$ faas-cli new --lang go hello-world
2020/03/24 20:25:27 No templates found in current directory.
2020/03/24 20:25:27 Attempting to expand templates from https://github.com/openfaas/templates.git
2020/03/24 20:25:28 Fetched 19 template(s) : [csharp csharp-armhf dockerfile go go-armhf java11 java11-vert-x java8 node node-arm64 node-armhf node12 php7 python python-armhf python3 python3-armhf python3-debian ruby] from https://github.com/openfaas/templates.git
Folder: hello-world created.
  ________  ____
 / _ \ _ _____ _ __|___ |_ _ ___/___ |
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
 \ ___/| .__ / \ ___|_| |_|_| \__ ,_|\ __,_|____ /
      |_|
Function created in the folder: hello-world
Stack file written: hello-world.yml
Notes:
You have created a new function which uses Golang 1.11 and the Classic
OpenFaaS template.
To include third-party dependencies, use a vendoring tool like dep:
dep documentation: https://github.com/golang/dep#installation
For high-throughput applications, we recommend using the golang-http
or golang-middleware templates instead available via the store.

The tool generated a default folder structure with a sample code.


$ tree
.
├── hello-world
│ └── handler.go
├── hello-world.yml
└── template
    ├── csharp
...
128 directories, 173 files

You will see a handler.go file with the following content,


package function
import (
  "fmt"
)
// Handle a serverless request
func Handle(req []byte) string {
  return fmt.Sprintf("Hello, Go. You said: %s", string(req))
}

and hello-world.yaml file with following content,


version: 1.0
provider:
  name: openfaas
  gateway: http://127.0.0.1:8080
functions:
  hello-world:
    lang: go
    handler: ./hello-world
    image: hello-world:latest

Let’s build the function in a FaaS way,


$ faas-cli build -f hello-world.yml
[0] > Building hello-world.
Clearing temporary build folder: ./build/hello-world/
Preparing: ./hello-world/ build/hello-world/function
Building: hello-world:latest with go template. Please wait..
Sending build context to Docker daemon 8.192kB
Step 1/30 : FROM openfaas/classic-watchdog:0.18.1 as watchdog
 ---> 94b5e0bef891
...
...
...
Step 30/30 : CMD ["./fwatchdog"]
 ---> Using cache
 ---> 26c4b9fccfd2
Successfully built 26c4b9fccfd2
Successfully tagged hello-world:latest
Image: hello-world:latest built.
[0] < Building hello-world done in 0.59s.
[0] Worker done.
Total build time: 0.59s

Now we can verify this using following,


$ docker run -p8080:8080 hello-world:latest
$ curl localhost:8080/ -d "Welcome to https://www.goglides.com"
Hello, Go. You said: Welcome to https://www.goglides.com

Everything looks fine so far. Let’s run a build with Kaniko.

Running Kaniko in Docker

I am using the Docker hub for this purpose. For this let’s login to hub account using docker login command


$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: bkpandey
Password: 
WARNING! Your password will be stored unencrypted in /Users/bkpandey/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded

Now run following command from your openfaas project root folder,


 docker run -v $PWD/build/hello-world:/workspace \
  -v ~/.docker/config.json:/kaniko/config.json \
  --env DOCKER_CONFIG=/kaniko gcr.io/kaniko-project/executor:latest \
  -d bkpandey/openfaas-hello-world:0.0.1

It will take sometime to complete, you will see output like this:


Output:
Unable to find image 'gcr.io/kaniko-project/executor:latest' locally
latest: Pulling from kaniko-project/executor
c30f0b4c9053: Pull complete 
ed36162ea2d3: Pull complete 
934aba279703: Pull complete 
958c06b88e30: Pull complete 
9fa803ac1ec6: Pull complete 
03a44a7314d7: Pull complete 
861e678c6f4c: Pull complete 
e60dcc1bf57d: Pull complete 
Digest: sha256:66be3f60f22b571faa82e0aaeb94731217ba0c58ac4a3b062bc84c6d8d545213
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
INFO[0001] Resolved base name openfaas/classic-watchdog:0.18.1 to openfaas/classic-watchdog:0.18.1 
INFO[0001] Resolved base name golang:1.13-alpine3.11 to golang:1.13-alpine3.11 
INFO[0001] Resolved base name alpine:3.11 to alpine:3.11 
INFO[0001] Resolved base name openfaas/classic-watchdog:0.18.1 to openfaas/classic-watchdog:0.18.1 
INFO[0001] Resolved base name golang:1.13-alpine3.11 to golang:1.13-alpine3.11 
INFO[0001] Resolved base name alpine:3.11 to alpine:3.11 
INFO[0001] Retrieving image manifest openfaas/classic-watchdog:0.18.1 
INFO[0002] Retrieving image manifest openfaas/classic-watchdog:0.18.1 
INFO[0002] Retrieving image manifest golang:1.13-alpine3.11 
INFO[0003] Retrieving image manifest golang:1.13-alpine3.11 
INFO[0004] Retrieving image manifest alpine:3.11        
INFO[0004] Retrieving image manifest alpine:3.11        
INFO[0005] Built cross stage deps: map[0:[/fwatchdog] 1:[/usr/bin/fwatchdog /go/src/handler/function/ /go/src/handler/handler]] 
INFO[0005] Retrieving image manifest openfaas/classic-watchdog:0.18.1 
INFO[0005] Retrieving image manifest openfaas/classic-watchdog:0.18.1 
INFO[0007] Taking snapshot of full filesystem...        
INFO[0007] Resolving paths                              
INFO[0007] Saving file /fwatchdog for later use         
INFO[0007] Deleting filesystem...                       
INFO[0007] Retrieving image manifest golang:1.13-alpine3.11 
INFO[0007] Retrieving image manifest golang:1.13-alpine3.11 
INFO[0008] Unpacking rootfs as cmd COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog requires it. 
INFO[0020] Taking snapshot of full filesystem...        
INFO[0021] Resolving paths                              
INFO[0022] ARG ADDITIONAL_PACKAGE                       
INFO[0022] ARG CGO_ENABLED=0                            
INFO[0022] ARG GO111MODULE="off"                        
INFO[0022] ARG GOPROXY=""                               
INFO[0022] ARG GOFLAGS=""                               
INFO[0022] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog 
INFO[0022] Resolving paths                              
INFO[0022] Taking snapshot of files...                  
INFO[0022] RUN chmod +x /usr/bin/fwatchdog              
INFO[0022] cmd: /bin/sh                                 
INFO[0022] args: [-c chmod +x /usr/bin/fwatchdog]       
INFO[0022] Taking snapshot of full filesystem...        
INFO[0022] Resolving paths                              
INFO[0023] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0023] ENV CGO_ENABLED=0                            
INFO[0023] WORKDIR /go/src/handler                      
INFO[0023] cmd: workdir                                 
INFO[0023] Changed working directory to /go/src/handler 
INFO[0023] Creating directory /go/src/handler           
INFO[0023] Resolving paths                              
INFO[0023] Taking snapshot of files...                  
INFO[0023] COPY . .                                     
INFO[0023] Resolving paths                              
INFO[0023] Taking snapshot of files...                  
INFO[0023] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0 
INFO[0023] cmd: /bin/sh                                 
INFO[0023] args: [-c cat function/GO_REPLACE.txt >> ./go.mod || exit 0] 
cat: can't open 'function/GO_REPLACE.txt': No such file or directory
INFO[0023] Taking snapshot of full filesystem...        
INFO[0023] Resolving paths                              
INFO[0024] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0024] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; } 
INFO[0024] cmd: /bin/sh                                 
INFO[0024] args: [-c test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }] 
INFO[0024] Taking snapshot of full filesystem...        
INFO[0024] Resolving paths                              
INFO[0025] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0025] WORKDIR /go/src/handler/function             
INFO[0025] cmd: workdir                                 
INFO[0025] Changed working directory to /go/src/handler/function 
INFO[0025] RUN go test ./... -cover                     
INFO[0025] cmd: /bin/sh                                 
INFO[0025] args: [-c go test ./... -cover]              
? handler/function [no test files]
INFO[0026] Taking snapshot of full filesystem...        
INFO[0026] Resolving paths                              
INFO[0027] WORKDIR /go/src/handler                      
INFO[0027] cmd: workdir                                 
INFO[0027] Changed working directory to /go/src/handler 
INFO[0027] RUN CGO_ENABLED=${CGO_ENABLED} GOOS=linux go build --ldflags "-s -w" -a -installsuffix cgo -o handler . 
INFO[0027] cmd: /bin/sh                                 
INFO[0027] args: [-c CGO_ENABLED=${CGO_ENABLED} GOOS=linux go build --ldflags "-s -w" -a -installsuffix cgo -o handler .] 
INFO[0030] Taking snapshot of full filesystem...        
INFO[0030] Resolving paths                              
INFO[0031] Saving file /usr/bin/fwatchdog for later use 
INFO[0031] Saving file /go/src/handler/function/ for later use 
INFO[0031] Saving file /go/src/handler/handler for later use 
INFO[0031] Deleting filesystem...                       
INFO[0032] Retrieving image manifest alpine:3.11        
INFO[0032] Retrieving image manifest alpine:3.11        
INFO[0033] Unpacking rootfs as cmd RUN apk --no-cache add ca-certificates && addgroup -S app && adduser -S -g app app && mkdir -p /home/app && chown app /home/app requires it. 
INFO[0033] Taking snapshot of full filesystem...        
INFO[0033] Resolving paths                              
INFO[0033] RUN apk --no-cache add ca-certificates && addgroup -S app && adduser -S -g app app && mkdir -p /home/app && chown app /home/app 
INFO[0033] cmd: /bin/sh                                 
INFO[0033] args: [-c apk --no-cache add ca-certificates && addgroup -S app && adduser -S -g app app && mkdir -p /home/app && chown app /home/app] 
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/1) Installing ca-certificates (20191127-r1)
Executing busybox-1.31.1-r9.trigger
Executing ca-certificates-20191127-r1.trigger
OK: 6 MiB in 15 packages
INFO[0034] Taking snapshot of full filesystem...        
INFO[0034] Resolving paths                              
INFO[0034] WORKDIR /home/app                            
INFO[0034] cmd: workdir                                 
INFO[0034] Changed working directory to /home/app       
INFO[0034] COPY --from=builder /usr/bin/fwatchdog . 
INFO[0034] Resolving paths                              
INFO[0034] Taking snapshot of files...                  
INFO[0034] COPY --from=builder /go/src/handler/function/ . 
INFO[0034] Resolving paths                              
INFO[0034] Taking snapshot of files...                  
INFO[0034] COPY --from=builder /go/src/handler/handler . 
INFO[0034] Resolving paths                              
INFO[0034] Taking snapshot of files...                  
INFO[0034] RUN chown -R app /home/app                   
INFO[0034] cmd: /bin/sh                                 
INFO[0034] args: [-c chown -R app /home/app]            
INFO[0034] Taking snapshot of full filesystem...        
INFO[0034] Resolving paths                              
INFO[0034] USER app                                     
INFO[0034] cmd: USER                                    
INFO[0034] ENV fprocess="./handler"                     
INFO[0034] EXPOSE 8080                                  
INFO[0034] cmd: EXPOSE                                  
INFO[0034] Adding exposed port: 8080/tcp                
INFO[0034] HEALTHCHECK --interval=3s CMD [-e /tmp/.lock] || exit 1 
INFO[0034] CMD ["./fwatchdog"]

 (adsbygoogle = window.adsbygoogle || []).push({});

Everything looks fine. You can verify this by login to docker hub account. I am seeing following output.

Let’s confirm by running this docker image,


$ docker run -p 8080:8080 bkpandey/openfaas-hello-world:0.0.1
Output:
2020/03/28 16:48:12 Version: 0.18.1 SHA: b46be5a4d9d9d55da9c4b1e50d86346e0afccf2d
2020/03/28 16:48:12 Timeouts: read: 5s, write: 5s hard: 0s.
2020/03/28 16:48:12 Listening on port: 8080
2020/03/28 16:48:12 Writing lock-file to: /tmp/.lock
2020/03/28 16:48:12 Metrics listening on port: 8081

Now run curl command


$ curl localhost:8080 -d "Welcome to https://www.goglides.com"
Output:
Hello, Go. You said: Welcome to https://www.goglides.com

Running Kaniko in Kubernetes Cluster

Running locally (using docker for mac)

To run kaniko in a Kubernetes cluster, you will need a standard running Kubernetes cluster. I am using docker for mac to run this locally. Feel free to use any variation shouldn’t matter. Also in this blog, I am going to use Aws ECR as a Docker registry and s3 as a source code holder.

I took pod spec directly from Kaniko official repo.


apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=./Dockerfile",
            "--context=s3://<bucket-name>/tutorial-s3.tar.gz",
            "--destination=<aws-account>.dkr.ecr.us-west-2.amazonaws.com/kaniko/hello-world:latest"]
    volumeMounts:
      - name: docker-config
        mountPath: /kaniko/.docker/
      # when not using instance role
      - name: aws-secret
        mountPath: /root/.aws/
    env:
      - name: AWS_REGION
        value: us-west-2
  restartPolicy: Never
  volumes:
    - name: docker-config
      configMap:
        name: docker-config
    # when not using instance role
    - name: aws-secret
      secret:
        secretName: aws-secret

The manifest has 2 configurations, configmap docker-config and secret aws-secret. As per the official page, if you use the Kubernetes cluster which exists on AWS, we can attach IAM roles policies for authentication. For this section, let’s create configmap and secrets with the appropriate config.

Create docker-config configmap as follows, replace <aws-account> and us-west-2 as per your need,


$ cat <<EOF >config.json
{
  "credHelpers": {
    "<aws-account>.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
  }
}
EOF
$ kubectl create configmap docker-config --from-file=config.json

Now create aws-secret as follows, make sure to replace access key and secret key.

$ cat <<EOF>credentials
[default]
aws_access_key_id = <change-me>
aws_secret_access_key = <change-me>
region = us-east-2
EOF

$ kubectl create secret generic aws-secret --from-file=credentials

Also, I am using an S3 bucket, so you will first need to create a compressed tar of your build context and upload it to your bucket. Once running, kaniko will then download and unpack the compressed tar of the build context before starting the image build.

To create a compressed tar, you can run:


$ tar -C <path to build context> -zcvf tutorial-s3.tar.gz .

In our case,


$ tar -C tutorial/build/hello-world/ -zcvf tutorial-s3.tar.gz .

And you can use aws s3 command to upload artificate as follows,


$ aws s3 cp tutorial-s3.tar.gz s3://<bucket-name>/tutorial-s3.tar.gz
upload: ./tutorial-s3.tar.gz to s3://<bucket-name>/tutorial-s3.tar.gz

Now you can apply the above pod manifest file using kubectl apply.

Running on AWS ec2 machine

The process is almost similar in the AWS k8s cluster. Except instead of creating aws-secret we can leverage instance profile to authenticate with AWS API. So follow the above process (minus aws-sceret creation part). And before you apply pod manifest make sure your cluster ec2 instances have the proper permission. Your worker nodes must possess the following IAM policy permissions for Amazon ECR.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:PutImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload"
            ],
            "Resource": "*"
        }
    ]
}

Now run kaniko as follows, make sure to replace <bucket-name>, <aws-account> and AWS_REGION based on your settings.


apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=./Dockerfile",
            "--context=s3://<bucket-name>/tutorial-s3.tar.gz",
            "--destination=<aws-account>.dkr.ecr.us-west-2.amazonaws.com/kaniko/hello-world:latest"]
    volumeMounts:
      - name: docker-config
        mountPath: /kaniko/.docker/
    env:
      - name: AWS_REGION
        value: us-west-2
  volumes:
    - name: docker-config
      configMap:
        name: docker-config

Verification

All done, you can verify by checking logs kubectl logs kaniko and running docker image using ECR,


$ docker run -p 8080:8080 <aws-account>.dkr.ecr.us-west-2.amazonaws.com/kaniko/hello-world:latest
$ curl localhost:8080 -d "hello goglides.com"
Hello, Go. You said: hello goglides.com

 (adsbygoogle = window.adsbygoogle || []).push({});

References:

https://github.com/GoogleContainerTools/kaniko

https://cloud.google.com/blog/products/gcp/introducing-kaniko-build-container-images-in-kubernetes-and-google-container-builder-even-without-root-access

https://docs.gitlab.com/ee/ci/docker/using_kaniko.html https://hub.docker.com/r/csanchez/kaniko https://blog.alexellis.io/quick-look-at-google-kaniko/ https://www.openfaas.com/

Goglides Dev 🌱

Running Kaniko in (Docker/ Kubernetes/ using docker hub and AWS ECR

Introduction

How Kaniko works?

Kaniko itself is running as root!

Using Kaniko

Running Kaniko in Docker

Running Kaniko in Kubernetes Cluster

Running locally (using docker for mac)

Running on AWS ec2 machine

Verification

References:

Top comments (0)

Read next

Regular Expressions In C#: 3 Examples You Need To Know

Regex Options In C#: A Beginner’s Guide To Powerful Pattern Matching

Collection Initializer Performance in C# – How To Get An 87% Boost!

How to Fix Suspect Database in SQL Server? [Resolved]