Kubernetes cluster with docker private registry

Kubernetes cluster with docker private registry

Prerequisites

  • One or more machines running one of Ubuntu 20.04+
  • 2 GB or more of RAM per machine (any less will leave little room for your apps)
  • 2 CPUs or more
  • Full network connectivity between all machines in the cluster (public or private network is fine)
* 192.168.147.3 is the IP of my machine which is used as master in the cluster

Putting it all together

1. Install Docker

# Install Docker CE
## Set up the repository:
### Update the apt package index
apt-get update

### Install packages to allow apt to use a repository over HTTPS
apt-get update && apt-get install apt-transport-https software-properties-common ca-certificates curl gnupg lsb-release

### Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

### Add docker apt repository.
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

## Install docker ce.
apt-get install docker-ce docker-ce-cli containerd.io

# Setup daemon.
tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "insecure-registries" : ["192.168.147.3:5000"]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart docker.
systemctl daemon-reload
systemctl restart docker

systemctl enable docker

2. Install kubeadm 

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

apt-get update
apt-get -y install vim git curl wget kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

# Restart kubelet.
systemctl daemon-reload
systemctl restart kubelet

3. Creating a single master cluster with kubeadm and Calico CNI on master node

# Reset.
kubeadm reset
rm -rf /var/etcd/calico-data
rm -rf $HOME/.kube

# Init.
kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=192.168.147.3
export KUBECONFIG=/etc/kubernetes/admin.conf
mkdir -p $HOME/.kube
chmod 775 /etc/kubernetes/admin.conf
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# Remove the taints on the master so that you can schedule pods on it.
kubectl taint nodes --all node-role.kubernetes.io/master-

# Install Calico CNI.
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/etcd.yaml
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/calico.yaml

4. Joining a node to cluster

# Reset.
kubeadm reset

# Resulted from running "kubeadm init...." on master.
kubeadm join 192.168.147.3:6443 --token ftnbv5.7yd3v97wl884n8q4 \
    --discovery-token-ca-cert-hash sha256:d5b0337ec0c0a5357361d8e05ce05a2079295bf8e32d04a29d45046d4e8b860e

By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the master node:

kubeadm token create

5. Verify the installation on master node

# List nodes.
kubectl get nodes --all-namespaces -owide

# List deployments.
kubectl get deployments --all-namespaces -owide

# List pods.
kubectl get pods --all-namespaces -owide

6. Creating a docker private registry on master node

# Set basic auth.
rm -rf /auth/*
mkdir -p /auth
docker run --entrypoint htpasswd registry:2 -Bbn test test > /auth/htpasswd

docker rm registry -f


# Set certificates auth.
rm -rf /certs/*
mkdir -p /certs
openssl genrsa 1024 > /certs/registrykey.pem
chmod 400 /certs/registrykey.pem
openssl req -new -x509 -nodes -sha1 -days 365 -key /certs/registrykey.pem  -out /certs/registry.pem  -subj "/C=/ST=/L=/O=/OU=/CN=registry.com" > /dev/null 2>&1
docker run -d -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 -p 5000:5000 --restart=always --name registry -v `pwd`/auth:/auth -e "REGISTRY_AUTH=htpasswd" \
    -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v `pwd`/certs:/certs \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.pem -e REGISTRY_HTTP_TLS_KEY=/certs/registrykey.pem registry:2

# Create secret to be used in "imagePullSecrets" section of a pod
kubectl create secret docker-registry regsecret --docker-server=192.168.147.3:5000 --docker-username=test --docker-password=test --namespace=kube-system

# Push image in private registry.
docker tag test-image:latest 192.168.147.3:5000/test-image
docker push 192.168.147.3:5000/test-image

7. YAML example for pod with image from private registry

apiVersion: v1
kind: Pod
metadata:
  name: test-site
  labels:
    app: web
spec:
  containers:
    - name: test
  image: 192.168.147.3:5000/test-image:latest
    ports:
      - containerPort: 8000
  imagePullPolicy: Always
    imagePullSecrets:
      - name: regsecret

Enjoy (ง°ل͜°)ง

Comments

Post a Comment

Popular Posts