Netmaker - Multi Cloud Networking
Rahul Kiran Gaddam
Posted on September 22, 2022
Philosophy
- Now Multi cloud is the new reality, because not everything is available with every cloud provider.
- With this being new reality, we need a mechanism to connect cloud environments and create a internal and secured network.
Overview
-
Here was my last post on installing kubernetes using
Oracle OCI cloud
. - Instead of creating a single node, can we create a cluster? can we create this cluster between multiple regions? how will they communicate?
Network Communication Is a PROBLEM >>>>
- Answer for all these is VPN or we can try Netmaker.io
Installation
- Netmaker works on
Client
-Server
architecture like any other network connectivity resolution applications. - We will have a
Netmaker Server
installed on a server andNetmaker Agent/Client
installed on the others. - Below installation is done on
OCI Arm64 - Oracle Linux
, which could not be achieved by following steps here - Create
3 Node
where2 Node
will act asAgents/Clients
and1 Server
- All nodes should have
Public IP
- Allow firewall for Ports 443 (tcp): for Traefik & 51821-518XX (udp): for WireGuard
- Register below domains freenom
- base-domain.extenssion
- broker-base-domain.extenssion
- api-base-domain.extenssion
- dashboard-base-domain.extenssion
- All nodes should have
Server Installation
- I created below folders
- Host
#!/bin/bash
PUBLIC_IP=$(curl -s ifconfig.me)
PRIVATE_IP=$(hostname -I | cut -f 1 -d " ")
DOMAIN_NAME="base-domain.extenssion"
MATCHES_IN_HOSTS="$(grep -n $PUBLIC_IP /etc/hosts)"
hostnamectl set-hostname $DOMAIN_NAME
if [ ! -z "$MATCHES_IN_HOSTS" ]
then
echo "Host Details Already Exisits"
else
echo "Registering Host Details";
echo "===================================";
cat /dev/null > /etc/hosts;
echo "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4" >> /etc/hosts;
echo "::1 localhost localhost.localdomain localhost6 localhost6.localdomain6" >> /etc/hosts;
echo "$PRIVATE_IP $PUBLIC_IP $DOMAIN_NAME" >> /etc/hosts;
echo "========Updated Details============";
cat /etc/hosts
echo "===================================";
fi
- Firewall
#!/bin/bash
# -- Disabling firewalld
systemctl disable firewalld
# -- Enabling iptables
yum install iptables-services -y
systemctl start iptables
systemctl enable iptables
# -- Flushing iptables
iptables -F
# -- Allowing everthing
iptables -A FORWARD -j ACCEPT
iptables -A INPUT -j ACCEPT
iptables -A OUTPUT -j ACCEPT
# -- Saving
service iptables save
systemctl restart iptables
# -- Display Settings
iptables -L -n
- Docker
#!/bin/bash
# -- Enable kernel modules
modprobe overlay
modprobe br_netfilter
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
# -- Disabling Swap Memory
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
## Podman is by default provided, K8 can run on Podman
## I was unable to install using Podman and need to move to docker
# -- Remove Podman
yum remove podman buildah -y
# -- Install Docker
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io
# -- Configure Docker
systemctl stop docker
/usr/sbin/usermod -a -G docker opc
/usr/sbin/sysctl net.ipv4.conf.all.forwarding=1
systemctl start docker
chmod 777 /var/run/docker.sock
tee /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# -- Start and enable Services
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker
- Netmaker
# docker-compose.yml
version: "3.4"
services:
netmaker:
container_name: netmaker
image: gravitl/netmaker:v0.15.2@sha256:1e4cb5ca0907eea83eb84b850fe5e242e481dd4c1be59b60f96d5e577c67f5a9
cap_add:
- NET_ADMIN
- NET_RAW
- SYS_MODULE
sysctls:
- net.ipv4.ip_forward=1
- net.ipv4.conf.all.src_valid_mark=1
- net.ipv6.conf.all.disable_ipv6=0
- net.ipv6.conf.all.forwarding=1
restart: always
volumes:
- /root/dnsconfig:/root/config/dnsconfig
- /root/sqldata:/root/data
- /root/shared_certs:/etc/netmaker
environment:
SERVER_NAME: "broker-base-domain.extenssion"
SERVER_HOST: "SERVER_PUBLIC_IP"
SERVER_API_CONN_STRING: "api-base-domain.extenssion:443"
COREDNS_ADDR: "SERVER_PUBLIC_IP"
DNS_MODE: "on"
SERVER_HTTP_HOST: "api-base-domain.extenssion"
API_PORT: "8081"
CLIENT_MODE: "on"
MASTER_KEY: "REPLACE_MASTER_KEY"
CORS_ALLOWED_ORIGIN: "*"
DISPLAY_KEYS: "on"
DATABASE: "sqlite"
NODE_ID: "netmaker-server-1"
MQ_HOST: "mq"
MQ_PORT: "443"
MQ_SERVER_PORT: "1883"
HOST_NETWORK: "off"
VERBOSITY: "1"
MANAGE_IPTABLES: "on"
PORT_FORWARD_SERVICES: "dns"
ports:
- "51821-51830:51821-51830/udp"
expose:
- "8081"
labels:
- traefik.enable=true
- traefik.http.routers.netmaker-api.entrypoints=websecure
- traefik.http.routers.netmaker-api.rule=Host(`api-base-domain.extenssion`)
- traefik.http.routers.netmaker-api.service=netmaker-api
- traefik.http.services.netmaker-api.loadbalancer.server.port=8081
netmaker-ui:
container_name: netmaker-ui
image: gravitl/netmaker-ui:v0.15.2@sha256:11fe0092e8a8e8a7a6a07e6aa50d448ed2de24ee6d2eb045e3956b3c6c24af50
depends_on:
- netmaker
links:
- "netmaker:api"
restart: always
environment:
BACKEND_URL: "https://api-base-domain.extenssion"
expose:
- "80"
labels:
- traefik.enable=true
- traefik.http.middlewares.nmui-security.headers.accessControlAllowOriginList=*-base-domain.extenssion
- traefik.http.middlewares.nmui-security.headers.stsSeconds=31536000
- traefik.http.middlewares.nmui-security.headers.browserXssFilter=true
- traefik.http.middlewares.nmui-security.headers.customFrameOptionsValue=SAMEORIGIN
- traefik.http.middlewares.nmui-security.headers.customResponseHeaders.X-Robots-Tag=none
- traefik.http.middlewares.nmui-security.headers.customResponseHeaders.Server= # Remove the server name
- traefik.http.routers.netmaker-ui.entrypoints=websecure
- traefik.http.routers.netmaker-ui.middlewares=nmui-security@docker
- traefik.http.routers.netmaker-ui.rule=Host(`dashboard-base-domain.extenssion`)
- traefik.http.routers.netmaker-ui.service=netmaker-ui
- traefik.http.services.netmaker-ui.loadbalancer.server.port=80
coredns:
container_name: coredns
image: coredns/coredns-arm64@sha256:224c4ecc9d9eea3765d0beee0e624e6cf837230c370440bd38a7d9901dd04dc4
command: -conf /root/dnsconfig/Corefile
depends_on:
- netmaker
restart: always
volumes:
- /root/dnsconfig:/root/dnsconfig
traefik:
image: traefik:v2.6@sha256:9aecceb73e3b24b6547d401c95eea6cdf475a99ddfd0b86464c5413925e062da
container_name: traefik
command:
- "--certificatesresolvers.http.acme.email=YOUR_EMAIL"
- "--certificatesresolvers.http.acme.storage=/letsencrypt/acme.json"
- "--certificatesresolvers.http.acme.tlschallenge=true"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.websecure.http.tls=true"
- "--entrypoints.websecure.http.tls.certResolver=http"
- "--log.level=INFO"
- "--providers.docker=true"
- "--providers.docker.exposedByDefault=false"
- "--serverstransport.insecureskipverify=true"
restart: always
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /root/traefik_certs:/letsencrypt
ports:
- "443:443"
mq:
container_name: mq
image: eclipse-mosquitto:2.0.11-openssl@sha256:459f8503a3a248156855501a8fa718b92783bf02fc5b1ea414fae07ca1d1396d
depends_on:
- netmaker
restart: unless-stopped
volumes:
- /root/mosquitto.conf:/mosquitto/config/mosquitto.conf
- /root/mosquitto_data:/mosquitto/data
- /root/mosquitto_logs:/mosquitto/log
- /root/shared_certs:/mosquitto/certs
expose:
- "8883"
labels:
- traefik.enable=true
- traefik.tcp.routers.mqtts.rule=HostSNI(`broker-base-domain.extenssion`)
- traefik.tcp.routers.mqtts.tls.passthrough=true
- traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
- traefik.tcp.routers.mqtts.service=mqtts-svc
- traefik.tcp.routers.mqtts.entrypoints=websecure
# Create Corefile by removing this comment
coredns.io {
log stdout
file /root/dnsconfig/coredns.dat
}
#!/bin/bash
# -- Prerequists
# yum install -y wireguard-tools net-tools jq
# modprobe ip_tables
# echo 'ip_tables' >> /etc/modules
# -- Install Docker Compose
# curl -L --fail https://github.com/docker/compose/releases/download/v2.11.0/docker-compose-linux-aarch64 -o /usr/sbin/docker-compose
# chmod +x /usr/sbin/docker-compose
# -- Setting env variables
NETMAKER_BASE_DOMAIN=$DOMAIN_NAME
COREDNS_IP=$(ip route get 1 | sed -n 's/^.*src \([0-9.]*\) .*$/\1/p')
SERVER_PUBLIC_IP=$(curl -s ifconfig.me)
MASTER_KEY=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 30 ; echo '')
EMAIL="gaddam.rahul@email.com"
MESH_SETUP="true"
VPN_SETUP="false"
NUM_CLIENTS=5
echo " ----------------------------"
echo " SETUP ARGUMENTS"
echo " ----------------------------"
echo " domain: $NETMAKER_BASE_DOMAIN"
echo " email: $EMAIL"
echo " coredns ip: $COREDNS_IP"
echo " public ip: $SERVER_PUBLIC_IP"
echo " master key: $MASTER_KEY"
echo " setup mesh?: $MESH_SETUP"
echo " setup vpn?: $VPN_SETUP"
if [ "${VPN_SETUP}" == "true" ]; then
echo " # clients: $NUM_CLIENTS"
fi
echo " ----------------------------"
sleep 5
# -- Installation
echo "setting mosquitto.conf..."
wget -q -O /root/mosquitto.conf https://raw.githubusercontent.com/gravitl/netmaker/master/docker/mosquitto.conf
echo "setting docker-compose..."
mkdir -p /root/dnsconfig
cp /home/opc/setup/netmaker/Corefile /root/dnsconfig/
mkdir -p /root/traefik_certs
mkdir -p /root/shared_certs
mkdir -p /root/sqldata
mkdir -p /root/mosquitto_data
mkdir -p /root/mosquitto_logs
cp /home/opc/setup/netmaker/docker-compose.yml /root/docker-compose.yml
sed -i "s/SERVER_PUBLIC_IP/$SERVER_PUBLIC_IP/g" /root/docker-compose.yml
sed -i "s/COREDNS_IP/$COREDNS_IP/g" /root/docker-compose.yml
sed -i "s/REPLACE_MASTER_KEY/$MASTER_KEY/g" /root/docker-compose.yml
sed -i "s/YOUR_EMAIL/$EMAIL/g" /root/docker-compose.yml
echo "starting containers..."
docker-compose -f /root/docker-compose.yml up -d
test_connection() {
echo "testing Traefik setup (please be patient, this may take 1-2 minutes)"
for i in 1 2 3 4 5 6
do
curlresponse=$(curl -vIs https://api-base-domain.extenssion 2>&1)
if [[ "$i" == 6 ]]; then
echo " Traefik is having an issue setting up certificates, please investigate (docker logs traefik)"
echo " exiting..."
exit 1
elif [[ "$curlresponse" == *"failed to verify the legitimacy of the server"* ]]; then
echo " certificates not yet configured, retrying..."
elif [[ "$curlresponse" == *"left intact"* ]]; then
echo " certificates ok"
break
else
secs=$(($i*5+10))
echo " issue establishing connection...retrying in $secs seconds..."
fi
sleep $secs
done
}
set +e
test_connection
cat << "EOF"
__ __ ______ ______ __ __ ______ __ __ ______ ______
/\ "-.\ \ /\ ___\ /\__ _\ /\ "-./ \ /\ __ \ /\ \/ / /\ ___\ /\ == \
\ \ \-. \ \ \ __\ \/_/\ \/ \ \ \-./\ \ \ \ __ \ \ \ _"-. \ \ __\ \ \ __<
\ \_\\"\_\ \ \_____\ \ \_\ \ \_\ \ \_\ \ \_\ \_\ \ \_\ \_\ \ \_____\ \ \_\ \_\
\/_/ \/_/ \/_____/ \/_/ \/_/ \/_/ \/_/\/_/ \/_/\/_/ \/_____/ \/_/ /_/
EOF
echo "visit https://dashboard-base-domain.extenssion to log in"
sleep 7
setup_mesh() {( set -e
echo "creating netmaker network (10.101.0.0/16)"
curl -s -o /dev/null -d '{"addressrange":"10.101.0.0/16","netid":"netmaker"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks
sleep 5
echo "creating netmaker access key"
curlresponse=$(curl -s -d '{"uses":99999,"name":"netmaker-key"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks/netmaker/keys)
ACCESS_TOKEN=$(jq -r '.accessstring' <<< ${curlresponse})
sleep 5
echo "configuring netmaker server as ingress gateway"
curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/netmaker)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})
curl -o /dev/null -s -X POST -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/netmaker/$SERVER_ID/createingress
sleep 5
echo "finished configuring server and network. You can now add clients."
echo ""
echo "For Linux, Mac, Windows, and FreeBSD:"
echo " 1. Install the netclient: https://docs.netmaker.org/netclient.html#installation"
echo " 2. Join the network: netclient join -t $ACCESS_TOKEN"
echo ""
echo "For Android and iOS clients, perform the following steps:"
echo " 1. Log into UI at dashboard-base-domain.extenssion"
echo " 2. Navigate to \"EXTERNAL CLIENTS\" tab"
echo " 3. Select the gateway and create clients"
echo " 4. Scan the QR Code from WireGuard app in iOS or Android"
echo ""
echo "Netmaker setup is now complete. You are ready to begin using Netmaker."
)}
setup_vpn() {( set -e
echo "creating vpn network (10.201.0.0/16)"
curl -s -o /dev/null -d '{"addressrange":"10.201.0.0/16","netid":"vpn","defaultextclientdns":"10.201.255.254"}' -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/networks
sleep 5
echo "configuring netmaker server as vpn inlet..."
curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})
curl -s -o /dev/null -X POST -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn/$SERVER_ID/createingress
echo "waiting 5 seconds for server to apply configuration..."
sleep 5
echo "configuring netmaker server vpn gateway..."
[ -z "$GATEWAY_IFACE" ] && GATEWAY_IFACE=$(ip -4 route ls | grep default | grep -Po '(?<=dev )(\S+)')
echo "gateway iface: $GATEWAY_IFACE"
curlresponse=$(curl -s -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn)
SERVER_ID=$(jq -r '.[0].id' <<< ${curlresponse})
EGRESS_JSON=$( jq -n \
--arg gw "$GATEWAY_IFACE" \
'{ranges: ["0.0.0.0/0"], interface: $gw}' )
echo "egress json: $EGRESS_JSON"
curl -s -o /dev/null -X POST -d "$EGRESS_JSON" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/nodes/vpn/$SERVER_ID/creategateway
sleep 3
echo "creating client configs..."
for ((a=1; a <= $NUM_CLIENTS; a++))
do
CLIENT_JSON=$( jq -n \
--arg clientid "vpnclient-$a" \
'{clientid: $clientid}' )
curl -s -o /dev/null -d "$CLIENT_JSON" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' https://api-base-domain.extenssion/api/extclients/vpn/$SERVER_ID
sleep 2
done
echo "finished configuring vpn server."
echo ""
echo "To configure clients, perform the following steps:"
echo " 1. log into dashboard-base-domain.extenssion"
echo " 2. Navigate to \"EXTERNAL CLIENTS\" tab"
echo " 3. Download or scan a client config (vpnclient-x) to the appropriate device"
echo " 4. Follow the steps for your system to configure WireGuard on the appropriate device"
echo " 5. Create and delete clients as necessary. Changes to netmaker server settings require regenerating ext clients."
)}
if [ "${MESH_SETUP}" != "false" ]; then
setup_mesh
fi
if [ "${VPN_SETUP}" == "true" ]; then
setup_vpn
fi
echo ""
echo "Netmaker setup is now complete. You are ready to begin using Netmaker."
- On successful completion, user will be requested to login to https://dashboard-base-domain.extenssion/
Agent Installation
- Agent will have same files expect the
Netmaker
- Netmaker
#!/bin/bash
# -- Prerequists
yum install -y wireguard-tools net-tools jq
modprobe ip_tables
echo 'ip_tables' >> /etc/modules
# -- Install Netmaker Client
wget https://github.com/gravitl/netmaker/releases/download/v0.15.2/netclient-arm64 -O /usr/sbin/netclient
chmod +x /usr/sbin/netclient
netclient daemon &
# -- Joining Netmaker Master
netclient join -t <token>
Verification
- On successful installation, User will be able to view new network on
ifconfig
-
Ping
to different systems will be accessible. - Dashboard will show all nodes connected
References
💖 💪 🙅 🚩
Rahul Kiran Gaddam
Posted on September 22, 2022
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.