Wazuh, as a Security Information and Event Management (SIEM) tool, is a critical component in securing Kubernetes clusters. It excels in real-time log data analysis, providing early threat detection and facilitating swift incident response. Wazuh’s ability to gather and interpret data from the Kubernetes environment allows security teams to identify and respond to threats promptly, minimizing potential damage. In addition to threat management, Wazuh aids in demonstrating compliance with industry-specific security standards by generating detailed reports on security events and measures. Post-incident, it offers valuable data for forensic analysis, enabling teams to understand the incident’s cause and prevent future occurrences. In this blog post we will go through the basic process of creating Alert for Kubernetes Events with Wazuh, especially for RKE2 cluster. **Steps**: 1. Create a webhook listener on the Wazuh server to receive logs from the Kubernetes cluster.(This process is same for all despite any kind of deployment tool for managing kubernetes cluster) 2. Enable auditing on the Kubernetes cluster and configure it to forward audit logs to the Wazuh webhook listener. (Differs in RKE2 from kubeadm and minikube) 3. Create rules on the Wazuh server to alert about audit events received from Kubernetes. **Requirements:** - A Wazuh server - A self managed kubernetes cluster ## Configure the Wazuh Server: We create a webhook listener on the Wazuh server to receive the Kubernetes audit logs. Here we create certificates for communication between the Wazuh server and Kubernetes. - Login to your Wazuh server: ```bash ssh user@wazuhServer ``` - Create a directory for the webhook endpoint: ```bash mkdir -p /var/ossec/integrations/kubernetes-webhook/ ``` - Create a certificate configuration file `/var/ossec/integrations/kubernetes-webhook/csr.conf`: Replace `` and `` with your server’s IP address. ```bash [ req ] prompt = no default_bits = 2048 default_md = sha256 distinguished_name = req_distinguished_name x509_extensions = v3_req [req_distinguished_name] C = US ST = California L = San Jose O = Wazuh OU = Research and development emailAddress = info@wazuh.com CN = [ v3_req ] authorityKeyIdentifier=keyid,issuer basicConstraints = CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment subjectAltName = @alt_names [alt_names] IP.1 = ``` - Create the root CA public and private keys: ```bash openssl req -x509 -new -nodes -newkey rsa:2048 -keyout /var/ossec/integrations/kubernetes-webhook/rootCA.key -out /var/ossec/integrations/kubernetes-webhook/rootCA.pem -batch -subj "/C=US/ST=California/L=San Jose/O=Wazuh" ``` - Create the certificate signing request (csr) and the server private key ```bash openssl req -new -nodes -newkey rsa:2048 -keyout /var/ossec/integrations/kubernetes-webhook/server.key -out /var/ossec/integrations/kubernetes-webhook/server.csr -config /var/ossec/integrations/kubernetes-webhook/csr.conf ``` - Generate the server certificate: ```bash openssl x509 -req -in /var/ossec/integrations/kubernetes-webhook/server.csr -CA /var/ossec/integrations/kubernetes-webhook/rootCA.pem -CAkey /var/ossec/integrations/kubernetes-webhook/rootCA.key -CAcreateserial -out /var/ossec/integrations/kubernetes-webhook/server.crt -extfile /var/ossec/integrations/kubernetes-webhook/csr.conf -extensions v3_req ``` ## Create the webhook listener: - Install Flask using pip: ```bash /var/ossec/framework/python/bin/pip3 install flask ``` - Create the Python webhook listener `/var/ossec/integrations/custom-webhook.py`. Replace `` with your Wazuh server IP address: ```python #!/var/ossec/framework/python/bin/python3 import json from socket import socket, AF_UNIX, SOCK_DGRAM from flask import Flask, request PORT = 8080 CERT = '/var/ossec/integrations/kubernetes-webhook/server.crt' CERT_KEY = '/var/ossec/integrations/kubernetes-webhook/server.key' socket_addr = '/var/ossec/queue/sockets/queue' def send_event(msg): string = '1:k8s:{0}'.format(json.dumps(msg)) sock = socket(AF_UNIX, SOCK_DGRAM) sock.connect(socket_addr) sock.send(string.encode()) sock.close() return True app = Flask(__name__) context = (CERT, CERT_KEY) @app.route('/', methods=['POST']) def webhook(): if request.method == 'POST': if send_event(request.json): print("Request sent to Wazuh") else: print("Failed to send request to Wazuh") return "Webhook received!" if __name__ == '__main__': app.run(host='', port=PORT, ssl_context=context) ``` - Create a systemd service at `/lib/systemd/system/wazuh-webhook.service` : ```bash [Unit] Description=Wazuh webhook Wants=network-online.target After=network.target network-online.target [Service] ExecStart=/var/ossec/framework/python/bin/python3 /var/ossec/integrations/custom-webhook.py Restart=on-failure [Install] WantedBy=multi-user.target ``` - Start service: ```bash systemctl daemon-reload systemctl enable wazuh-webhook.service systemctl start wazuh-webhook.service ``` ## Note: 1. Enable access to port 8080 if a firewall is running on the Wazuh server. ## Configure the Kubernetes Cluster: ```bash Create log directory: mkdir -p -m 700 /var/lib/rancher/rke2/server/logs ``` - Create an audit policy at `/var/lib/rancher/rke2/server/audit.yaml` : ```yaml apiVersion: [audit.k8s.io/v1](http://audit.k8s.io/v1) kind: Policy rules: # Don’t log requests to the following API endpoints - level: None nonResourceURLs: - '/healthz*' - '/logs' - '/metrics' - '/swagger*' - '/version' # Limit requests containing tokens to Metadata level so the token is not included in the log - level: Metadata omitStages: - RequestReceived resources: - group: authentication.k8s.io resources: - tokenreviews # Extended audit of auth delegation - level: RequestResponse omitStages: - RequestReceived resources: - group: authorization.k8s.io resources: - subjectaccessreviews # Log changes to pods at RequestResponse level - level: RequestResponse omitStages: - RequestReceived resources: # core API group; add third-party API services and your API services if needed - group: '' resources: ['pods'] verbs: ['create', 'patch', 'update', 'delete'] # Log everything else at Metadata level - level: Metadata omitStages: - RequestReceived ``` - Replace `` and create the webhook configuration file `/var/lib/rancher/rke2/server/audit-webhook.yaml` : ```yaml apiVersion: v1 kind: Config preferences: {} clusters: - name: wazuh-webhook cluster: insecure-skip-tls-verify: true server: https://:8080 # kubeconfig files require a context. Provide one for the API server. current-context: webhook contexts: - context: cluster: wazuh-webhook user: kube-apiserver # Replace with name of API server if it’s different name: webhook ``` - Add the Kubeapi parameters to load the auditing and webhook configurations: Path for systemd service file for rke2-server.service: `/usr/lib/systemd/system/rke2-server.service` : ```bash ExecStart=/usr/local/bin/rke2 \ server \ '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log' \ '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/rke2/server/audit.yaml' \ '--kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/rke2/server/audit-webhook.yaml' \ '--kube-apiserver-arg=audit-webhook-batch-max-size=1' \ ``` ## Create detection rule on Wazuh Server: - Add the following rules to the Wazuh server at `/var/ossec/etc/rules/local_rules.xml` : ```xml k8s audit Kubernetes audit log. 110002 requestURI\":.+", \"verb\": \"create Kubernetes request to create resource 110002 requestURI\":.+", \"verb\": \"delete Kubernetes request to delete resource ``` - Restart wazuh-manager service: ```bash systemctl restart wazuh-manager ``` - Test the configuration Test the rules by creating and deleting a deployment on the Kubernetes cluster. - Run the following command on the Kubernetes master node to create a new deployment: ```bash kubectl create deployment hello-minikube --image=[k8s.gcr.io/echoserver:1.4](http://k8s.gcr.io/echoserver:1.4) ``` - Run the following command to delete the deployment: ```bash kubectl delete deployment hello-minikube ``` ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/we2q8hf4ayzk95ke0x16.jpg) ## Slack Alerting: The integrations are configured on the Wazuh manager `ossec.conf` file. You can find this file in the Wazuh installation folder `/var/ossec/etc/`. To configure an integration, add the following configuration within the `` section: ```xml slack SLACK_WEBHOOK_URL 10 110003 110004 json ``` - Restart the Wazuh manager to apply the changes: ```xml systemctl restart wazuh-manager ``` - Once the configuration is complete, alerts start showing in the selected channel: ![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bezkz9gg1z3cj2fc5nad.png) Special thanks to the awesome [documentation](https://wazuh.com/blog/auditing-kubernetes-with-wazuh/) from the Wazuh team. Where I got the first part of this blog post from.

shitalnyaupane

Wazuh-Audit Kubernetes Cluster [RKE2]

Join Our Newsletter. No Spam, Only the good stuff.