Observing and managing the performance of a Kubernetes cluster is crucial for maintaining application health, identifying issues, and ensuring high availability. I'll walk you through setting up a comprehensive monitoring solution using kubectl and Helm, deploying Grafana, Loki, and Prometheus to your cluster, and setting up Ingress for external access.

We will cover the following key steps:

1. Applying a Kubernetes namespace for monitoring.
2. Installing Helm and setting up the necessary repositories.
3. Deploying Loki, Prometheus, and Grafana using Helm.
4. Applying Ingress rules to expose the services externally.

Prerequisites
Before you begin, make sure you have the following:

1. A Kubernetes cluster with kubectl configured.
2. Helm installed on your local machine.
3. Proper access to apply YAML configurations and install charts.

Step 1: Create a Monitoring Namespace

Namespaces help you logically divide and organize your Kubernetes resources. To avoid conflicts and keep monitoring resources separate, we’ll create a dedicated namespace for monitoring tools.

1. Apply the monitoring namespace: Save the following content into a monitoring-namespace.yml file:

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  labels:
    app.kubernetes.io/name: monitoring
    app.kubernetes.io/instance: monitoring

2. Apply the namespace using kubectl:

kubectl apply -f monitoring-namespace.yml

This command creates a new namespace called monitoring in your Kubernetes cluster.

Step 2: Install Helm and Add the Grafana Repository
Helm, the Kubernetes package manager, makes it easier to deploy complex applications like Grafana, Loki, and Prometheus. Here’s how to install Helm and set up the necessary repository.

1. Install Helm (if it's not installed already):

sudo snap install helm --classic

This command installs Helm using Snap, a package management system for Linux.

1. Add the Grafana Helm chart repository:

helm repo add grafana https://grafana.github.io/helm-charts

1. Update the Helm repositories:

helm repo update

This ensures that Helm has the latest charts from the Grafana repository.

Step 3: Deploy Loki, Prometheus, and Grafana with Helm

Now that Helm is installed and configured, we’ll deploy Loki, Prometheus, and Grafana using the Grafana Helm chart.

1. Run the Helm installation command:

helm upgrade --install loki --namespace=monitoring grafana/loki-stack \
--set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=gp2,loki.persistence.size=100Gi --set nodeSelector.name=node.kubernetes.io/description=all_production

A: Grafana: **This enables Grafana, the monitoring dashboard tool, within the Helm chart.
**B: Prometheus: **Prometheus is enabled for collecting metrics, while persistent volumes for Alertmanager and Prometheus server are disabled to simplify storage configuration.
**C: Loki: Loki, the log aggregation tool, is enabled with persistent volume storage of 100Gi using the gp2 storage class.

The --install flag ensures that the stack is installed if it hasn’t been deployed previously. The --upgrade flag updates the stack to the latest version if it is already installed.

2. Verify the Installation: After a successful installation, check the status of the pods running in the monitoring namespace:

kubectl get pods -n monitoring

You should see pods for Grafana, Prometheus, Loki, and Promtail (which ships logs to Loki).

Step 4: Set Up Ingress for External Access

To access Grafana, Prometheus, or Loki from outside your cluster, you need to configure an Ingress resource. This allows external HTTP/S access to the monitoring services.

*1. Create an Ingress Resource: Save the following example to a monitoring-ingress.yml file:
*

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: monitoring-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
    - hosts:  
        - your-domain.com
      secretName: certname
  rules: 
    - host: your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 80 

This Ingress configuration sets up routing for Grafana, Prometheus, and Loki under the domain name {your-domain.com}. You’ll need to replace your-domain.com with your actual domain and configure DNS to point to your cluster’s external IP.

2. Apply the Ingress Resource:

kubectl apply -f monitoring-ingress.yml

Once applied, the Ingress controller will route traffic to the appropriate services based on the hostname.

1. Verify Ingress Setup: Check the status of your Ingress resource to ensure it was configured properly:

kubectl get ingress -n monitoring

Ensure that your ADDRESS column has an external IP for the services to be accessible externally.

Step 5: Access the Monitoring Dashboard

Once the Ingress is properly configured and DNS is pointing to your cluster, you can access Grafana, Prometheus, and Loki.

Conclusion

With just a few commands, you’ve successfully deployed a complete monitoring stack using Helm on your Kubernetes cluster. By leveraging Helm charts, you simplify the deployment of complex applications like Grafana, Loki, and Prometheus while also integrating Ingress for easy access. You now have a powerful observability setup that allows you to monitor logs and metrics in real-time, helping you manage and optimize your Kubernetes applications more effectively.