Monitoring and Alerting: Setting Up Prometheus + Grafana

Why Monitoring Is Essential

Without an effective monitoring system, you only learn about problems when a customer calls to complain. This means the issue already occurred and affected users before you discovered it. Proactive monitoring flips the equation and alerts you before a small problem becomes a disaster.

A professional monitoring system watches three core pillars: infrastructure (CPU, memory, storage), applications (response time, errors), and business metrics (request count, transactions). Ignoring any pillar leaves a dangerous blind spot.

Essential Server Metrics

CPU Metrics

• CPU utilization: Percentage of processing capacity in use. Alert when exceeding 80% for 5 minutes
• Load average: Number of processes waiting. Normal value is below the number of cores
• CPU steal: Important in VPS environments. If it exceeds 10%, the physical server is overloaded

Memory Metrics

• Memory used: Alert when exceeding 85%
• Swap used: Any swap usage indicates memory shortage. Immediate alert
• Memory available: More important than free memory because it includes reclaimable memory

Storage Metrics

• Space used: Alert at 80%, critical at 90%
• IOPS: Input/output operations per second
• Latency: Alert if exceeding 20ms
• Inodes used: Running out prevents creating new files even with available disk space

Network Metrics

• Bandwidth: Inbound and outbound in megabits
• Packet loss: Any loss above 1% requires investigation
• Active connections: Sudden spikes may indicate a DDoS attack

Setting Up Prometheus and Grafana

Step 1: Install Prometheus

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus /var/lib/prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
tar xvf prometheus-2.52.0.linux-amd64.tar.gz
sudo cp prometheus-2.52.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.52.0.linux-amd64/promtool /usr/local/bin/

Step 2: Configure Prometheus

# /etc/prometheus/prometheus.yml
global:
 scrape_interval: 15s
 evaluation_interval: 15s

rule_files:
 - "alerts.yml"

alerting:
 alertmanagers:
 - static_configs:
 - targets: ['localhost:9093']

scrape_configs:
 - job_name: 'node'
 static_configs:
 - targets: ['localhost:9100']

 - job_name: 'nginx'
 static_configs:
 - targets: ['localhost:9113']

Step 3: Install Node Exporter

Node Exporter collects server metrics and exposes them to Prometheus:

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz
tar xvf node_exporter-1.8.0.linux-amd64.tar.gz
sudo cp node_exporter-1.8.0.linux-amd64/node_exporter /usr/local/bin/

Step 4: Install Grafana

sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install grafana -y
sudo systemctl enable grafana-server && sudo systemctl start grafana-server

Grafana runs on port 3000. Access it via browser and add Prometheus as a data source.

Setting Up Alert Rules

Alerts File

# /etc/prometheus/alerts.yml
groups:
 - name: server_alerts
 rules:
 - alert: HighCPU
 expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
 for: 5m
 labels:
 severity: warning
 annotations:
 summary: "High CPU usage ({{ $value }}%)"

 - alert: HighMemory
 expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
 for: 3m
 labels:
 severity: warning
 annotations:
 summary: "High memory usage ({{ $value }}%)"

 - alert: DiskSpaceLow
 expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 80
 for: 5m
 labels:
 severity: warning
 annotations:
 summary: "Low disk space ({{ $value }}%)"

 - alert: DiskSpaceCritical
 expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 90
 for: 2m
 labels:
 severity: critical
 annotations:
 summary: "Critical disk space ({{ $value }}%)"

Configuring Alertmanager

Alertmanager receives alerts from Prometheus and routes them to appropriate channels:

# /etc/alertmanager/alertmanager.yml
global:
 resolve_timeout: 5m

route:
 receiver: 'default'
 group_wait: 30s
 group_interval: 5m
 repeat_interval: 4h

 routes:
 - match:
 severity: critical
 receiver: 'critical-alerts'
 repeat_interval: 1h

receivers:
 - name: 'default'
 email_configs:
 - to: 'ops@example.com'

 - name: 'critical-alerts'
 email_configs:
 - to: 'ops@example.com'
 webhook_configs:
 - url: 'https://hooks.slack.com/services/xxx'

Alert Channels

Channel	Use Case	Response Time
Email	Low-severity alerts	Minutes to hours
Slack/Teams	Medium-severity alerts	Minutes
SMS	Critical alerts	Seconds
PagerDuty	Automatic escalation with on-call rotations	Seconds
Webhook	Integration with custom systems	Immediate

Essential Grafana Dashboards

Overview Dashboard

• CPU usage per server (line chart)
• Memory usage (gauge)
• Storage space (bar chart)
• Network traffic (dual line chart)
• Uptime per service

Application Dashboard

• Requests per second
• Response time (p50, p95, p99)
• Error rate (4xx, 5xx)
• Active connections

Additional Monitoring Tools

Tool	Specialty	Type
Uptime Kuma	External availability monitoring	Open source
Blackbox Exporter	HTTP/TCP/ICMP probing	Open source
Loki	Log aggregation and search	Open source
Netdata	Real-time monitoring with ready UI	Open source

Common Mistakes

1. Too many alerts: Alert fatigue leads to ignoring them. Set thresholds carefully
2. Not testing alerts: An untested alert may not fire when needed
3. Monitoring metrics only: Logs and traces are equally important
4. No response documentation: Every alert should link to a clear action plan

Conclusion

A good monitoring system transforms your team from reactive to proactive. Start with Prometheus, Grafana, and Node Exporter as your foundation, then add more as needed. On CLOUD (cloud.net.sa) servers, you get consistent performance with very low latency and SLA عالي مع ضمان تعاقدي, but monitoring from your side remains essential for tracking your applications and services.

---

Get Started

Want to try it yourself? Choose the right hosting plan or build your custom server with the specs you need.