Monitoring is one of those things that seems complicated from the outside and is actually quite approachable once you just sit down and do it. This guide walks through setting up a real Prometheus and Grafana monitoring stack on a single Ubuntu 22.04 server, from nothing to working dashboards.
Before we start: a quick explanation of what these tools are and how they relate to each other.
Prometheus is a time-series database and metrics collection system. It works by "scraping" - periodically making HTTP requests to configured targets and storing the numeric values it gets back. Those targets are called "exporters."
Node Exporter is an exporter that exposes system metrics from the machine it runs on: CPU usage, memory, disk I/O, network traffic. It's the most common starting point.
Grafana is the visualisation layer. It connects to Prometheus (and many other data sources) and lets you build dashboards from the metrics it has stored.
The flow is: Node Exporter exposes metrics → Prometheus scrapes and stores them → Grafana queries Prometheus and displays them.
Step 1: Create a dedicated user for Prometheus
Never run monitoring daemons as root:
sudo useradd --no-create-home --shell /bin/false prometheus
sudo useradd --no-create-home --shell /bin/false node_exporterCreate directories Prometheus will use:
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheusStep 2: Download and install Prometheus
Find the latest version at https://prometheus.io/download/ - at time of writing it's 2.51.0. Replace the version below if a newer one is available:
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.51.0/prometheus-2.51.0.linux-amd64.tar.gz
tar xvf prometheus-2.51.0.linux-amd64.tar.gz
cd prometheus-2.51.0.linux-amd64Copy the binaries:
sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtoolCopy the default configuration files:
sudo cp -r consoles /etc/prometheus/
sudo cp -r console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus/consoles /etc/prometheus/console_librariesStep 3: Configure Prometheus
Create the main Prometheus config file:
sudo nano /etc/prometheus/prometheus.ymlPaste this configuration:
global:
scrape_interval: 15s
evaluation_interval: 15salerting:
alertmanagers:
- static_configs:
- targets: []
rule_files: []
scrapeconfigs:
- jobname: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- jobname: 'nodeexporter'
- static_configs:
- - targets: ['localhost:9100']
- ```
This tells Prometheus to scrape itself (port 9090) and Node Exporter (port 9100) every 15 seconds.
Set ownership:
sudo chown prometheus:prometheus /etc/prometheus/prometheus.ymlStep 4: Create a systemd service for Prometheus
sudo nano /etc/systemd/system/prometheus.servicePaste:
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--storage.tsdb.retention.time=30d
Restart=always
[Install]
WantedBy=multi-user.target
```
--storage.tsdb.retention.time=30d keeps 30 days of metrics. Adjust based on your available disk space - 30 days of typical system metrics is around 1-2GB per host.
Enable and start:
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheusYou should see active (running). If not, check logs:
sudo journalctl -u prometheus -fPrometheus is now running at http://your-server-ip:9090.
Step 5: Install Node Exporter
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporterCreate the systemd service:
sudo nano /etc/systemd/system/node_exporter.service[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target[Service]
User=nodeexporter
Group=nodeexporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
[Install]
WantedBy=multi-user.target
```
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporterTest that Node Exporter is exposing metrics:
curl http://localhost:9100/metrics | head -20You should see a wall of text that starts with # HELP lines describing available metrics.
Step 6: Verify Prometheus is scraping Node Exporter
Open the Prometheus UI at http://your-server-ip:9090. Go to Status > Targets.
You should see two targets: prometheus and node_exporter, both with state UP. If Node Exporter shows DOWN, check that it's running and that the port is correct.
Try a quick query in the Prometheus UI. In the search box, type:
node_cpu_seconds_totalPress Execute. You should see a table of values. This is CPU usage broken down by mode (idle, user, system, etc.).
Step 7: Install Grafana
sudo apt install -y software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana -yEnable and start:
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-serverGrafana is now running at http://your-server-ip:3000.
Step 8: Configure Grafana
Open the Grafana UI in your browser. Default credentials are admin / admin. You'll be prompted to change the password immediately - do it.
Add Prometheus as a data source:
- Click the gear icon (Configuration) in the left sidebar
- Click Data Sources
- Click Add data source
- Select Prometheus
- Set URL to
http://localhost:9090 - Click Save & Test
You should see "Data source is working."
Import the Node Exporter dashboard:
Someone has already built a comprehensive Node Exporter dashboard and published it to Grafana's dashboard library. No need to build your own from scratch.
- Click the + icon in the left sidebar
- Click Import
- Enter dashboard ID
1860in the "Import via grafana.com" field - Click Load
- Select your Prometheus data source from the dropdown
- Click Import
You now have a full dashboard showing CPU, memory, disk, network, and system load for your server.
Step 9: Set up a firewall rule
By default, Prometheus (9090), Node Exporter (9100), and Grafana (3000) are accessible from any IP. In production, restrict this.
Prometheus and Node Exporter should not be publicly accessible - they expose sensitive system information. Restrict them to localhost or to your monitoring network:
# Block external access to Prometheus and Node Exporter
sudo ufw deny 9090
sudo ufw deny 9100# Allow Grafana only from specific IPs (replace with your IP)
sudo ufw allow from YOUR_IP to any port 3000
```
Or put Grafana behind a reverse proxy (nginx or Caddy) with HTTPS and basic auth. That's the right production setup but beyond the scope of this post.
Useful PromQL queries to get started
Once everything is running, here are some queries worth knowing. You can run these in the Grafana query editor or directly in the Prometheus UI:
# CPU usage percentage (non-idle CPU time)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)# Available memory in GB
nodememoryMemAvailable_bytes / 1024 / 1024 / 1024
# Disk usage percentage
100 - ((nodefilesystemavailbytes{mountpoint="/"} / nodefilesystemsizebytes{mountpoint="/"}) * 100)
# Network traffic in/out (bytes per second)
rate(nodenetworkreceivebytestotal{device="eth0"}[5m])
rate(nodenetworktransmitbytestotal{device="eth0"}[5m])
```
---
From here, the natural next steps are: adding more exporters for specific services (there are exporters for PostgreSQL, Redis, Nginx, and most other common tools), setting up Alertmanager to send alerts when things go wrong, and adding more servers to the scrape config. Each of those could be its own post and probably will be.
But this is a working stack. Real metrics, real dashboards, real data. That's the foundation everything else builds on.
