Grafana Setup

Overview¶

Grafana is deployed as a Podman container in the test/development environment and managed programmatically via HTTP API. This enables debugging and creating dashboard panels without manual UI interaction.

Note: This documentation focuses on concepts and patterns, not specific IP addresses. Examples use localhost:3000 for consistency.

Environment Setup¶

Local Grafana Container¶

Container Details: - Container name: grafana-infra or grafana-svc - Internal endpoint: http://127.0.0.1:3000 - Optional proxy: HTTPS access via Traefik reverse proxy - Admin password: ~/.secrets/grafana.admin.pass

API Authentication:

# Read admin password from file
admin:$(cat ~/.secrets/grafana.admin.pass)

Data Source Endpoints¶

InfluxDB: - Internal: http://localhost:8086 - Remote: Use WireGuard tunnel or direct hostname

Loki: - Internal: http://localhost:3100 - Remote: Use WireGuard tunnel or direct hostname

Useful Grafana API Endpoints¶

List All Dashboards¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/search?type=dash-db | python3 -m json.tool

Output:

[
  {
    "id": 1,
    "uid": "fail2ban",
    "title": "Fail2ban Activity",
    "url": "/d/fail2ban/fail2ban-activity",
    "type": "dash-db"
  }
]

List Data Sources¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/datasources | python3 -m json.tool

Output:

[
  {
    "id": 1,
    "uid": "influxdb-uid",
    "name": "InfluxDB",
    "type": "influxdb",
    "url": "http://localhost:8086",
    "isDefault": false
  },
  {
    "id": 2,
    "uid": "loki-uid",
    "name": "Loki",
    "type": "loki",
    "url": "http://localhost:3100",
    "isDefault": true
  }
]

Get Dashboard by UID¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/dashboards/uid/fail2ban | jq '.dashboard.title'

Output:

"Fail2ban Activity"

Get Organization Info¶

curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/org

Output:

{
  "id": 1,
  "name": "Main Org.",
  "address": {
    "address1": "",
    "address2": "",
    "city": "",
    "zipCode": "",
    "state": "",
    "country": ""
  }
}

Create Dashboard¶

curl -s -X POST \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  -H "Content-Type: application/json" \
  -d @dashboard.json \
  http://localhost:3000/api/dashboards/db

Response:

{
  "id": 5,
  "uid": "new-dashboard-uid",
  "url": "/d/new-dashboard-uid/dashboard-name",
  "status": "success",
  "version": 1,
  "slug": "dashboard-name"
}

Update Dashboard¶

curl -s -X POST \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  -H "Content-Type: application/json" \
  -d @dashboard-updated.json \
  http://localhost:3000/api/dashboards/db

Payload format:

{
  "dashboard": { ... },
  "message": "Updated fail2ban pie chart",
  "overwrite": true
}

Delete Dashboard¶

curl -s -X DELETE \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/dashboards/uid/DASHBOARD_UID

Connection Verification¶

Test that Grafana and data sources are accessible before creating or modifying dashboards.

Verify Grafana API¶

# Health check
curl -I http://localhost:3000/api/health

# Expected output
HTTP/1.1 200 OK

Verify InfluxDB Connection¶

# From Grafana host
curl -I http://localhost:8086/health

# Expected output
HTTP/1.1 200 OK

Verify Loki Connection¶

# From Grafana host
curl -s http://localhost:3100/ready

# Expected output
ready

Test Data Source Query¶

InfluxDB:

curl -s -X POST http://localhost:8086/api/v2/query \
  -H "Authorization: Token YOUR_TOKEN" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "telegraf") |> range(start: -5m) |> limit(n: 1)'

Loki:

curl -s -G http://localhost:3100/loki/api/v1/query \
  --data-urlencode 'query={hostname="ispconfig3"}' \
  --data-urlencode 'limit=1'

Python Helper Functions¶

Grafana API Client¶

#!/usr/bin/env python3
from pathlib import Path
import subprocess
import json

class GrafanaAPI:
    def __init__(self, url="http://localhost:3000"):
        self.url = url
        self.password = Path.home() / '.secrets' / 'grafana.admin.pass'
        self.auth = f"admin:{self.password.read_text().strip()}"

    def get_dashboards(self):
        """List all dashboards"""
        cmd = ['curl', '-s', '-u', self.auth,
               f'{self.url}/api/search?type=dash-db']
        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

    def get_dashboard(self, uid):
        """Get dashboard by UID"""
        cmd = ['curl', '-s', '-u', self.auth,
               f'{self.url}/api/dashboards/uid/{uid}']
        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

    def create_dashboard(self, dashboard_json, message="Created via API"):
        """Create or update dashboard"""
        payload = {
            "dashboard": dashboard_json,
            "message": message,
            "overwrite": True
        }

        # Write payload to temp file
        payload_file = Path('/tmp/grafana-payload.json')
        payload_file.write_text(json.dumps(payload, indent=2))

        cmd = ['curl', '-s', '-X', 'POST',
               '-u', self.auth,
               '-H', 'Content-Type: application/json',
               '-d', f'@{payload_file}',
               f'{self.url}/api/dashboards/db']

        result = subprocess.run(cmd, capture_output=True, text=True)
        return json.loads(result.stdout)

# Usage
api = GrafanaAPI()
dashboards = api.get_dashboards()
for d in dashboards:
    print(f"{d['uid']}: {d['title']}")

Data Source Tester¶

#!/usr/bin/env python3
import subprocess
import json

def test_loki_query(query, limit=5):
    """Test Loki query and return results"""
    cmd = [
        'curl', '-s', '-G', 'http://localhost:3100/loki/api/v1/query',
        '--data-urlencode', f'query={query}',
        '--data-urlencode', f'limit={limit}'
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)
    data = json.loads(result.stdout)

    if data['status'] == 'success' and data['data']['result']:
        print(f"✅ Query works! {len(data['data']['result'])} results")
        return data['data']['result']
    else:
        print(f"❌ Query failed: {data.get('error', 'no results')}")
        return None

# Usage
test_loki_query('{service_type="fail2ban"}')

Deployment Considerations¶

Container vs. Production¶

Development/Test (Podman container): - Grafana runs as local container - Data sources on localhost or via WireGuard - Admin password in ~/.secrets/ - HTTP only (no TLS)

Production (Optional proxy): - Same Grafana container - Optional Traefik reverse proxy for HTTPS - DNS name points to proxy - TLS certificates managed by Traefik

Key Point: The API access pattern is the same - always use http://localhost:3000 for API calls, regardless of external access method.

Data Source Configuration¶

Local Development:

# All services on localhost
influxdb_url: "http://localhost:8086"
loki_url: "http://localhost:3100"

Distributed Deployment:

# Remote data sources via WireGuard
influxdb_url: "http://10.10.0.11:8086"  # monitor11 via VPN
loki_url: "http://10.10.0.11:3100"

Concepts Apply to Both: - Token-based authentication - HTTP API access - Dashboard JSON structure - Query syntax (Flux, LogQL)

Reference¶

For programmatic dashboard creation: - See CLAUDE.md "Creating Grafana Dashboards Programmatically" - See CLAUDE.md "Grafana Dashboard Development Workflow"

For troubleshooting blank graphs: - See CLAUDE.md "Troubleshooting Blank Grafana Graphs"