Troubleshooting Blank Graphs

Problem: Dashboard panels show data in query results but display blank graphs. This is a recurring issue.

Quick Diagnosis Tool¶

/tmp/grafana-blank-graph-troubleshoot.py <dashboard_uid> [panel_id]

# Examples:
/tmp/grafana-blank-graph-troubleshoot.py e0c1fce5-6583-459a-91c6-ad19bc708247  # All panels
/tmp/grafana-blank-graph-troubleshoot.py fail2ban 12  # Specific panel

Most Common Causes (Priority Order)¶

1. Browser Cache (Fix First)¶

Symptom: Graphs blank after dashboard updates

Fix: Hard refresh browser

Ctrl+Shift+R (Windows/Linux)
Cmd+Shift+R (Mac)

2. Dashboard Time Range Mismatch¶

Symptom: "No data" or blank graphs

Fix: Change dashboard time range to Last 6h, Last 24h, or Last 7d

3. Pie Chart Shows Single Value¶

Symptom: Pie chart displays "Value #A" or single aggregated value instead of multiple segments

Root Cause: reduceOptions.values = False collapses all series into one

Fix:

panel['options']['reduceOptions'] = {
    'values': True,  # Show multiple values
    'calcs': []      # Empty to show all
}

4. Table Shows Labels But No Values¶

Symptom: Table displays only label names (e.g., jail names) without associated values

Root Cause: Missing labelsToFields transformation for Loki/Prometheus instant queries

Fix:

panel['transformations'] = [
    {
        "id": "labelsToFields",
        "options": {"mode": "columns", "valueLabel": "Value"}
    },
    {
        "id": "organize",
        "options": {
            "indexByName": {"hostname": 0, "jail": 1, "Value": 2},
            "renameByName": {"Value": "Count"}
        }
    }
]

5. Missing Dashboard Variables¶

Symptom: Panels using ${variable} show blank

Diagnosis: Troubleshooter reports missing variables or no default values

Fix: Add variable or set default value in dashboard JSON

6. Hardcoded Datasource¶

Symptom: Panels work in one environment but not another

Fix: Change datasource to ${datasource} instead of hardcoded UID

7. Invisible Lines (lineWidth = 0)¶

Symptom: Graph area visible but no lines

Fix: Set lineWidth: 1 in panel fieldConfig

Systematic Debugging Process¶

Run Troubleshooter: Use the diagnosis tool to identify issues
Hard Refresh Browser: Ctrl+Shift+R (fixes 60% of cases)
Check Time Range: Switch to Last 24h
Verify Queries Return Data: Test queries directly against data sources
Fix Configuration Issues: Address issues reported by troubleshooter
Verify Fix: Hard refresh and check multiple time ranges

Dashboard Debug/Fix Workflow¶

Step 1: Identify the Problem¶

User reports "no data" or incorrect data in specific panels
Note panel names/titles

Step 2: Fetch Dashboard JSON¶

# Get dashboard by UID
curl -s -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  http://localhost:3000/api/dashboards/uid/fail2ban > /tmp/dashboard.json

# List all panels
python3 << 'EOF'
import json
with open('/tmp/dashboard.json', 'r') as f:
    d = json.load(f)
panels = d['dashboard']['panels']
for p in panels:
    print(f"Panel {p['id']}: {p.get('title', 'No title')}")
    print(f"  Query: {p['targets'][0].get('expr', 'EMPTY')[:80]}")
EOF

Step 3: Test Queries Directly Against Loki¶

# Test a Loki query BEFORE deploying to dashboard
import subprocess, json, time

now_ns = int(time.time() * 1e9)
query = 'sum by(jail) (count_over_time({service_type="fail2ban"} [24h]))'
cmd = f'curl -s -G "http://monitor11.example.com:3100/loki/api/v1/query" \
  --data-urlencode \'query={query}\' \
  --data-urlencode time={now_ns}'

result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
data = json.loads(result.stdout)

if data['status'] == 'success' and data['data']['result']:
    print(f"✅ Query works! {len(data['data']['result'])} results")
else:
    print(f"❌ Query failed: {data.get('error', 'unknown')}")

Step 4: Fix Dashboard JSON¶

import json

# Load dashboard
with open('/tmp/dashboard.json', 'r') as f:
    d = json.load(f)
dashboard = d['dashboard']

# Fix specific panel
for panel in dashboard['panels']:
    if panel['id'] == 10:  # Panel ID
        # Update query
        panel['targets'][0]['expr'] = 'your_tested_query_here'
        panel['targets'][0]['queryType'] = 'instant'  # or 'range'
        print(f"✅ Updated panel {panel['id']}")

# Save
with open('/tmp/dashboard-fixed.json', 'w') as f:
    json.dump(dashboard, f, indent=2)

Step 5: Deploy to Grafana¶

import json, subprocess

with open('/tmp/dashboard-fixed.json', 'r') as f:
    dashboard = json.load(f)

payload = {
    "dashboard": dashboard,
    "message": "Fix panel X: description of change",
    "overwrite": True
}

with open('/tmp/payload.json', 'w') as f:
    json.dump(payload, f)

cmd = 'curl -s -X POST -H "Content-Type: application/json" \
  -u admin:$(cat ~/.secrets/grafana.admin.pass) \
  -d @/tmp/payload.json \
  http://localhost:3000/api/dashboards/db'

result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
response = json.loads(result.stdout)
print(f"Status: {response.get('status')}")

Step 6: Verify in Browser¶

Hard refresh: Ctrl+Shift+R (or Cmd+Shift+R)
Check that data appears correctly

Troubleshooting Tips¶

Check label availability: curl http://monitor11.example.com:3100/loki/api/v1/labels
Check label values: curl http://monitor11.example.com:3100/loki/api/v1/label/LABELNAME/values
Test basic query: Start with {service_type="fail2ban"} before adding filters
Compare old vs new data: Use count_over_time() to see which source has data
Dashboard variables: Use $hostname, $jail in queries to enable filtering
Query type matters: Tables need instant, graphs need range

Real-World Fixes¶

Alloy Dashboard (2026-01-03): - Problem: Resource Usage and Processing Details panels blank - Root Cause: Browser cache - Fix: Hard refresh (Ctrl+Shift+R)

Fail2ban Pie Chart (2026-01-03): - Problem: Showing "Value #A: 106" instead of jail names - Root Cause: reduceOptions.values = False - Fix: Set values: True to show multiple segments

Fail2ban Table (2026-01-03): - Problem: Showing jail names but no ban counts - Root Cause: Missing labelsToFields transformation - Fix: Added transformation to convert labels to table columns