Application Logs¶

Overview¶

Application logs are collected by Alloy from various sources and sent to Loki. The solti-monitoring collection supports multiple application log types from reference deployments.

Supported Log Sources¶

Fail2ban Logs¶

Purpose: Security monitoring - track banned/unbanned IPs

Source: systemd journal (journald)

Configuration:

alloy_config_sources:
  - fail2ban

Alloy configuration:

loki.source.journal "fail2ban" {
  matches {
    _SYSTEMD_UNIT = "fail2ban.service"
  }

  labels = {
    service_type = "fail2ban",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

Log format:

[sshd] Ban 192.168.1.100
[sshd] Unban 192.168.1.100

LogQL query:

{service_type="fail2ban"}
| regexp `\[(?P<jail>[^\]]+)\]\s+(?P<action>Ban|Unban)\s+(?P<ip>\d+\.\d+\.\d+\.\d+)`
| action="Ban"

Apache Logs¶

Purpose: Web server access and error logging

Source: Log files

Configuration:

alloy_config_sources:
  - apache

Alloy configuration:

loki.source.file "apache_access" {
  targets = [
    {
      __path__ = "/var/log/apache2/access.log",
      service_type = "web",
      log_type = "access",
    },
  ]

  forward_to = [loki.write.default.receiver]
}

loki.source.file "apache_error" {
  targets = [
    {
      __path__ = "/var/log/apache2/error.log",
      service_type = "web",
      log_type = "error",
    },
  ]

  forward_to = [loki.write.default.receiver]
}

Log formats: - Access: Combined format (IP, user, timestamp, request, status, size, referrer, user-agent) - Error: Timestamp, level, client, message

LogQL queries:

# Find 500 errors
{service_type="web", log_type="access"} |~ "HTTP.*500"

# Count requests by status code
sum by(status) (count_over_time({service_type="web", log_type="access"} [1h]))

Bind9 (DNS) Logs¶

Purpose: DNS query monitoring

Source: systemd journal (journald)

Configuration:

alloy_config_sources:
  - bind9

Alloy configuration:

loki.source.journal "bind9" {
  matches {
    _SYSTEMD_UNIT = "named.service"
  }

  labels = {
    service_type = "dns",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

LogQL query:

{service_type="dns"} |= "query"

Mail Logs (Postfix)¶

Purpose: Email server monitoring

Source: systemd journal (journald)

Configuration:

alloy_config_sources:
  - mail

Alloy configuration:

loki.source.journal "postfix" {
  matches {
    _SYSTEMD_UNIT = "postfix.service"
  }

  labels = {
    service_type = "mail",
    mail_service = "postfix",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

LogQL queries:

# Find sent messages
{service_type="mail"} |= "status=sent"

# Find bounced messages
{service_type="mail"} |= "status=bounced"

WireGuard Logs¶

Purpose: VPN connection monitoring

Source: systemd journal (journald)

Configuration:

alloy_config_sources:
  - wireguard

Alloy configuration:

loki.source.journal "wireguard" {
  matches {
    _SYSTEMD_UNIT = "wg-quick@wg0.service"
  }

  labels = {
    service_type = "vpn",
    vpn_type = "wireguard",
    hostname = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

Gitea Logs¶

Purpose: Git service monitoring

Source: Log files

Configuration:

alloy_config_sources:
  - gitea

Alloy configuration:

loki.source.file "gitea" {
  targets = [
    {
      __path__ = "/var/log/gitea/gitea.log",
      service_type = "git",
      application = "gitea",
    },
  ]

  forward_to = [loki.write.default.receiver]
}

Syslog (System Logs)¶

Purpose: General system logging

Source: systemd journal (journald)

Configuration:

alloy_config_sources:
  - syslog

Alloy configuration:

loki.source.journal "syslog" {
  matches {
    SYSLOG_IDENTIFIER = ".*"
  }

  labels = {
    service_type = "system",
    hostname     = env("HOSTNAME"),
  }

  forward_to = [loki.write.default.receiver]
}

Journald vs File-Based Logs¶

Journald Sources¶

Advantages: - Structured metadata (fields automatically available) - No log rotation issues - Consistent format across applications - Automatic enrichment (PID, UID, etc.)

Configuration pattern:

loki.source.journal "service_name" {
  matches {
    _SYSTEMD_UNIT = "service.service"
  }
  labels = { ... }
  forward_to = [loki.write.default.receiver]
}

File-Based Sources¶

Advantages: - Works for applications not using systemd - Can read custom log formats - Direct access to log files

Configuration pattern:

loki.source.file "log_name" {
  targets = [
    {
      __path__ = "/path/to/log.log",
      label1 = "value1",
    },
  ]
  forward_to = [loki.write.default.receiver]
}

Log Parsing¶

Regex Parsing¶

Extract fields from unstructured logs:

loki.process "parse_fail2ban" {
  forward_to = [loki.write.default.receiver]

  stage.regex {
    expression = "\[(?P<jail>[^\]]+)\]\s+(?P<action>Ban|Unban)\s+(?P<ip>\d+\.\d+\.\d+\.\d+)"
  }

  stage.labels {
    values = {
      jail = "",
      action = "",
      banned_ip = "ip",
    }
  }
}

JSON Parsing¶

For JSON-formatted logs:

loki.process "parse_json" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      level = "level",
      message = "msg",
    }
  }

  stage.labels {
    values = {
      level = "",
    }
  }
}

Multi-Line Logs¶

Handle logs that span multiple lines:

loki.source.file "multiline_logs" {
  targets = [...]

  // Aggregate lines starting with timestamp into single log entry
  stage.multiline {
    firstline = "^\d{4}-\d{2}-\d{2}"
    max_wait_time = "3s"
  }
}

Common LogQL Queries¶

Count logs by service type¶

sum by(service_type) (count_over_time({hostname="server1"} [24h]))

Find errors across all logs¶

{hostname="server1"} |~ "(?i)error|fail|exception"

Top IPs banned by fail2ban¶

topk(20, sum by(banned_ip) (
  count_over_time(
    {service_type="fail2ban"}
    | regexp `Ban\s+(?P<banned_ip>\d+\.\d+\.\d+\.\d+)` [24h]
  )
))

Web server status codes¶

{service_type="web", log_type="access"}
| regexp `HTTP.*\s(?P<status>\d{3})\s`
| status="500"

Reference Deployment¶

See Reference Deployments chapter for complete Alloy configuration: - ispconfig3.example.com monitors: Apache, Fail2ban, Gitea, Mail, Bind9, WireGuard