.:: Marcos Dione/StyXman's glob ::. (Posts about grafana)

Monitoring Apache with SQL and Grafana

Marcos Dione — Tue, 12 May 2026 18:07:46 GMT

Ever since my last job I have been wanting to make this. I think it's not the first time I do it, but for one reason or another, I did it (again?) in two evenings only.

In that job we had an internet facing API with Apache as the router in front of several services. All our metrics and even our billing was based on the Apache logs. We had a system that ingested the logs into a PostgreSQL database, and we tried to create Grafana panels and alerts based on that info. At the same time, I wanted to reproduce awstats in Grafana, and found it was almost impossible.

Another problem is that the usual tools to solve this, Loki or Prometheus, have big problems to handle this type of too arbitrary data (think of the referer or user_agent columns) or whose space is too big (client is an IPv4, with 4Bi different values). They effectively suffer (in principle) of what they call "cardinality bomb": since they build one time series database (TSDB) per combination of fields (they call them "labels"), storage use is big and aggregation operations inter TSDBs are expensive.

Last night I sat down to reimplement the ingestion side. Instead of PostgreSQL I used SQLite mostly because almost all of my services (with low traffic and mostly only me as user) already use it. To be fair, and one really can't expect anything else, the script is quite straight forward. It uses regexps to parse the logs, which for the moment is good enough. I'm "releasing" it as is, because I'm tired, but you'll find some surprises around parsing the request line (see request_re and its handling); some janky ways to convert from str to int or datetime; and an iteration trick to use dataclasses as execute() argument. I omited some comments and all the testing:

#! /usr/bin/env python3

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone, tzinfo
import pathlib
import re
import sqlite3
import sys

# I miss dinant

# no 0-255 range check since this is written by apache
# if the number is not in that range, we have bigger problems
octet_re = r'\d{1,3}'
ip_re = r'\.'.join([ octet_re ] * 4)

word_re = r'[^ ]+'
identd_user_re = word_re  # it can be '-'
user_id_re = word_re      # it can be '-'

month_names = [ 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec' ]

day_re = r'\d{1,2}'
month_re = f"(?:{'|'.join(month_names)})"
year_re = r'\d{4}'
date_re = f"({day_re})/({month_re})/({year_re})"

time_re = r'(\d{2}):(\d{2}):(\d{2})'

utc_offset_re = r'(?:\+|\-)\d{4}'  # no capture

# fscking double escaping :(
date_time_re = f"\\[{date_re}:{time_re} ({utc_offset_re})\\]"

method_re = word_re
url_re = word_re  # technically not a word, but word_re is too generic

proto_re = r'HTTP'      # who are we kidding
version_re = r'\d\.\d'  # who are we kidding
proto_and_version_re = f"({proto_re})/({version_re})"

# idiot skrip kidz send no method or proto/version!
# and re is silly? enough to produce empty matches for the ()s here
# oh, but re.compile().match().groups() returns things like
# (None, None, None, None, '', '\\x16\\x03\\x02\\x01o\\x01', '', '')
# so we gained nothing
request_re = f'"(?:({method_re}) ({url_re}) {proto_and_version_re}|()({url_re})()())"'

number_re = r'\d+'

http_status_re = number_re
bytes_rx_re = number_re
bytes_tx_re = number_re
ttfb_re = f"(?:{number_re}|-)"
response_time_re = number_re

double_quoted_text_re = r'"([^"]+)"'
referer_re = double_quoted_text_re
user_agent_re = double_quoted_text_re

log_line_re = re.compile(f"^({ip_re}) ({identd_user_re}) ({user_id_re}) {date_time_re} {request_re} ({http_status_re}) ({bytes_rx_re}) ({bytes_tx_re}) ({ttfb_re}) ({response_time_re}) {referer_re} {user_agent_re}$")


@dataclass
class LogRecord:
    client_ip: str         # 0
    indent_user: str
    user_id: str
    date_time: datetime
    method: str
    url: str               # 5
    protocol: str
    protocol_version: str  # could be float, but we don't really care; besides, x.y.z?
    status: int
    bytes_rx: int
    bytes_tx: int          # 10
    ttfb: int  # maybe -!
    response_time: int
    referer: str
    user_agent: str

    @classmethod
    def from_log_line(cls, line):
        match = log_line_re.match(line)

        if match is None:
            raise ValueError(f"Malformed line: {line.strip()}")

        data = list(match.groups())
        new_data = []

        group_index = 0
        for field_index, (name, field) in enumerate(cls.__dataclass_fields__.items()):
            if field.type == datetime:
                # [11/May/2026:20:15:28 +0200]

                # convert month str to number
                data[group_index+1] = month_names.index(data[group_index+1]) + 1

                # convert to ints
                data[group_index:group_index+6] = [ int(x) for x in data[group_index:group_index+6] ]

                new_data.append( datetime(data[group_index+2], data[group_index+1], data[group_index],
                                          data[group_index+3], data[group_index+4], data[group_index+5], 0,
                                          utc_offset2tzinfo(data[group_index+6])) )

                group_index += 7

                continue

            # handle ttfb as -
            if field_index == 11 and data[group_index] == '-':
                # 
                data[group_index] = data[group_index+1]

            if data[group_index:group_index+4] == [ None, None, None, None ]:
                if group_index in (10, 14):
                    # handle (None, None, None, None, '', '\\x16\\x03\\x02\\x01o\\x01', '', '')
                    # handle ('GET', '/', 'HTTP', '1.0', None, None, None, None)

                    # no need to add anything, it's handled by the fallback
                    # but we still need to skip this cruft
                    group_index += 4
                else:
                    raise ValueError(f"Got confused: {(field_index, field.type, group_index, data[group_index], new_data)}")

            # convert ints
            if field.type == int:
                data[group_index] = int(data[group_index])

            # fallback
            new_data.append(data[group_index])
            group_index += 1

        return cls(*new_data)


    # implement the iterator protocol so we can mostly be passed as argument to execute()
    def __iter__(self):
        return self


    def __iter__(self):
        for value in self.__dict__.values():
            # the whole protocol could be replaced with .__dataclass_fields__.values() :shrug:
            # but this way I can do further conversions
            if type(value) == datetime:
                value = int(value.timestamp())

            yield value


def utc_offset2tzinfo(offset: str) -> tzinfo:
    # +0200
    hours = int(offset[:3])    # +02
    minutes = int(offset[3:])  # 00

    return timezone(timedelta(hours=hours, minutes=minutes), offset)


def connect():
    # if we test after sqlite3.connect(), the file is already created
    create = not pathlib.Path('./apache_logs.db').exists()

    conn = sqlite3.connect('./apache_logs.db')

    if create:
        conn.cursor().execute('''
CREATE TABLE "logs" (
    "client"    TEXT,
    "indent_user"   TEXT,
    "user_id"   TEXT,
    "timestamp" INTEGER,
    "method"    TEXT,
    "url"   TEXT,
    "protocol"  TEXT,
    "protocol_version"  TEXT,
    "status"    INTEGER,
    "bytes_rx"  INTEGER,
    "bytes_tx"  INTEGER,
    "ttfb"  INTEGER,
    "response_time" INTEGER,
    "referer"   TEXT,
    "user_agent"    TEXT
);''')

    return conn



def main():
    conn = connect()
    cursor = conn.cursor()

    for line in sys.stdin:
        try:
            log_record = LogRecord.from_log_line(line)
        except ValueError as e:
            print(e.args[0])
            continue

        cursor.execute('''INSERT INTO logs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)''',
                       tuple(log_record))
        conn.commit()


if __name__ == '__main__':
    main()

One of the things I didn't do was to further play with the URLs. One could make list of different apps based on whether there is any routing to different services, like in the cases of my previous job and my own server; or even different subdivisions on a single app, like for NextCloud:

ocs/v2.php/apps/serverinfo
remote.php/dav/files/USER
remote.php/dav/calendars/USER/CALENDAR/

etc. I haven't really thought about it; it could be implemented either as more columns or extra tables.

Today I managed to finish the rest.

The next step is to install this so it runs constantly with the output of tail --follow=name --retry piped into its stdin¹. Left as an exercise for the reader; use SystemD units :)

Next is installing Grafana's plugin to read SQLite and declare the new Grafana datasource.

The hard part was to query the data in a way that was useful for Grafana. I managed to get a query like:

-- round to the minute
SELECT timestamp/60*60 AS time, status, COUNT(status) as "count"
FROM logs
-- $__from and $__to are defined by Grafana based on the dashboards's time range
WHERE timestamp >= $__from / 1000 and timestamp < $__to / 1000
GROUP BY timestamp/60, status

to get the count of different status codes per minute². But this returns a table that looks like:

      time | status | count
1778533620 |    200 |    30
1778533620 |    207 |     3
1778533620 |    403 |     1

while Grafana is expecting one line per sample (but remember we're aggregating data) and one column per data series:

      time | 200 | 207 | 403
1778533620 |  30 |   3 |   1

I read how to pivot this in SQL, but it mostly works only if you know the different values for the status column beforehand. This might be feasible with HTTP status codes (I count 63 standard ones, including the joke 418 I'm a teapot), but that would be impossible for the referer or user_agent columns.

Thanks to iRobbery#postgresql@libera.chat I found about Grafana's Partition by values data transformation. Aplying it to the column that defines the time series (status, etc), it gives us exactly what we want!

And one can even include a pure table with all the logs to inspect when one finds weird spikes or values. I made almost impossible queries like transferred bytes per URL, methods per client and more! One missing piece, i possible, would be to implement histograms, like last time we looked into this.

One could cite the UNIX philosophy, but seriously, who wants to reimplement all the corner cases of that tail invocation? See for instance the 113 bugs found in the coreutils Rust reimplementaiton ↩
One could use a dashboard variable to control this arbitrarily. One could get granularity per second! ↩

Monitoring apache with prometheus and grafana

Marcos Dione — Sat, 18 Nov 2023 22:45:08 GMT

NE: this one is not going to be that coherent either, even when I'm actually writing it. Again, sorry for the mess.

Since nobody seems to have ever written a blog post about this beyond the basic 'explain + install + graph one metric in grafana', here's my take.

Analyzing Apache activity with tools like prometheus and grafana is not easy task, mostly because the kind of things that Apache can do is quite varied: it could be serving static files, of any size, receiving uploads, answering API calls, thin REST ones or big fat heavy ones, or even offloading SSL or just proxying to other services. So the first thing you have to ask yourself is: what does my Apache do?

In my case it's two things:

Serving static files, mostly map tiles and photos. These vary in size a lot: tiles are usually small, but I also have big 12MiB photos and at least 2 scaled down versions, the thumbnail and the gallery main image. These thumbnails and main image are generated from the originals and cached.
Serving a few services: webmail, rss reader and 'cloud' suit (file sharing, calendars, etc).

Second, you have to define what do you actually want to know about it. There are many options: What's my error rate? How long does it take to serve requests? What about queries whose payload can wildly vary in size? What about uploads?

You can also ask yourself about those values: Do I wan to know it in general, by any individual URL, by certain prefixes...?

These question should lead you to define certain things, including changes you might need to do to Apache. For instance, Apache's 'combined' log format include only status and written size (including headers), but neither read size, response time or time to first byte. TTFB is an important value to know for requests that either upload or download lots of data; for instance, in my case, the gallery's images. TTFB will tell you much time you spend before starting your answer. Even then, this value can be misleading, since there's nothing to prevent a framework or application to send headers as long as their values are calculated, but still have to much computation before actually sending the payload of the response.

# enable time to first byte
LogIOTrackTTFB ON

LogFormat "%h %l %u %t \"%r\" %>s %I %O %^FB %D \"%{Referer}i\" \"%{User-Agent}i\"" combined_io

So, where do we start about this? A good starting point is Brian Brazil's oldish post, where he mentions the Grok Exporter, but there is no version in Debian. This means, I need a binary release. Unfortunately, the GE seems to be abandoned by its original developer. The people at SysDigLabs maintain a fork, but they seem to have mostly focused on doing only maintenance around CI, dependencies, but no major code changes. None of the 11 pull requests or the 70 issues in the orignal repo seem to be addressed. I don't blame them, I'm just pointing it out. Besides, they're not providing binaries, except if you count a docker image as such.

So I decided to pick another fork. Of them all, a couple of them do provide binary releases, so I just picked one.

Meanwhile, I also thought that since I was already modifying Apache's log, I could do it in such a way that it would write CSV instead and use the CVS exporter. But since the GE is already there, some people seem to be using it, and this other one for some reason mentions SSH, I decided to stick to GE.

Ok, we install the GE, enable and start the systemd service. This involves extracting the zip, putting the binary and the systemd service unit file somewhere, call systemd link so it sees the new service, write the configuration file and place the pattern files you want in one of the pattern directories declare in the config file. Now what? Well, you can say this is the fun part.

I decided to cut my monitoring by service. First to monitor was my raster tile based map. I butchered the original patterns for apache, splitting it into a prefix, a middle part that will match the path in the request, and a postfix that also adds the three new fields. This way I can write a middle section for each service. Things could be different if I was serving each one a different vhost. It also allowed me to capture parts of the path for instance to count served map tiles per zoom level. This was easy once I figured I had to do the last step of the previous paragraph, otherwise you get errors like Pattern %{IP:client_ip} not defined.

HTTPD_LOG_PREFIX %{IP:client_ip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\]
HTTPD_LOG_POSTFIX %{NUMBER:status} (?:%{NUMBER:read}|-) (?:%{NUMBER:wrote}|-) (?:%{NUMBER:first_byte}|-) (?:%{NUMBER:total_time}|-) %{QS:referrer} %{QS:agent}

ELEVATION_LOG %{HTTPD_LOG_PREFIX} "(?:%{WORD:verb} /~mdione/Elevation/%{NUMBER:zoom_level}/%{NOTSPACE}(?: HTTP/%{NUMBER:http_version})?|%{DATA:rawrequest})" %{HTTPD_LOG_POSTFIX}

Have you noticed Apache's log have this ugly format for timestamps? 18/Nov/2023:23:44:13 +0100. Luckily we don't really needed them, otherwise this would be a nighmare just to parse and convert to something we can use.

Now it's time to define the metrics. I wrote one metrics file per service. Since this is a static site, I built histograms for response time and response size. Just for fun, I also wrote one for FTTB. This is quite straightforward, except for one thing: defining the buckets. What I did was use awk to get the field I wanted, then sort -n and less -N. I was looking at around 1000 requests, so I made buckets of around 100 requests. This value will not be so easy to write for more wildly variant services. For instance, at $WORK we receive and send documents or bundles of documents to be signed. The size of these PDFs and bundles of them are virtually unlimited, and in fact, we're starting to experience issues where the servers freeze for up to 10m just chewing and digesting big uploads.

- type: histogram
  name: elevation_tile_ttfb
  help: Elevation tile Time-To-First-Byte
  match: '%{ELEVATION_LOG}'
  value: '{{multiply .first_byte 0.001}}'  # µs to ms
  buckets: [ 1, 10, 20, 30, 40, 50, 75, 100, 150 ]

- type: counter
  name: elevation_tile_zoom_level_count
  help: Counts tiles per zoom level
  match: '%{ELEVATION_LOG}'
  labels:
    zoom_level: '{{.zoom_level}}'

Last step of the chain, and quite frankly, the one that took me more time: having a nice graph in grafana. My grafana version is somewhat old (6.7.2 v 10.2.1!¹), so my only option is a heatmap panel. One thing to know about the grok exporter's metrics is that they're counters, so the first thing you have to do is to use increase() to get the derivative. I took me quite a long time to find out this (no, I can't say I figure it out, I read it here). Second, Grafana by default assumes it has to massage the data to build the heat map, so you have to tell it that the data is already a histogram. In the same vein, you also have to tell it a second time you want a heatmap, this time as the query's format. I also hid zero values and made the tooltip a histogram. Play around with the rest of the settings.

{
  "type": "heatmap",
  "targets": [
    {
      "expr": "increase(gallery_ttfb_bucket[1m])",
      "legendFormat": "{{le}}",
  "format": "heatmap"
}
  ],
  "heatmap": {},
  "dataFormat": "tsbuckets",
  "yAxis": {
    "format": "ms",
  },
  "tooltip": {
    "show": true,
    "showHistogram": true
  },
  "hideZeroBuckets": true
}

Here's the Gallery's dashboard:

It's me traversing my gallery. The first part is me getting into a directory that has never been seen, so the gallery had to generate all the thumbnails. That's why we're seeing TTFBs and RTs so high, but the sizes stay low, because they're all thumbnails. Then it's me poking around here and there, most of the time seing the thumbnail table, but sometimes navigating the photos one by one in their big version. No original version was downloaded. I think I'll have to let these run for a while an revisit the buckets. The one for the response size already needs an adjustment. Also notice that my home server is quite boring, I have to do synthetic usage just to have values to graph.

One thing to notice here. Not once in this post I mention the word latency, even when it's one of the buzzwords around web services. About that I strongly suggest you to watch Gil Tene's How NOT to measure latency talk. It's an oldie, but man it packs a lot of info, including demystifying quantiles and nines.

Update

Today I noticed that an anternative to the Grok Exporter is mtail. It has the advantages that it's well maintained and already available in Debian, but on the other hand you have to program it. I'll stick to GE until a reason raises to switch to it.

Also:

Slight additions and clarifications.
Fixed typos and removed dupes.

This was because I was using grafana from Debian and not their repository. ↩