.:: Marcos Dione/StyXman's glob ::. (Posts about monitoring)

Monitoring Apache with SQL and Grafana

Marcos Dione — Tue, 12 May 2026 18:07:46 GMT

Ever since my last job I have been wanting to make this. I think it's not the first time I do it, but for one reason or another, I did it (again?) in two evenings only.

In that job we had an internet facing API with Apache as the router in front of several services. All our metrics and even our billing was based on the Apache logs. We had a system that ingested the logs into a PostgreSQL database, and we tried to create Grafana panels and alerts based on that info. At the same time, I wanted to reproduce awstats in Grafana, and found it was almost impossible.

Another problem is that the usual tools to solve this, Loki or Prometheus, have big problems to handle this type of too arbitrary data (think of the referer or user_agent columns) or whose space is too big (client is an IPv4, with 4Bi different values). They effectively suffer (in principle) of what they call "cardinality bomb": since they build one time series database (TSDB) per combination of fields (they call them "labels"), storage use is big and aggregation operations inter TSDBs are expensive.

Last night I sat down to reimplement the ingestion side. Instead of PostgreSQL I used SQLite mostly because almost all of my services (with low traffic and mostly only me as user) already use it. To be fair, and one really can't expect anything else, the script is quite straight forward. It uses regexps to parse the logs, which for the moment is good enough. I'm "releasing" it as is, because I'm tired, but you'll find some surprises around parsing the request line (see request_re and its handling); some janky ways to convert from str to int or datetime; and an iteration trick to use dataclasses as execute() argument. I omited some comments and all the testing:

#! /usr/bin/env python3

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone, tzinfo
import pathlib
import re
import sqlite3
import sys

# I miss dinant

# no 0-255 range check since this is written by apache
# if the number is not in that range, we have bigger problems
octet_re = r'\d{1,3}'
ip_re = r'\.'.join([ octet_re ] * 4)

word_re = r'[^ ]+'
identd_user_re = word_re  # it can be '-'
user_id_re = word_re      # it can be '-'

month_names = [ 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec' ]

day_re = r'\d{1,2}'
month_re = f"(?:{'|'.join(month_names)})"
year_re = r'\d{4}'
date_re = f"({day_re})/({month_re})/({year_re})"

time_re = r'(\d{2}):(\d{2}):(\d{2})'

utc_offset_re = r'(?:\+|\-)\d{4}'  # no capture

# fscking double escaping :(
date_time_re = f"\\[{date_re}:{time_re} ({utc_offset_re})\\]"

method_re = word_re
url_re = word_re  # technically not a word, but word_re is too generic

proto_re = r'HTTP'      # who are we kidding
version_re = r'\d\.\d'  # who are we kidding
proto_and_version_re = f"({proto_re})/({version_re})"

# idiot skrip kidz send no method or proto/version!
# and re is silly? enough to produce empty matches for the ()s here
# oh, but re.compile().match().groups() returns things like
# (None, None, None, None, '', '\\x16\\x03\\x02\\x01o\\x01', '', '')
# so we gained nothing
request_re = f'"(?:({method_re}) ({url_re}) {proto_and_version_re}|()({url_re})()())"'

number_re = r'\d+'

http_status_re = number_re
bytes_rx_re = number_re
bytes_tx_re = number_re
ttfb_re = f"(?:{number_re}|-)"
response_time_re = number_re

double_quoted_text_re = r'"([^"]+)"'
referer_re = double_quoted_text_re
user_agent_re = double_quoted_text_re

log_line_re = re.compile(f"^({ip_re}) ({identd_user_re}) ({user_id_re}) {date_time_re} {request_re} ({http_status_re}) ({bytes_rx_re}) ({bytes_tx_re}) ({ttfb_re}) ({response_time_re}) {referer_re} {user_agent_re}$")


@dataclass
class LogRecord:
    client_ip: str         # 0
    indent_user: str
    user_id: str
    date_time: datetime
    method: str
    url: str               # 5
    protocol: str
    protocol_version: str  # could be float, but we don't really care; besides, x.y.z?
    status: int
    bytes_rx: int
    bytes_tx: int          # 10
    ttfb: int  # maybe -!
    response_time: int
    referer: str
    user_agent: str

    @classmethod
    def from_log_line(cls, line):
        match = log_line_re.match(line)

        if match is None:
            raise ValueError(f"Malformed line: {line.strip()}")

        data = list(match.groups())
        new_data = []

        group_index = 0
        for field_index, (name, field) in enumerate(cls.__dataclass_fields__.items()):
            if field.type == datetime:
                # [11/May/2026:20:15:28 +0200]

                # convert month str to number
                data[group_index+1] = month_names.index(data[group_index+1]) + 1

                # convert to ints
                data[group_index:group_index+6] = [ int(x) for x in data[group_index:group_index+6] ]

                new_data.append( datetime(data[group_index+2], data[group_index+1], data[group_index],
                                          data[group_index+3], data[group_index+4], data[group_index+5], 0,
                                          utc_offset2tzinfo(data[group_index+6])) )

                group_index += 7

                continue

            # handle ttfb as -
            if field_index == 11 and data[group_index] == '-':
                # 
                data[group_index] = data[group_index+1]

            if data[group_index:group_index+4] == [ None, None, None, None ]:
                if group_index in (10, 14):
                    # handle (None, None, None, None, '', '\\x16\\x03\\x02\\x01o\\x01', '', '')
                    # handle ('GET', '/', 'HTTP', '1.0', None, None, None, None)

                    # no need to add anything, it's handled by the fallback
                    # but we still need to skip this cruft
                    group_index += 4
                else:
                    raise ValueError(f"Got confused: {(field_index, field.type, group_index, data[group_index], new_data)}")

            # convert ints
            if field.type == int:
                data[group_index] = int(data[group_index])

            # fallback
            new_data.append(data[group_index])
            group_index += 1

        return cls(*new_data)


    # implement the iterator protocol so we can mostly be passed as argument to execute()
    def __iter__(self):
        return self


    def __iter__(self):
        for value in self.__dict__.values():
            # the whole protocol could be replaced with .__dataclass_fields__.values() :shrug:
            # but this way I can do further conversions
            if type(value) == datetime:
                value = int(value.timestamp())

            yield value


def utc_offset2tzinfo(offset: str) -> tzinfo:
    # +0200
    hours = int(offset[:3])    # +02
    minutes = int(offset[3:])  # 00

    return timezone(timedelta(hours=hours, minutes=minutes), offset)


def connect():
    # if we test after sqlite3.connect(), the file is already created
    create = not pathlib.Path('./apache_logs.db').exists()

    conn = sqlite3.connect('./apache_logs.db')

    if create:
        conn.cursor().execute('''
CREATE TABLE "logs" (
    "client"    TEXT,
    "indent_user"   TEXT,
    "user_id"   TEXT,
    "timestamp" INTEGER,
    "method"    TEXT,
    "url"   TEXT,
    "protocol"  TEXT,
    "protocol_version"  TEXT,
    "status"    INTEGER,
    "bytes_rx"  INTEGER,
    "bytes_tx"  INTEGER,
    "ttfb"  INTEGER,
    "response_time" INTEGER,
    "referer"   TEXT,
    "user_agent"    TEXT
);''')

    return conn



def main():
    conn = connect()
    cursor = conn.cursor()

    for line in sys.stdin:
        try:
            log_record = LogRecord.from_log_line(line)
        except ValueError as e:
            print(e.args[0])
            continue

        cursor.execute('''INSERT INTO logs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)''',
                       tuple(log_record))
        conn.commit()


if __name__ == '__main__':
    main()

One of the things I didn't do was to further play with the URLs. One could make list of different apps based on whether there is any routing to different services, like in the cases of my previous job and my own server; or even different subdivisions on a single app, like for NextCloud:

ocs/v2.php/apps/serverinfo
remote.php/dav/files/USER
remote.php/dav/calendars/USER/CALENDAR/

etc. I haven't really thought about it; it could be implemented either as more columns or extra tables.

Today I managed to finish the rest.

The next step is to install this so it runs constantly with the output of tail --follow=name --retry piped into its stdin¹. Left as an exercise for the reader; use SystemD units :)

Next is installing Grafana's plugin to read SQLite and declare the new Grafana datasource.

The hard part was to query the data in a way that was useful for Grafana. I managed to get a query like:

-- round to the minute
SELECT timestamp/60*60 AS time, status, COUNT(status) as "count"
FROM logs
-- $__from and $__to are defined by Grafana based on the dashboards's time range
WHERE timestamp >= $__from / 1000 and timestamp < $__to / 1000
GROUP BY timestamp/60, status

to get the count of different status codes per minute². But this returns a table that looks like:

      time | status | count
1778533620 |    200 |    30
1778533620 |    207 |     3
1778533620 |    403 |     1

while Grafana is expecting one line per sample (but remember we're aggregating data) and one column per data series:

      time | 200 | 207 | 403
1778533620 |  30 |   3 |   1

I read how to pivot this in SQL, but it mostly works only if you know the different values for the status column beforehand. This might be feasible with HTTP status codes (I count 63 standard ones, including the joke 418 I'm a teapot), but that would be impossible for the referer or user_agent columns.

Thanks to iRobbery#postgresql@libera.chat I found about Grafana's Partition by values data transformation. Aplying it to the column that defines the time series (status, etc), it gives us exactly what we want!

And one can even include a pure table with all the logs to inspect when one finds weird spikes or values. I made almost impossible queries like transferred bytes per URL, methods per client and more! One missing piece, i possible, would be to implement histograms, like last time we looked into this.

One could cite the UNIX philosophy, but seriously, who wants to reimplement all the corner cases of that tail invocation? See for instance the 113 bugs found in the coreutils Rust reimplementaiton ↩
One could use a dashboard variable to control this arbitrarily. One could get granularity per second! ↩

Monitoring the maximum latency for a scrape period with prometheus

Marcos Dione — Mon, 11 Mar 2024 18:56:18 GMT

Ever since I watched (several times) Gil Tene's talk about how we are measuring latency wrong I set myself to try get such values in a graph. His thesis is that we are using histograms and other similar tools for measuring percentiles of our response times and that that is wrong because we never see the maximum time; our tools are hiding them behind averages. What can we do about it with our current tools, in particular Prometheus and Grafana? Let's first define exactly what we want:

On one side we have a stream of log lines. These lines will be parsed and some values will be extracted from them. On the other side we have Prometheus trying to scrape a representative of those values. The representative that we want for every scrape is the max of all the values collected in that period between scrapes. For the next scraping period, we must forget this max and calculate the new one, and so on for each scrape interval.

Prometheus offers us three different metric types that could allow us to get such value. The first and obvious one would be a gauge, but there is no tool that does what I described in the previous paragraph.

The second one would be a histogram. For a histogram, we define a set of values that become boundaries. For each interval defined by those boundaries, the histogram gives us the amount of values that fall in them, and also below the lowest boundary and above the highest one. This is not enough because we can never have even a ballpark of what's the maximum value is, much less it's actual value. The max is lost among all the values that fall in the interval above the las boundary we define; we only know which interval contains the max. To at least get a ballpark you need to figure out an upper boundary, otherwise the max could be anywhere between the highest boundary and infinite. Maybe you're lucky and you have something like a timeout, but it's not often that a timeout fires during active data transmissions. On the other hand, if you're measuring transfer times for a service that returns files, you don't have an upper bound of how long it can take because it depends on the whole network path and the size of the file.

The last one is summaries. Summaries work slightly different. Instead of partitioning in terms of values, it partitions in terms of percentiles. One aspect of summaries is that you cannot aggregate them. Not only Prometheus' documentation mentions this, it also cites one of Gil Tene's articles about it¹. But one of the peculiarities of percentiles is that if you take the 0 percentile, you get the minimum, and if you take the 100 percentile, you get the maximum. And as Gil Tene mentions in that same article, those are the only two values that you can aggregate because if you have all the minimums, the global minimum is just the minimum of all those values, and similarly for the maximum.

Now, how do we actually implement it? I had mentioned before, we have two tools for converting log lines into metrics for Prometheus. One is mtail, and the other one is the Grok exporter. mtail doesn't seem to have any support for summaries, so we are only left with the Grok exporter.

I tried to set up a Grok exporter configuration to do exactly that, only the minimum and the maximum for a given value, which in my case was time to first byte (TTFB), because I'm trying to apply it to a service that gives files as responses. The total latency of such service depends on the size of the file, but TTFB should not, so I'm interested in that.

I want the minimum and the maximum, so I set up a metric for that. I define it as summary, and I set it up for the zero and a hundred percentiles. And this is where the problem starts getting slightly more complicated.

First, the Grok exporter uses a library for computing the percentiles. That library does not accept just an array of the percentiles you are interested in; quoting the docs:

grok_exporter does not provide exact values for the quantiles, but only estimations. For each quantile, you also specify an uncertainty that is tolerated for the estimation

I don't know how much that impacts the two values I'm looking for, minimum and maximum, but I started with a 1% tolerance just to try:

- type: summary
  name: elevation_tile_ttfb_extremes
  help: Minimum and Maximum for TTFB
  match: '%{ELEVATION_LOG}'
  value: '{{multiply .first_byte 0.001}}'  # µs to ms
  quantiles: { 0: 0.01, 100: 0.01 }  # min, max
  max_age: 15s

And it didn't work.

First, it started complaining that when it was calculating some values, it was trying to convert the string that it got from the log lines to an integer, but there were empty strings and basically it was failing:

grok-exporter[1402204]: WARNING: skipping log line: error processing metric elevation_tile_ttfb:
    unexpected error while evaluating template for label __value__: template: __value__:1:2:
    executing "__value__" at <multiply .first_byte 0.001>: error calling multiply: error executing multiply function:
    cannot convert  to floating point number: strconv.ParseFloat: parsing "": invalid syntax

(I had to split it into several lines so you don't have to scroll horizontally [so much]. I'm not going to comment on whether Golang's tendency to wrap errors to the point, shown above, where the chain of error messages becomes a type of horizontal stack trace, is a good thing or not).

Once I fixed that, instead I got a traceback that points to an index error inside that library.

2024/03/10 13:42:33 http: panic serving 127.0.0.1:49912: runtime error: index out of range [99] with length 1
goroutine 39 [running]:
net/http.(*conn).serve.func1(0xc000420000)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0xb7d0e0, 0xc000038060)
        /usr/local/go/src/runtime/panic.go:975 +0x3e9
github.com/fstab/grok_exporter/vendor/github.com/beorn7/perks/quantile.(*Stream).Query(0xc00045cd20, 0x4059000000000000, 0x4051e3d70a3d70a4)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/beorn7/perks/quantile/stream.go:165 +0x16f
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*summary).Write(0xc000144c00, 0xc000175b20, 0x3, 0xc00023f412)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/summary.go:310 +0x3c7
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus.processMetric(0xca4840, 0xc000144c00, 0xc00023f5c0, 0xc00023f5f0, 0x0, 0x0, 0x0)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:596 +0xa2
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000100410, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:490 +0x8ca
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0x7fe9cd2298f8, 0xc00050e280, 0xc00022e300)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/http.go:126 +0x99
net/http.HandlerFunc.ServeHTTP(0xc00017a230, 0x7fe9cd2298f8, 0xc00050e280, 0xc00022e300)
        /usr/local/go/src/net/http/server.go:2042 +0x44
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1(0x7fe9cd2298f8, 0xc00050e280, 0xc00022e300)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:40 +0xab
net/http.HandlerFunc.ServeHTTP(0xc00050d170, 0x7fe9cd2298f8, 0xc00050e280, 0xc00022e300)
        /usr/local/go/src/net/http/server.go:2042 +0x44
github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1(0xca9840, 0xc00024a000, 0xc00022e300)
        /go/src/github.com/fstab/grok_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:100 +0xda
net/http.HandlerFunc.ServeHTTP(0xc00050d260, 0xca9840, 0xc00024a000, 0xc00022e300)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0x131bd20, 0xca9840, 0xc00024a000, 0xc00022e300)
        /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000196000, 0xca9840, 0xc00024a000, 0xc00022e300)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000420000, 0xcac140, 0xc000244000)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c

In my last post about this, I mentioned that the Grok exporter has been unsupported for four years, nobody has written any single line of code after that, and there are many open issues and even several requests still open and nothing has happened in the last four years. And this library, the one that calculates the percentiles, is even worse. It's been five years since they last updated it, which means that there's no chance that even if I report the bug, I will get releases of both that library and then the Grok exporter to fix it. And I don't feel like reading a whole paper in statistics at this point of my life just to solve this problem².

So what options do we have? Well, I think there's only one, and it's writing your own tool that fulfills the description in the second paragraph. I'm not really sure I have the energy to do it, but I think this will keep jumping around my head and I will succumb and do it, mostly because I think it's important.

But until that moment occurs, I have no solution for this problem, and that means that in the meantime I will have to live with histograms.

It's actually interesting that Prometheus people are aware of Gil Tene's ideas and yet don't have a solution for them. ↩
I remember being a frevient free software advocate for reasons exactly like this. But then, I don't have the energy nor I think I can convince my manager to get someone paid to fix it. ↩