Optimizing the render stack

Since I started playing with rendering maps locally I've been modifying and extending the original generate_tiles.py script from mapnik-stilesheets. I added option parsing and lots of features; here's the current usage:

$ ./generate_tiles.py --help
usage: generate_tiles.py [-h] [-b BBOX] [-B BBOX_NAME] [-n MIN_ZOOM]
                        [-x MAX_ZOOM] [--tiles Z,X,Y [Z,X,Y ...]]
                        [-i MAPFILE] [-f FORMAT] [-o TILE_DIR]
                        [-m METATILE_SIZE] [-t THREADS]
                        [-p {threads,fork,single}] [-X] [-N DAYS]
                        [-E {skip,link,render}] [-d] [--dry-run]

optional arguments:
-h, --help            show this help message and exit
-b BBOX, --bbox BBOX
-B BBOX_NAME, --bbox-name BBOX_NAME
-n MIN_ZOOM, --min-zoom MIN_ZOOM
-x MAX_ZOOM, --max-zoom MAX_ZOOM
--tiles Z,X,Y [Z,X,Y ...]
-i MAPFILE, --input-file MAPFILE
-f FORMAT, --format FORMAT
-o TILE_DIR, --output-dir TILE_DIR
-m METATILE_SIZE, --metatile-size METATILE_SIZE
-t THREADS, --threads THREADS
-p {threads,fork,single}, --parallel-method {threads,fork,single}
-X, --skip-existing
-N DAYS, --skip-newer DAYS
-E {skip,link,render}, --empty {skip,link,render}
-d, --debug
--dry-run

BBoxes are stored in a file called bboxes.ini, so I can say -B Europe instead of remembering the coords. The idea of --format is that I should be supporting slippy maps .png file structure or mbtiles, but the latter support is a little lagging behind because I don't have a use for them yet. You can choose to whether use threads (broken because mapnik cannot handle the situation; I can't find a reference to the problem now), child processes (probably the only one working correctly) or a single main process (so no parallelism). It handles resuming a stopped render by not rendering if the tile exists or it's too new. It also can skip writing empty seas tiles.

I use it to rerender my style everytime I make a modification (or just update to the latest openstreetmap-carto, of which is a fork). I usually bulk render a great part of Europe up to ZL 11 or 14, and them some regions down to ZL 18 or 19 as needed for trips or other interests.

For Europe, it can take a long while, so I've been thinking on ways to optimize the rendering. Besides tuning the database, I first found that rendering big metatiles (8x8, for instance) gave a big boost in rendering time. The next idea is to reuse disk cache. When you render a (meta)tile in ZL N, the same data used for rendering it is going to be used for the 4 sub(meta)tiles of ZL N+1 (except when you remove features, which is rare but exists; city labels come to mind). I don't think something could be done at mapnik level, but one can think of the tiles as a tree: a node in ZL N has 4 subtiles in level N+1 and the leafs are the last ZL rendered. The original algorithm did a breath first traveling of this tree, but if you do a depth first algorithm, it could reuse the kernel's page/disk cache for the data collected by mapnik from the database or other files. Also, I can check whether the subtiles are render worthy: if they're only sea, I don't need to render it or its subtiles; I can cut down whole tile trees. The only point at which this could no longer be true is when we start rendering more things on sea, which currently ammounts to ferry routes at ZL 7.

I finished implementing all these ideas, but I don't have any numbers to prove it works. Definitely not rendering sea tiles should be a great improvement, but I don't really know whether the caching idea works. At least it was fun to implement.

So the rendering batch will be cut in 3: ZLs 0-6 in one run, then 7 and 8 with less threads (these ZLs of my style use so much memory the machine starts thrashing), then 9-18/19 with full threads.