Optimizing mapnik rendering with metatiles
One of the most cited ways to accelerate rendering maps with Mapnik is metatiling.
This technique is simple: instead of rendering each individual 256x256 PNG file
that normally backs a slippy map, render NxN times that (normally with N=8) and
split afterwards. mapnik-stylesheet's generate_tiles.py was not capable of
doing it, but I hacked a little bit and got it to do it, so after integrating
openstreetmap-carto v2.14.1 and fixing some local bugs, I gave it another spin.
This time I also imported the whole Europe data, and expanded the terrain to cover anything above 60°N, taking the height files form De Ferranti's viewfinderpanoramas. This meant that whatever rendering times I got this time, they are not completely comparable to what I obtained before because it's simply handling way more data, specially relative to the terrain data.
So, first, a short revisit to importing. Here's the summary:
mdione@diablo:/var/lib/data/mdione/src/projects/osm/data/osm$ osm2pgsql \ --create --drop --database gis --slim --flat-nodes /home/mdione/src/projects/osm/nodes.cache \ --cache 2048 --number-processes 4 --unlogged europe-latest.osm.pbf osm2pgsql SVN version 0.82.0 (64bit id space) Node-cache: cache=2048MB, maxblocks=262145*8192, allocation method=11 Mid: loading persistent node cache from /home/mdione/src/projects/osm/nodes.cache Allocated space for persistent node cache file Maximum node in persistent node cache: 0 Mid: pgsql, scale=100 cache=2048 Reading in file: europe-latest.osm.pbf Processing: Node(1225816k 799.6k/s) Way(151242k 17.71k/s) Relation(1872470 338.91/s) parse time: 15596s [4h20m] Node stats: total(1225816996), max(2884224404) in 1533s [0h26m] Way stats: total(151242682), max(284701738) in 8538s [2h22m] Relation stats: total(1872475), max(3776697) in 5525s [1h32m] Going over pending ways... Maximum node in persistent node cache: 2884632575 110980610 ways are pending Using 4 helper-processes [but only 1 is used, the others failed, dunno why] Mid: loading persistent node cache from /home/mdione/src/projects/osm/nodes.cache Maximum node in persistent node cache: 2884632575 Process 0 finished processing 110980610 ways in 34202 sec [9h30m] 110980610 Pending ways took 34202s at a rate of 3244.86/s Going over pending relations... Maximum node in persistent node cache: 2884632575 0 relations are pending Using 4 helper-processes Process 2 finished processing 0 relations in 2 sec Process 3 finished processing 0 relations in 2 sec Process 0 finished processing 0 relations in 3 sec Process 1 finished processing 0 relations in 3 sec 0 Pending relations took 3s at a rate of 0.00/s node cache: stored: 203128264(16.57%), storage efficiency: 75.67% (dense blocks: 138131, sparse nodes: 63494657), hit rate: 17.62% Stopped table: planet_osm_rels in 1s Stopped table: planet_osm_nodes in 1s Stopped table: planet_osm_ways in 3s All indexes on planet_osm_roads created in 4679s [1h18m] All indexes on planet_osm_point created in 6093s [1h41m] All indexes on planet_osm_line created in 9703s [2h42m] All indexes on planet_osm_polygon created in 12735s [3h32m] Osm2pgsql took 62780s overall [17h26m]
Only ~3h30m more than importing a part of Europe, I think it's manageable. I still don't know what to make out of the cache/storage efficiency line, but it's clearly related to the small size of the cache.
About the rendering, I only rendered up to zoom level 11, as I discussed before, but that limit was mostly due to the fact that going all the way down to 14 took too much time. But after viewing how much less rendering took this time, I most probably revert that change. Take a look at the graph:
This graph shows the average and total time for rendering up to zoom level 11 with single tiling (actually it's the old data I showed previously) and metatiling 8x8. The total time uses the logarithmic scale on the right. The average in the second case is the division between the total time and the amount of final individual tiles, which are produced by cutting the metatile in as many single tiles as the minimum between 2^z and 64, so zoom levels 0-2 don't produce spurious images. The times are amazing. Notice that from time to time the total time took more in the metatile version than in the single tile version, but the are two factors that impact here. One is that due to metatiling, I'm rendering tiles that with single tiling I wouldn't due to the fact that now I can only cut in increments of 8 instead of one. Two, now I'm rendering the whole Europe instead of just a big part of it. This makes the amount of tiles produced to be between 6 to 10 times more.
Still, take a look at the average rendering times for metatiling, and specially at the last column in the table, which is not graphed: it's the proportion between the average tile rendering time, meta to single. It's mostly below the 10%, except for a couple of zoom levels that either produce few tiles (ZL 2, 16 tiles) or stay under 15% (ZL 4). This is ridiculously low, which means fast. I will first finish rendering my zone down to ZL 18 and then render the rest down to ZL 14 to have more complete, comparable graphs.