.:: Marcos Dione/StyXman's glob ::. (Posts about gis)

Block sizes from OSM data

Marcos Dione — Sat, 14 Mar 2026 11:16:41 GMT

My city has big blocks. One close to mine is around 6km of circumference. I wanted to make a map that show the sizes of the blocks. What I have is a rendering DB for my area. Let's see what I can do.

An obvious solution would be to use any routing database, which would convert squiggly segments into edges of a graph. I feebly tried that, but no routing system I know is packaged for Debian, and one of them asked me to compile it. I could compile it, but I was lazy that way.

So the plan is this: take the whole street network¹ and recursively remove the dead end streets. The recursion part is because there are areas where there are whole branches of dead end streets, so if in one pass a street A might look not dead because it connects to another B, but B is removed, the next pass should remove A too. So, for each street, take each end, and see how many other segments it's connected to; if 0, it's a dead end and we should remove it. Keep going until no new streets were removed.

One problem is how the data is represented. In a rendering database, streets do not exist, just segments with a consistent tagging, so streets can be split if, for instance, max speed changes, or there's a bridge. But one thing they're not split on is where another street joins them (which would be reflected in a routing db, lazy me! :)

This does not matter when searching for connected streets to a point, but a segment might have parts that are a dead end, and parts that are not, because they're connected to other streets(segments) that are connected too. So now, for each end, take the point, and if it's not connected, removed the point and try again. The full algo:

Find all the segments
While no segment has been removed
- For each segment
  - For each end
    - While the end is not connected
      - Remove the end from the segment
    - If not enough nodes left, remove the segment

Looks like an infinite problem! The city I'm interested in has 16k segments, and this looks worse than O(N²)! But we can do some shortcuts. One of the expensive operations is "the end is connected" or "find all the other segments that include this point". Luckily, a rendering DB has a geographic index, and we can do tricks like looking for segments only around the area of the segment you're looking at, so reducing the amount of comparisons by a lot: less that 10 instead of 16k! And of course, you should not consider segments you already removed.

To be honest, I thought this would take way more effort and code. Yes, I changed the algo thrice, but all in all it took me like 6h to get it as it is. Now, it is not perfect. It does not detect some artifacts (mostly cycles connected to a single segment), but is good enough for me for now. Not bad for <200 lines of Python and SQL :)

#! /usr/bin/env python3

import psycopg2
from shapely import from_wkb, set_srid, LineString, Point


def main():
    print('Fetching all segments, takes a while.')
    conn = psycopg2.connect(dbname='europe')
    cur = conn.cursor()

    cur.execute(f"""
        WITH marseille AS (
            SELECT way
            FROM planet_osm_polygon
            WHERE osm_id = -76469
        )
        SELECT line.osm_id, line.way
        FROM
            planet_osm_line AS line,
            marseille
        WHERE
            line.way && marseille.way AND
            line.highway IN (
                'primary',
                'primary_link',
                'secondary',
                'secondary_link',
                'tertiary',
                'tertiary_link',
                'residential',
                'unclassified',
                'living_street',
                'road'
            ) AND
            (line.access IS NULL OR line.access = 'yes')
            { f" LIMIT {sys.argv[1]}" if len(sys.argv) > 1 else '' };
    """)

    segments = {}
    segments_removed = set([666])  # fake osm_id so line.osm_id NOT IN %s AND works (empty sets are syntax errors)

    # first pass: collect all segments
    for data in cur.fetchall():
        osm_id = data[0]
        segment = from_wkb(data[1])

        segments[osm_id] = segment

    print(f"Found {len(segments)} segments.")

    # second pass: eliminate segments for which we can find an unconnected end
    pass_number = 1

    while len(segments) > 0:
        print(f"PASS {pass_number:2d}")

        total_count = 0
        total_changed = 0
        to_remove = set()

        for osm_id, segment in segments.items():
            changed = False
            removed = False
            connections = 0

            for direction in -1, 0:
                while len(segment.coords) > 0:
                    point = Point(segment.coords[direction])

                    cur.execute("""
                        WITH segment AS (
                            SELECT way
                            FROM planet_osm_line
                            WHERE osm_id = %s
                        )
                        SELECT
                            line.osm_id,
                            line.name
                        FROM planet_osm_line AS line, segment
                        WHERE
                            line.way && ST_Buffer(ST_Envelope(segment.way), 10) AND
                            line.osm_id != %s AND
                            line.osm_id NOT IN %s AND
                            line.highway IN (
                                'trunk',
                                'trunk_link',
                                'primary',
                                'primary_link',
                                'secondary',
                                'secondary_link',
                                'tertiary',
                                'tertiary_link',
                                'residential',
                                'unclassified',
                                'living_street',
                                'road'
                            ) AND
                            (access IS NULL OR access = 'yes') AND
                            ST_Intersects(line.way, ST_GeomFromWKB(%s, 3857));
                    """, (osm_id, osm_id, tuple(segments_removed), point.wkb))

                    data = cur.fetchall()

                    if len(data) == 0:
                        if len(segment.coords) == 2:
                            # removing one point removes the segment
                            removed = True
                            to_remove.add(osm_id)
                            break

                        # remove the point
                        if direction == 0:
                            segment = LineString(segment.coords[1:])
                        else:
                            segment = LineString(segment.coords[:-1])

                        changed = True
                    else:
                        connections += len(data)
                        break

            if changed:
                segments[osm_id] = segment
                total_changed += 1

            total_count += 1
            match total_count % 10, changed, removed:
                case _, True, _:
                    print('*', end='', flush=True)
                case _, False, True:
                    print('_', end='', flush=True)
                case 0, _, _:
                    print('|', end='', flush=True)
                case 5, _, _:
                    print(',', end='', flush=True)
                case _:
                    print('.', end='', flush=True)

            if total_count % 100 == 0:
                print()

        segments_removed.update(to_remove)

        print()
        print(', '.join([ str(id) for id in to_remove ]))

        for osm_id in to_remove:
            del segments[osm_id]

        print(f"{total_changed} changed, {len(to_remove)} removed, {len(segments)} left.")
        print()

        # if only some segments were changed, the topology does not change, so we save one pass
        if len(to_remove) == 0:
            break

        pass_number += 1

    print("Saving segments...")
    for segment in segments.values():
        cur.execute("""
            INSERT INTO streets (way)
            VALUES (ST_GeomFromWKB(%s, 3857))
        """, (segment.wkb, ))

    conn.commit()

    conn.commit()
    print()

    print("Cutting...")
    for index, segment in enumerate(segments.values()):
        buffered = segment.buffer(1)

        marseille = difference(marseille, buffered)

        match (index + 1) % 10:
            case 0:
                print('|', end='', flush=True)
            case 5:
                print(',', end='', flush=True)
            case _:
                print('.', end='', flush=True)

        if (index + 1) % 100 == 0:
            print()

    print()

    out = open('blocks.geojson', 'w+')
    # GeoJSONstream, one GeoJSON object per line
    # GDAL/QGIS won't accept 
    for polygon in marseille.geoms:
        out.write(f"{json.dumps(mapping(polygon))}\n")


main()

Runtime takes a while because someone correctly micromapped a 350m dead end road due to diffrences in road side parking, 14 segments/passes in total; otherwise 8 would have suffised. I could have added a heuristic where, when a pass removed just a few segments, the next pass would only search among the segments close to those. Technically I could do this from the first pass.

One thing that surprised me whas this: QGIS can read GeoJSON files, but if they're going to be a colleciton of things, better be a GeoJSONseq, that is a file with a GeoJSON object per line. But since these files do not include info about the EPSG, you have to set it by hand on QGIS. now, this would be fine if QGIS would ask about it when loading the layer, but instead two things happen: it confuses the EPSG, but would still correctly zoom to the layer assuming that the layer has the same projection as the project. Thanks to uglyhack#qgis@libera.chat.

Several caveats:

I didn't include motorways or tunks in the block computation, because they create a complication in the definition of block, specially with bridges over or tunnels under other roads, of which there are a lot here. I know at least 3 bridges that still break the graph; there are few enough that I can ignore them.

The city has big blocks of industrial areas. Service roads are not included, and in any case most are dead ends.

The biggest areas are actually not urban, including a big chunk of a National Park in the two big red areas to the South² and the ports and the sea nearby³. Extracting a real urban area is possible, but harder. The road network should give you an idea of what it looks like.

On top I also drew three things, roads (grey, white, blue), rivers (thick blue) and train tracks (black), to give a better idea of the complexity of the city. This time it includes motorways, trunks, service and private roads. The latter also give an idea of how much is in private hands (white). The thin blue lines are the streets thad define the blocks, and the grey are public but dead ends.

Edit: while looking for something else I found this function:

https://postgis.net/docs/ST_Polygonize.html

Maybe one day I'll use this one instead :)

I'm focused on a traffic issue: the city is not navigable by car, leading to lots of traffic in the few streets available, leading to lots of noise from impatient drivers. ↩
The map is rotated 108° so I could zoom in as much as possible, so South is rougly to the left. ↩
See where all the ferry lines go to; that's the Vieux Port (Old Port), and the new, industrial one is in the same area to the North/right. ↩

Automating blender based hillshading with Python

Marcos Dione — Sun, 05 Nov 2023 16:19:45 GMT

Remember my Blend based hillshading? I promised to try to automate it, right? Well, it seems I have the interest and stamina now, so that's what I'm doing. But boys and girls and anything in between and beyond, the stamina is waning and the culprit is Blender's internals being exposed into a non-Pythonic API³. I swear if I worked in anything remotely close to this, I would be writing a wrapper for all this. But in the meantime, it's all a discovery path to something that does not resemble a hack. Just read some of Blender's Python Quickstart:

When you are familiar with other Python APIs you may be surprised that new data-blocks in the bpy API cannot be created by calling the class:

bpy.types.Mesh()
Traceback (most recent call last):
File "<blender_console>", line 1, in <module>
TypeError: bpy_struct.__new__(type): expected a single argument

This is an intentional part of the API design. The Blender Python API can’t create Blender data that exists outside the main Blender database (accessed through bpy.data), because this data is managed by Blender (save, load, undo, append, etc).

Data is added and removed via methods on the collections in bpy.data, e.g:

mesh = bpy.data.meshes.new(name="MyMesh")

That is, instead of making the constructor call this internal API, they make it fail miserably and force you to use the internal API! Today I was mentioning that Asterisk's programming language was definitely designed by a Telecommunications Engineer, so I guess this one was designed by a 3D artist? But I digress...

One of the first thing about Blender's internals is that one way to work is based on Contexts. This makes sense when developing plugins, where you mostly need to apply things to the selected object, but for someone really building everything from scratch like I need to, it feels weird.

One of the advantages is that you can open a Python console and let Blender show you the calls it makes for every step you make on the UI, but it's so context based that the results is useless as a script. Or for instance, linking the output of a thing into he the input of another is registered as a drag-and-drop call that includes the distance the mouse moved during the drag, so it's relative of the output dot where you started and what it links to also depends on the physical and not logical position of the things you're linking,

bpy.ops.node.link(detach=False, drag_start=(583.898, 257.74))

It takes quite a lot of digging around in a not very friendly REPL¹ with limited scrollback and not much documentation to find more reproducible, less context dependent alternatives. This is what's eating up my stamina, it's not so fun anymore. Paraphrasing someone on Mastodon: What use is a nice piece of Open Software if it's documentation is not enough to be useful²?

Another very important thing is that all objects have two views: one that has generic properties like position and rotation, which can be reacheched by bpy.data.objects; and one that has specific properties like a light's power or a camera's lens angle, which can be reached by f.i. bpy.data.cameras. This was utterly confusing, specially since all bpy.data's documentation is 4 lines long. Later I found out you can get specific data from the generic one in the .data attribute, so the take out is: always get your objects from bpy.data.objects.

Once we get over that issue, things are quite straightforward, but not necessarily easy. The script as it is can already be used with blender --background --python <script_file>, but have in account that when you do that, you start with the default generic 3D setup, with a light, a camera and a cube. You have to delete the cube, but you can get a reference to the other two to reuse them.

Then comes the administrative stuff around just rendering the scene. To industrialize it and be able to quickly test stuff, you can try to get command line options. You can use Python's argparser module for this, but have in account that those --background --python blender.py options are going to be passed to the script, so you either ignore unknown options or you declare those too:

mdione@ioniq:~/src/projects/elevation$ blender --background --python blender.py
Blender 3.6.2
Read prefs: "/home/mdione/.config/blender/3.6/config/userpref.blend"
usage: blender [-h] [--render-samples RENDER_SAMPLES] [--render-scale RENDER_SCALE] [--height-scale HEIGHT_SCALE] FILE
blender: error: unrecognized arguments: --background --python

Also, those options are going to be passed to Blender! So at the end of your run, Blender is going to complain that it doesn't understand your options:

unknown argument, loading as file: --render-samples
Error: Cannot read file "/home/mdione/src/projects/elevation/--render-samples": No such file or directory
Blender quit

The other step you should do is to copy the Geo part of GeoTIFF to the output file. I used rasterio, mostly because at first I tried gdal (I was already using gdal_edit.py to do this in my previous manual procedure), but it's API was quite confusing and rasterio's is more plain. But, rasterio can't actually open a file just to write the metadata like gdal does, so I had to open the output file, read all data, open it again for writing (this truncates the file) and write metadata and data.

Now, some caveats. First, as I advanced in my last post, the method as it is right now has issues at the seams. Blender can't read GDAL VRT files, so either I build 9 planes instead of 1 (all the neighbors are needed to properly calculate the shadows because Blender is also taking in account light reflected back from other features, meaning mountains) or for each 1x1 tile I generate another with some buffer. I will try the first one and see if it fixes this issue without much runtime impact.

Second, the script is not 100% parametrized. Sun size and power are fixed based on my tests. Maybe in the future. Third, I will try to add a scattering sky, so we get a bluish tint to the shadows, and set the Sun's color to something yellowish. These should probably be options too.

Fourth, and probably most important. I discovered that this hillshading method is really sensible to missing or bad data, because they look like dark, deep holes. This is probably a deal breaker for many, so you either fix your data, or you search for better data, or you live with it. I'm not sure what I'm going to do.

So, what did I do with this? Well, first, find good parameters, one for render samples and another for height scale. Render time grows mostly linearly with render samples, so I just searched for the one before detail stopped appearing; the value I found was 120 samples. When we left off I was using 10 instead of 5 for height scale, but it looks too exaggerated on hills (but it looks AWESOME in mountains like the Mount Blanc/Monte Bianco! See below), so I tried to pinpoint a good balance. For me it's 8, maybe 7.

Why get these values right? Because like I mentioned before, a single 1x1°, 3601x5137px tile takes some 40m in my laptop at 100 samples, so the more tuned the better. One nice way to quickly test is to lower the samples or use the --render-scale option of the script to reduce the size of the output. Note that because you reduce both dimensions at the same time, the final render (and the time that takes) is actually the square of this factor: 50% is actually 25% (because 0.50 * 0.50 = 0.25).

So, without further addo, here's my script. If you find it useful but want more power, open issues or PRs, everything is welcome.

https://github.com/StyXman/blender_hilllshading ⁵

Try to use the main branch; develop is considered unstable and can be broken.

A couple of images of the shadows applied to my style as teaser, both using only 20 samples and x10 height scale:

Dhaulagiri:

Mont Blanc/Monte Bianco:

Man, I love the fact that the tail of the Giacchiaio del Miage is in shadows, but the rest is not; or how Monte Bianco/Mont Blanc's shadow reaches across the valley to the base of la Tête d'Arp. But also notice the bad data close to la Mer de Glace.

Ok, TBH here, I'm very much used to ipython's console, it's really closer to the plain python one. No tab completion, so lots of calls to dir() and a few help()s. ↩
I couldn't find it again. Mastodon posts are not searchable by default, which I understand is good for privacy, but on the other hand the current clients don't store anything locally, so you can't even search what you already saw. I have several semi-ranting posts about this and I would show them to you, but they got lost on Mastodon. See what I mean? ↩
So you have an idea, this took me a whole week of free time to finish, including but not in the text, my old nemesis, terracing effect. This thing is brittle. ↩
Yeah, maybe the API is mostly designed for this. ↩
My site generator keeps breaking. This is the second time I have to publicly admit this. Maybe next weekend I'll gather steam and replace it with nikola. ↩