Ever since I started working in Elevation I faced the problem that it's mostly a openstreetmap-carto fork. This means that each time osm-carto has new changes, I have to adapt mine. We all know this is not an easy task.

My first approach was to turn to osm-carto's VCS, namely, git. The idea was to keep a branch with my version, then pull from time to time, and merge the latest released version into my branch, never merging mine into master, just in case I decided to do some modifications that could benefit osm-carto too. In that case, I would work on my branch, make a commit there, then cherry pick it into master, push to my fork in GitHub, make a Pull Request and voilà.

All this theory is nice, but in practice it doesn't quite work. Every time I tried to merge the release into my local branch, I got several conflicts, not to mention modifications that made some of my changes obsolete or at least forcing me to refactor them in the new code (this is the developer in me talking, you see...). While I resolved these conflicts and refactorings, the working copy's state was a complete mess, forcing me to fix them all just to be able to render again.

As this was not a very smooth workflow, I tried another approach: keeping my local modifications in a big patch. This of course had the same and other problems that the previous approach, so I gained nothing but more headaches.

Then I thought: who else manages forks, and at a massive scale? Linux distributions. See, distros have to patch the packaged software to compile on their environments. They also keep security patches that also are sent upstream for inclusion. Once a patch is accepted upstream, they can drop their local patch. This sounds almost exactly the workflow I want for Elevation.

And what do they use for managing the patches? quilt. This tool is heavily used in the Debian distro and is maintained by someone working at SuSE. Its documentation is a little bit sparse, but the tool is very simple to use. You start doing quilt init just to create the directories and files that it will use to keep track of your changes.

The modification workflow is a little bit more complex that with git:

  1. You mark the beginning of a new patch with quilt new <patch_name>;
    1. Then either tell quilt to track the files you will modify for this patch with quilt add <file> ... (in fact it just needs to be told before you save the new version of each file, because it will save the current state of the file for producing the diff later),
    2. Or use quilt edit <file> to do both things at the same time (it will use $EDITOR for editing the file);
  2. Then do your changes;
  3. Check everything is ok (it compiles, passes tests, renders, whatever);
  4. And finally record the changes (in the form of a patch) with quilt refresh.

In fact, this last 4 items (add, edit, test, refresh) can be done multiple times and they will affect the current patch.

Why do I say current? Because quilt keeps a stack of patches, in what it calls a series. Every time you do quilt new a new patch is put on top of the stack, and in the series just behind the current patch; all the other commends affect the patch that is currently on top. You can 'move' through the series with quilt pop [<patch_name>] and quilt push [<patch_name>]. if a patch name is provided, it will pop/push all the intermediate patches in the series.

How does this help with my fork? It actually does not save me from conflicts and refactorings, it just makes these problems much easier to handle. My update workflow is the following (which non incidentally mimics the one Debian Developers and Maintainers do every time they update their packages):

  1. I quilt pop -a so I go back to the pristine version, with no modifications;
  2. I git pull to get the newest version, then git tag and git checkout <last_tag> (I just want to keep in sync with releases);
  3. quilt push -a will try to apply all the patches. If one fails, quilt stops and lets me check the situation.
    1. quilt push -f will try harder to apply the troubling patch; sometimes is just a matter of too many offset lines or too much fuzzyness needed.
    2. If it doesn't apply, a .rej wil be generated and you should pick up from there.
    3. In any case, once everything is up to date, I need to run quilt refresh and the patch will be updated[1].
    4. Then I try quilt push -a again.
    5. If a patch is no longer useful (because it was fixed upstream or because it doesn't make any sense), then I can simply quilt delete <patch>; this will remove it from the series, but the patch file will still exist (unless I use the option -r[2]).

As long as I keep my patches minimal, there are big chances that they will be easier to integrate into new releases. I can even keep track of the patches in my own branch without fearing having conflicts, and allowing me to independently provide fixes upstream.

Let's see how this works in the future; at least in a couple of months I will be rendering again. Meanwhile, I will be moving to a database update workflow that includes pulling diffs from geofabrik.


[1] Being an old timer VCS user (since cvs times), I wish they would call this command update.

[2] Why not --long-options?


elevation openstreetmap utils

Posted Wed 30 Dec 2015 06:37:04 PM CET Tags: utils

El otro día dí una charlita un poquito larga de ssh/scp y screen en el IATE, el lugar donde laburo. En la misma hablo de ssh/scp básico, mas cómo poner claves públicas/privadas y cómo usar el ssh-agent para administrarlas (algo de lo que ya hablé en este glob), mas otros temas como el X11 forwarding y el agent-forwarding. Además hablé de screen, una herramineta que se complementa muy bien con ssh. Dejo las filminas acá.

security utils

Posted Wed 27 Jan 2010 11:55:55 PM CET Tags: utils

En este post veremos cómo crear pares de claves pública/privada con passphrases, y cómo usar ssh-agent para no tener que andar tipeándolas a cada rato.

Lo primero es generar el par. Hay dos tipos de pares de claves, RSA y DSA. Los pares se guardan en nuestro .ssh/, así que antes de generar un par nos fijamos qué hay allí:

mdione@cobra:~$ ls -l .ssh/
total 28
-rw-r--r-- 1 mdione mdione   190 2008-02-26 15:29 config
-rw------- 1 mdione mdione  1743 2008-09-19 16:55 id_dsa
-rw-r--r-- 1 mdione mdione   394 2008-09-19 16:55 id_dsa.pub
-rw------- 1 mdione mdione 11598 2008-09-02 10:35 known_hosts
-rw------- 1 mdione mdione  9388 2008-06-05 22:59 known_hosts.old

Hay una par DSA (id_dsa e id_dsa.pub), así que creamos un par RSA para no pisarlo:

mdione@cobra:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/mdione/.ssh/id_rsa):
*** Enter passphrase (empty for no passphrase):
*** Enter same passphrase again:
Your identification has been saved in /home/mdione/.ssh/id_rsa.
Your public key has been saved in /home/mdione/.ssh/id_rsa.pub.
The key fingerprint is:
1c:da:2b:a8:97:72:7e:ab:ff:ea:bb:e8:6b:64:4c:93 mdione@cobra
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|     .  .        |
|    E  + .       |
|   o .. S        |
|    +.   .       |
|   o... .        |
|  ..=...         |
|  .B**B*.        |
+-----------------+

Notar que en *** ssh-keygen pide dos veces una passphrase. El término passphrase viene de la época en que las passwords sólo podían ser de un maximo de 8 caracteres. La passphrase puede ser de cualquier longitud; una longitud buena está en los 15-20 caracteres. Se puede poner una passphrase nula, pero por motivos de seguridad les sugiero que no lo hagan. Si aprenden bien a usar ssh-agent van a tener que tipearla poco.

Ahora copiamos la parte pública al servidor al que queremos entrar:

mdione@cobra:~$ ssh-copy-id -i .ssh/id_rsa.pub beetroot.except.com.ar
*** mdione@beetroot.except.com.ar's password:
Now try logging into the machine, with "ssh 'beetroot.except.com.ar'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Probamos entrar al server para ver que el par ande:

mdione@cobra:~$ ssh beetroot.except.com.ar
*** Enter passphrase for key '/home/mdione/.ssh/id_rsa':
Last login: Fri Sep 19 12:18:16 2008 from ginger.except.com.ar
mdione@beetroot:~$ logout
Connection to beetroot.except.com.ar closed.

Notar que en *** pide la passphrase del par que acabamos de crear.

Ahora a usar ssh-agent. ssh-agent se inicia en toda sesión en una distro moderna, podemos verlo dando vueltas en nuestro background:

mdione@cobra:~$ ps fax | grep ssh-agent
 2997 ?        Ss     0:00                  \_ ssh-agent screen

En este caso lo tengo corriendo antes de un screen, pero en un desktop podemos ver algo de la pinta /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session x-session-manager.

Una vez que vemos el ssh-agent dando vueltas por ahí, veamos qué claves tiene:

mdione@cobra:~$ ssh-add -l
The agent has no identities.

Ok, agreguemos identidades al agente:

mdione@cobra:~$ ssh-add /home/mdione/.ssh/id_rsa
*** Enter passphrase for /home/mdione/.ssh/id_rsa:
Identity added: /home/mdione/.ssh/id_rsa (/home/mdione/.ssh/id_rsa)
mdione@cobra:~$ ssh-add -l
2048 1c:da:2b:a8:97:72:7e:ab:ff:ea:bb:e8:6b:64:4c:93 /home/mdione/.ssh/id_rsa (RSA)

En *** ssh-add nos pide la passphrase de la clave. Ok, probemos entrar al servidor:

mdione@cobra:~$ ssh beetroot.except.com.ar
Last login: Fri Sep 19 20:58:57 2008 from 200.69.231.1
mdione@beetroot:~$

No pidió la passphrase, que es lo que queríamos.

Las passphrases quedan en el ssh-agent por un período determinado de tiempo. Por defecto duran para siempre, pero con ssh-add -t <segundos> podemos cambiar eso. También podemos borrar una identidad con ssh -d <clave privada> o todas las identidades con ssh-add -D (que molesta costumbre de estas herramientas que tienen nombre de creación pero con algunas opciones de destrucción; ssh-keygen tiene una opción -R que no sólo borra claves, sino que lo hace sobre el .ssh/known-hosts, cuando el comando por sí solo actúa en los pares de claves).

En otro post mostraré cómo sacar identidades de un agent y ponerlo en otro.

utils security

Posted Wed 27 Jan 2010 11:55:55 PM CET Tags: utils