Migrating from ikiwiki to nikola

Marcos Dione — Thu, 30 Nov 2023 20:38:19 GMT

As I mentioned several times already, my ikiwiki setup for this glob is falling apart in my machine. As it is written in perl, a language I haven't touched in may many years, and its community seems to have dwindled and almost disappeared, I've been thinking of migrating to something else. As a pythonista, one obvious option is nikola. Also because I know the original developer :)

But what would it take to do this? Well, my ikiwiki posts are written in Markdown, and nikola also reads that format. At the beginning I thought of converting to reStructuredText because I have an issue: because of a bad command (probably a cp instead of rsync or tar), I lost the original file times. With reStructuredText, I can provide the date as a directive, and I can recover the original dates from archive.org's snapshots of my glob. But then I read that the same data can be put in a sidecar .meta file, so I can keep my original file format. Also, many things I wanted work best with Markdown, most notably footnotes, which, I don't know if you noticed, never worked on this glob :) Thanks +ChrisWarrick#nikola@libera.chat for all the help!

Still, ikiwiki handles a few things not very Markdown'ly, including images, code snippets and tags. To be honest, the last two are not really a part of Markdown, but it still means I have to convert one markup into another.

I had used pytest in the past, but not much really. I usually write a test() function where I test with assert everything, and once all tests pass, I call main() at script start instead. This was another quick hack, but I wanted to give it a spin. I started with some pure TDD, writing input and outputs in test functions and just assert f(input) == output and pytest did everything else for me, including showing me a diff that points out to the small errors I was making. The iteration pace was feverish.

All in all, it took me 3 21-23h hackatons to mostly finish it. I wrote one function for each step (footnotes, tags, images and code snippets), all of them looking all the input lines all over again, but it doesn't really matter, as I have to import many files by hand to specify the original publishing date. I also tested each regexp¹ individually, like I was discussing the other day². They were short enough not to follow my first tip, but by $GOD I used the other two a lot. There are another four helper functions (convert slugs to titles; convert time.timezone format to UTC offset (for instance, +0100); convert timestamps to a certain date format; and convert another date format to the same one), all also well tested. Then one short function to write the sidecar file, one that glues everything together, and one for parsing command line parameters. All that, tests and their data and all, in 538 lines of very hacky Python :) I'll try to post the code some other day, but frankly I run out of steam and I still have lots of posts to import by hand.

And that's it! Hopefully this will be the first post in the new glob version. I imported a few old posts already and it's working just fine. I expect a few tweaks in the future, as we're talking about ~300 posts and I can't promise the very old ones follow the same format. I set the feed size to one and I'll grow for the next nine posts so I don't break planets and feed readers. I hope I got that right :)

Yes, of course this includes regexps! But thanks to testing them, it didn't become the proverbial extra problem. ↩
Are you also on mastodon? Come say hi! ↩

.:: Marcos Dione/StyXman's glob ::. (Posts about pytest)

Migrating from ikiwiki to nikola