Note: this is a translation of an old post. I decided to translate it because now I'm looking for a SysAdmin position (tell your friends!) and I would like this post to show how I work.

Last Saturday I received an email from one of the guys from work with the subject «urgennnnnnnnt: heeeeeeeeeelp»[sic]. He says he was idling on Friday night when his machine stopped emiting sound via the soundcard and then it behaved erratically. When he tried rebooting it, it didn't boot anymore. «It says something about disk not bootable...».

Monday morning I go to work and go to see the machine. Precisely, it said something about «disk not bootable». I boot with a USB key with GRML and I find that the disk has no partitions.

Panic.

The guy is doing a PostDoc in something astronomical (literally) and all his work is in that machine. No backups, as usual, so I prepare myself to rescue the partitions.

In that same USB key I have a system with parted. I boot with it and I try using parted's rescue tool. Nothing. I ask the guy how the disk was partitioned, etc. He tells me that he only installed Kubuntu clicking 'Next'. Kubuntu by default creates a swap partition and an ext3 partition for / and that's it, which made what was coming relatively easy.

I reboot in GRML and I use hexdump -C /dev/sda | more to see the disk's content. This is not the first time that I juggle with partitions and MBRs, but last time I did it, I used a tool that is now discontinued (the tool was called DiskEdit, included in The Norton Utilities), which had special edit modes for MBRs, FATs, and a lot of useful things... in MS universe.

First I confirm that, yes, the first sector is a MBR (at least it has the 0x55aa signature at the end), and that the whole partition table is empty, but in the second sector of the disk there seems to be a copy. I take pen and paper, write down what I found, but it turns out not only I have half the data, the partition I thought I found was too small.

So I decide to look for the partition by hand. To do that I needed to find out first how does the ext3 kernel code know wether a partition is ext3 or not. I knew it would be some kind of magic signature, but I had no idea which. So I installed the sources for 2.6.29 in my laptop and started to look at ext3's code. After going around a lot, including reading the code that is excuted when you mount a filesystem of type ext3, where we can see that it uses a magic signature[3] and the structure of the ext3 superblock, where we can see the magic's offset is 0x38.

So the problem of finding an ext3 partition is reduced to the problem of finding 0x53ef (damn little endian) at a sector's offset 0x38 in the disk. Luckily more has a find tool, so I sit down to search every occurrence of 53 ef, hoping that the address at the left ends in 30 and that they would be the 9th and 10th bytes in the line (damn 0 based offsets).

A few 'next' after, I get my first candidate. It looks good, because I was also comparing my findings with a similar dump from my USB key (which I have formatted as ext2, and luckily ext2 and ext3 share those structures), and also I spot something that looks like a uuid.

This candidate's address is 0x80731038. I substract 0x38 and I get the address 0x80731000, a nice round number for a superblock. Converted to decimal that's 2.155.024.384, some 2GiB from the disk's begginning. Looks really good! The swap partition could be before the root one, and could have that size.

I use fdisk /dev/sda to see the (still empty) partition table. It says there's 16.065 sectors per cylinder, times 512 bytes per sector, equals 8.225.280 bytes per cylinder. Almost all distros (actually I think all of them) partition disks at cylinder boundaries[1], so if the sector I found is right at the beginning of a cylinder...

I divide 2.155.024.384/8.225.280=...

(suspense pause)[2]

262.000124494...

¡Damn! I almost had it... Hmm, how much is the factional part? (262.000124494-262)*8.225.280=... ¡1024! ¿Is it that...?

I run strace debugfs -R show_super_stats /dev/sdb1 (the partition in my USB key) and I see that it actually seeks 1024 bytes within the partition for reading the superblock!

This is it. With 262 in my head, I fire fdisk /dev/sda and I create two partitions: swap in cylinders 1-261 and root from cylinder 262 till the end. I save, cross my fingers and I run debugfs -R show_super_stats /dev/sda1. It fails! What's wrong? I reboot and I try again, just in case the kernel did not re-read correctly the partition table. It fails again. WTF?

Ah, duh, it's sda2, where do I have my head... Ok, debugfs -R show_super_stats /dev/sda2... it works, the sonofabitch works! I can't believe it. I risk it: fsck -n /dev/sda2. «Filesystem is clean». Damn, I try harder: fsck -n -f /dev/sda2...

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda2 etc etc...

It's fine! But the MBR doesn't have GURB installed, so I do the usual GRUB reinstall process, I reboot...

It boots like nothing has happened, and it finishes in a beautiful login. Satisifed, I pat myself in the back, pack my things and I start my weekend.

sysadmin rescue


[1] ... wasting some 8MiB between the MBR and the first partition.

[2] The sharp ones reading this will notice that this can not give an integer by no means.

[3] Reiser magics are funny. Looks like he started the fad that now AdOlEsCeNtS use.

Posted Thu 07 Apr 2011 11:46:15 PM CEST Tags: tags/rescue
09

After several months thinking about it, and just two requests, I finally decided to publish satyr's code. I decided to use github because I already switched to satyr from hg to git, mainly for testing and understanding it. I think I can live with hg, althought branch management in git seems to be given more thought and a good implementation.

So, without further ado: satyr in github

Remember, it's still a test software, by no means polished or ready for human consumption, and with very low development force. Still, I think it has some nice features, like interchangeable skins and a well defined backend, D-Bus support, quick tag edition, reasonable collection managment, and, thanks to Phonon, almost-gapless playback and things like «stop after playing the current this file» (but not «after any given file» yet).

In Debian Sid it mostly works only with the GStreamer backend; I haven't tried the xine one and I know VLC does not emit a signal needed for queueing the next song, so you have to press «next» after each song. AFAIK this is fixed upstream.

satyr pykde python

Posted Sat 03 Sep 2011 02:59:43 PM CEST Tags:
Posted Wed 06 Apr 2011 08:28:18 PM CEST
01
Posted Wed 06 Apr 2011 08:28:18 PM CEST
05
Posted Wed 06 Apr 2011 08:28:18 PM CEST
08
Posted Wed 06 Apr 2011 08:28:18 PM CEST

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:

Yesterday[1] I wrote: «Then I found out that it wouldn't be easy to use the tiles [generated by CloudMade] with marble». How wrong was I.

Here you can read how to access the tiles in CM's servers. The only thing you need is an API key, which you can get once you registered (it practically begs you to get one :). Another piece of the puzzle is marble itself. I was specting to copy OSM's service description file and bend it and twist it until it worked with CM's tiles. But things are, luckily, much more simple.

In marble you simply do «File», «Create a new map...», then select «Online map providing indexed tiles», which clearly describes CM. Next, you have to provide and URL template, which you can simply derive from the doc in CM's site. Mine's like this:

http://tile.cloudmade.com/5f806ad32bb44b38a464020fa2223193/51084/256/{zoomLevel}/{x}/{y}.png

You, of course, will have to change the API key and the style number for yours. One thing I found is that the @2x doesn't seem to work with marble; without it it works just fine. Maybe it's a User-Agent thing. Then you keep answering what marble asks you in the wizard. By the end of it, you'll have marble working with your tiles! It's amazing.

One extra hack was to add support for hillshading. These are tiles that are alpha-blended with OSM's, wich gives you the impression of viewing a 3D map (bah, at least in the covered regions with mountains). The service files (.dgml files) are actually XML files quite easy to edit if you defocus and just look at the big picture ot if. So I just simply opened both files (OSM's is in /usr/share/kde4/apps/marble/data/maps/earth/openstreetmap/openstreetmap.dgml, the one you created in $HOME/.local/share/marble/maps/earth/foo/foo.dgml), copied the texture tag that describes hillshading, pasted it in my .dgml file, saved, reopened marble and... voilà! Even better, as hillshading is actually another tile server, marble can share it for OSM's, OSMaRenderer and yours.

The final result is simply amazing. You can see a screenshot here.

More on my use of marble and OSM pretty soon.


[1] When I say "yesterday", I actually mean "almost 7 months ago". That's what took me to fix my glob and come back to writing, finishing this post.


openstreetmap marble

Posted Wed 25 Jul 2012 03:35:42 PM CEST Tags:

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:
Posted Thu 29 Dec 2011 10:35:03 PM CET

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:
Posted Thu 29 Dec 2011 10:34:58 PM CET

On November 26, the Qt/KDE Debian team (qkd from now on) released an experimental packaging of KDE SC 4.7. This means a substantial and very much awaited upgrade from the actual version available in sid, 4.6.5 . From what I've been told, it has taken so long because there have been a lot of changes in source code structure (KDE has recently restructured its source modularization) and in packaging itself, and even with recent aditions to the group, the resources in terms of developers is still low. In any case, thanks to them for getting this version out, I really appreciate your hard work.

But because of this lack of resources, they only got out packages for the amd64 architecture, and I'm running i386 at home, so I decided to try and compile it by myself. This is the log of the attempt.

First, I added deb-src http://qt-kde.debian.net/debian experimental-snapshots main to my sources.list and got the code for kde4libs, the package I think is the root of the whole repo[1][2]:

mdione@mustang:~/src/system/debian$ apt-get source kde4libs
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'kde4libs' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/pkg-kde/kde-sc/kde4libs.git
Need to get 12.4 MB of source archives.
Get:1 http://qt-kde.debian.net/debian/ experimental-snapshots/main kde4libs 4:4.7.2-0r3 (dsc) [4,883 B]
Get:2 http://qt-kde.debian.net/debian/ experimental-snapshots/main kde4libs 4:4.7.2-0r3 (tar) [12.1 MB]
Get:3 http://qt-kde.debian.net/debian/ experimental-snapshots/main kde4libs 4:4.7.2-0r3 (diff) [334 kB]
Fetched 12.4 MB in 1min 54s (109 kB/s)
gpgv: Signature made Sat 22 Oct 2011 02:29:45 AM CEST using RSA key ID 73A85F31
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./kde4libs_4.7.2-0r3.dsc
dpkg-source: info: extracting kde4libs in kde4libs-4.7.2
dpkg-source: info: unpacking kde4libs_4.7.2.orig.tar.bz2
dpkg-source: info: unpacking kde4libs_4.7.2-0r3.debian.tar.gz
dpkg-source: info: applying kconf_update_migrate_from_kde3_icon_theme.diff
dpkg-source: info: applying add_debian_build_type.diff
dpkg-source: info: applying disable_usr_lib_install_rpath.diff
dpkg-source: info: applying make_libkdeinit4_private.diff
dpkg-source: info: applying default_kde4_xdg_menu_prefix.diff
dpkg-source: info: applying qt4_designer_plugins_path.diff
dpkg-source: info: applying hardcode_ptm_device.diff
dpkg-source: info: applying kfreebsd_support.diff
dpkg-source: info: applying debian_menu.diff
dpkg-source: info: applying findservicebydesktoppath_try_realfilepath.diff
dpkg-source: info: applying findqt4_optional_x11_pthread.diff
dpkg-source: info: applying use_dejavu_as_default_font.diff
dpkg-source: info: applying hack_in_etc_kde4_in_kstandarddirs.diff
dpkg-source: info: applying ld_exclude_libs_qtuitools.diff
dpkg-source: info: applying konsole_kfreebsd_fix.diff
dpkg-source: info: applying hurd_support.diff
dpkg-source: info: applying kfileshare_kdesu_fileshareset.diff
dpkg-source: info: applying relax_plugin_kde_version_check.diff
dpkg-source: info: applying add_dlrestrictions_support.diff
dpkg-source: info: applying findpythonlibrary_layout_deb_on_debian.diff
dpkg-source: info: applying ktar_header_checksum_fix.diff
dpkg-source: info: applying ktar_longlink_length_in_bytes.diff
dpkg-source: info: applying nepomuk_unicode.diff

Nice, now get the build-deps:

mdione@mustang:~/src/system/debian$ sudo apt-get build-dep kde4libs
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Build-Depends dependency for kde4libs cannot be satisfied because candidate version of package shared-desktop-ontologies can't satisfy version requirements

Dang, I have an old shared-desktop-ontologies (v. 0.6.x in sid), and peeking in qkd's repo I find no new version of it. The answer must be in experimental, so I add deb http://ftp.nl.debian.org/debian/ experimental main to my sources.list and run apt-get update. Now it must be just a matter of:

mdione@mustang:~/src/system/debian$ sudo apt-get install -t experimental shared-desktop-ontologies
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  shared-desktop-ontologies
1 upgraded, 0 newly installed, 0 to remove and 103 not upgraded.
Need to get 129 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Get:1 http://ftp.nl.debian.org/debian/ experimental/main shared-desktop-ontologies all 0.8.0-1 [129 kB]
Fetched 129 kB in 1s (92.7 kB/s)
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
Reading changelogs... Done
apt-listchanges: Mailing root: apt-listchanges: changelogs for mustang
(Reading database ... 218251 files and directories currently installed.)
Preparing to replace shared-desktop-ontologies 0.6.0-1 (using .../shared-desktop-ontologies_0.8.0-1_all.deb) ...
Unpacking replacement shared-desktop-ontologies ...
Setting up shared-desktop-ontologies (0.8.0-1) ...

And I try again:

mdione@mustang:~/src/system/debian$ sudo apt-get build-dep kde4libs
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  hspell libacl1-dev libattica-dev libattr1-dev libdbusmenu-qt-dev libdlrestrictions-dev libpolkit-qt-1-dev libqca2-dev libstreamanalyzer-dev libstreams-dev
  libutempter-dev pkg-kde-tools
0 upgraded, 12 newly installed, 0 to remove and 8 not upgraded.
Need to get 1,170 kB of archives.
After this operation, 3,551 kB of additional disk space will be used.
Do you want to continue [Y/n]?
[...]

We're set! now to fire the compilation itself. The method I use must not be the best one, but I remember it by heart and works for me :)

mdione@mustang:~/src/system/debian$ cd kde4libs-4.7.2/
mdione@mustang:~/src/system/debian/kde4libs-4.7.2$ nice -n 19 fakeroot debian/rules binary

Notice I use nice to not hog too much my computer and fakeroot. I don't remember why I must use it :| , but I remember I must. Later, asking in #debian-devel I was pointed to dpkg-buildpackage, to which you can give the -j# option, which makes the package to be compiled with # processes in parallel (actually it only works if the compilation is make based, as is in this case), so from now on I'll use that.

The resulting packages are these:

kdelibs5-data_4.7.2-0r3_all.deb
kdelibs5-dbg_4.7.2-0r3_i386.deb
kdelibs5-dev_4.7.2-0r3_i386.deb
kdelibs5-plugins_4.7.2-0r3_i386.deb
kdelibs-bin_4.7.2-0r3_i386.deb
kdoctools_4.7.2-0r3_i386.deb
libkcmutils4_4.7.2-0r3_i386.deb
libkde3support4_4.7.2-0r3_i386.deb
libkdeclarative5_4.7.2-0r3_i386.deb
libkdecore5_4.7.2-0r3_i386.deb
libkdesu5_4.7.2-0r3_i386.deb
libkdeui5_4.7.2-0r3_i386.deb
libkdewebkit5_4.7.2-0r3_i386.deb
libkdnssd4_4.7.2-0r3_i386.deb
libkemoticons4_4.7.2-0r3_i386.deb
libkfile4_4.7.2-0r3_i386.deb
libkhtml5_4.7.2-0r3_i386.deb
libkidletime4_4.7.2-0r3_i386.deb
libkimproxy4_4.7.2-0r3_i386.deb
libkio5_4.7.2-0r3_i386.deb
libkjsapi4_4.7.2-0r3_i386.deb
libkjsembed4_4.7.2-0r3_i386.deb
libkmediaplayer4_4.7.2-0r3_i386.deb
libknewstuff2-4_4.7.2-0r3_i386.deb
libknewstuff3-4_4.7.2-0r3_i386.deb
libknotifyconfig4_4.7.2-0r3_i386.deb
libkntlm4_4.7.2-0r3_i386.deb
libkparts4_4.7.2-0r3_i386.deb
libkprintutils4_4.7.2-0r3_i386.deb
libkpty4_4.7.2-0r3_i386.deb
libkrosscore4_4.7.2-0r3_i386.deb
libkrossui4_4.7.2-0r3_i386.deb
libktexteditor4_4.7.2-0r3_i386.deb
libkunitconversion4_4.7.2-0r3_i386.deb
libkutils4_4.7.2-0r3_i386.deb
libnepomuk4_4.7.2-0r3_i386.deb
libnepomukquery4a_4.7.2-0r3_i386.deb
libnepomukutils4_4.7.2-0r3_i386.deb
libplasma3_4.7.2-0r3_i386.deb
libsolid4_4.7.2-0r3_i386.deb
libthreadweaver4_4.7.2-0r3_i386.deb

Next package is kadebase. Again, we find the missing deps with:

mdione@mustang:~/src/system/debian$ sudo apt-get build-dep kdebase
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Build-Depends dependency for kdebase cannot be satisfied because candidate version of package kde-sc-dev-latest can't satisfy version requirements

The message is simlar to the one about shared-desktop-ontologies, but I have not even the slightest idea which one of the packages above is the one responsible for it, so I'll just install everything, as most probably they're build-deps for most of the remaining packages (it's kde5libs, after all):

mdione@mustang:~/src/system/debian$ sudo dpkg -i *.deb
(Reading database ... 218481 files and directories currently installed.)
[...]

After that I try again, but still the same thing. Poking around I find that a package with that name exists (it's the first time I met him, enchanté), so I get the source in the usual way:

mdione@mustang:~/src/system/debian$ apt-get source meta-kde
Reading package lists... Done
Building dependency tree
Reading state information... Done
Need to get 14.0 kB of source archives.
Get:1 http://qt-kde.debian.net/debian/ experimental-snapshots/main meta-kde 5:71~pre15 (dsc) [2,098 B]
Get:2 http://qt-kde.debian.net/debian/ experimental-snapshots/main meta-kde 5:71~pre15 (tar) [11.9 kB]
Fetched 14.0 kB in 0s (49.3 kB/s)
gpgv: Signature made Mon 21 Nov 2011 09:38:12 PM CET using RSA key ID 73A85F31
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./meta-kde_71~pre15.dsc
dpkg-source: info: extracting meta-kde in meta-kde-71~pre15
dpkg-source: info: unpacking meta-kde_71~pre15.tar.gz

And to compile:

mdione@mustang:~/src/system/debian$ ( cd meta-kde-71~pre15 && nice -n 19 dpkg-buildpackage -j3 -b -us -uc )
dpkg-buildpackage: source package meta-kde
dpkg-buildpackage: source version 5:71~pre15
dpkg-buildpackage: source changed by Debian Qt/KDE Maintainers <debian-qt-kde@lists.debian.org>
dpkg-buildpackage: host architecture i386
[...]

That generates several packages, but I just install the one I want:

mdione@mustang:~/src/system/debian$ sudo dpkg -i kde-sc-dev-latest_4.7.2+5.71~pre15_all.deb
Selecting previously unselected package kde-sc-dev-latest.
(Reading database ... 218469 files and directories currently installed.)
Unpacking kde-sc-dev-latest (from kde-sc-dev-latest_4.7.2+5.71~pre15_all.deb) ...
Setting up kde-sc-dev-latest (4:4.7.2+5.71~pre15) ...

Now the build-deps again:

mdione@mustang:~/src/system/debian$ sudo apt-get build-dep kdebase
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  libkatepartinterfaces4
The following NEW packages will be installed:
  libqimageblitz-dev libtidy-dev
0 upgraded, 2 newly installed, 1 to remove and 8 not upgraded.
Need to get 205 kB of archives.
After this operation, 1,932 kB disk space will be freed.
Do you want to continue [Y/n]?
[...]

You'll probably won't notice (as I didn't) that it has removed a package, libkatepartinterfaces. This is because in my system I have the old version of it and there seems to be some kind of conflict somewhere. I just cross my fingers it won't break much[3].

Now get the sources and compile:

mdione@mustang:~/src/system/debian$ apt-get source kdebase
Reading package lists... Done
Building dependency tree
Reading state information... Done
NOTICE: 'kdebase' packaging is maintained in the 'Git' version control system at:
git://git.debian.org/pkg-kde/kde-sc/kdebase.git
Need to get 2,721 kB of source archives.
Get:1 http://qt-kde.debian.net/debian/ experimental-snapshots/main kdebase 4:4.7.1-0r2 (dsc) [3,217 B]
Get:2 http://qt-kde.debian.net/debian/ experimental-snapshots/main kdebase 4:4.7.1-0r2 (tar) [2,685 kB]
Get:3 http://qt-kde.debian.net/debian/ experimental-snapshots/main kdebase 4:4.7.1-0r2 (diff) [33.1 kB]
Fetched 2,721 kB in 26s (105 kB/s)
gpgv: Signature made Tue 27 Sep 2011 09:43:53 PM CEST using RSA key ID 73A85F31
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./kdebase_4.7.1-0r2.dsc
dpkg-source: info: extracting kdebase in kdebase-4.7.1
dpkg-source: info: unpacking kdebase_4.7.1.orig.tar.bz2
dpkg-source: info: unpacking kdebase_4.7.1-0r2.debian.tar.gz
dpkg-source: info: applying enable_debianabimanager.diff
dpkg-source: info: applying enable_dlrestrictions.diff

mdione@mustang:~/src/system/debian$ ( cd kdebase-4.7.1 && nice -n 19 dpkg-buildpackage -j3 -b -us -uc )
[...]

Installing the resulting packages is another history, as some of them depend on kde-runtime, so it's better if you compile kde-runtime before[4]. Trying to install its build-deps, you realize taht you actually need to compile soprano first.

You get the idea. Now I'll just show the generated packages I installed by hand after each compilation, but only those that suited my needs:

soprano:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
soprano-daemon_2.7.3+dfsg.1-0r0_i386.deb \
libsoprano4_2.7.3+dfsg.1-0r0_i386.deb \
libsoprano-dev_2.7.3+dfsg.1-0r0_i386.deb

kde-runtime:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
kdebase-runtime_4.7.2-0r3_all.deb \
khelpcenter4_4.7.2-0r3_i386.deb \
kde-config-phonon-xine_4.7.2-0r3_i386.deb \
kde-runtime-data_4.7.2-0r3_all.deb \
kde-runtime_4.7.2-0r3_i386.deb \
plasma-scriptengine-javascript_4.7.2-0r3_i386.deb

While installing kde-runtime-data I had to remove kdebase-runtime-data:

[...]
Selecting previously unselected package kde-runtime-data.
dpkg: regarding kde-runtime-data_4.7.2-0r3_all.deb containing kde-runtime-data:
 kde-runtime-data breaks kdebase-runtime-data (<< 4:4.7.2)
  kdebase-runtime-data (version 4:4.6.5-1) is present and installed.
[...]

mdione@mustang:~/src/system/debian$ sudo dpkg --remove kdebase-runtime-data
(Reading database ... 218612 files and directories currently installed.)
Removing kdebase-runtime-data ...
[...]

kdebase:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
dolphin_4.7.1-0r2_i386.deb \
kdebase-bin_4.7.1-0r2_i386.deb \
kdebase-data_4.7.1-0r2_all.deb \
kdepasswd_4.7.1-0r2_i386.deb \
konq-plugins_4.7.1-0r2_i386.deb \
konqueror_4.7.1-0r2_i386.deb \
konqueror-nsplugins_4.7.1-0r2_i386.deb \
libkonq5abi1_4.7.1-0r2_i386.deb \
libkonq-common_4.7.1-0r2_i386.deb \
plasma-widget-folderview_4.7.1-0r2_i386.deb

While trying to get akonadi I got this error message:

Failed to fetch http://qt-kde.debian.net/debian/pool/main/a/akonadi/akonadi_1.6.2-0r1.dsc  Hash Sum mismatch

Checking the MD5, SHA1 and SHA256 checksums and comparing to the ones in the .dsc file revealed no difference. lisandro told me to use dget instead:

mdione@mustang:~/src/system/debian$ dget http://qt-kde.debian.net/debian/pool/main/a/akonadi/akonadi_1.6.2-0r1.dsc
dget: retrieving http://qt-kde.debian.net/debian/pool/main/a/akonadi/akonadi_1.6.2-0r1.dsc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2620  100  2620    0     0   6842      0 --:--:-- --:--:-- --:--:--  8161
dget: using existing akonadi_1.6.2.orig.tar.bz2
dget: using existing akonadi_1.6.2-0r1.debian.tar.gz
akonadi_1.6.2-0r1.dsc:
dscverify: akonadi_1.6.2-0r1.dsc failed signature check:
gpg: Signature made Thu 03 Nov 2011 08:55:52 AM CET using RSA key ID 73A85F31
gpg: Can't check signature: public key not found
Validation FAILED!!

Unpacking by hand:

mdione@mustang:~/src/system/debian$ dpkg-source -x akonadi_1.6.2-0r1.dsc
gpgv: Signature made Thu 03 Nov 2011 08:55:52 AM CET using RSA key ID 73A85F31
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./akonadi_1.6.2-0r1.dsc
dpkg-source: info: extracting akonadi in akonadi-1.6.2
dpkg-source: info: unpacking akonadi_1.6.2.orig.tar.bz2
dpkg-source: info: unpacking akonadi_1.6.2-0r1.debian.tar.gz
dpkg-source: info: applying x11_not_required.diff

akonadi:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
akonadi-server_1.6.2-0r1_i386.deb \
akonadi-backend-sqlite_1.6.2-0r1_i386.deb \
libakonadi-dev_1.6.2-0r1_i386.deb \
libakonadiprotocolinternals1_1.6.2-0r1_i386.deb

Here I cheated: I really didn't want to spend 10 minutes of iterative attempts to find the minimal set of lib packages to install:

libkdepim:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
kdepimlibs5-dev_4.7.2-0r1_i386.deb \
kdepimlibs-kio-plugins_4.7.2-0r1_i386.deb \
lib*.deb

kde-wallpapers:

mdione@mustang:~/src/system/debian$ sudo dpkg -i \
kde-wallpapers-default_4.7.2-0r0_all.deb

kde-workspace:

sudo dpkg -i --auto-deconfigure \
systemsettings_4.7.2-0r7_i386.deb \
plasma-desktop_4.7.2-0r7_i386.deb \
plasma-scriptengine-python_4.7.2-0r7_all.deb \
plasma-widgets-workspace_4.7.2-0r7_i386.deb \
plasma-dataengines-workspace_4.7.2-0r7_i386.deb \
klipper_4.7.2-0r7_i386.deb \
kdm_4.7.2-0r7_i386.deb \
kde-workspace-bin_4.7.2-0r7_i386.deb \
kde-workspace-data_4.7.2-0r7_all.deb \
kde-window-manager_4.7.2-0r7_i386.deb \
kdebase-workspace_4.7.2-0r7_all.deb \
kdebase-workspace-bin_4.7.2-0r7_all.deb \
libkworkspace4_4.7.2-0r7_i386.deb \
kde-workspace-kgreet-plugins_4.7.2-0r7_i386.deb \
kde-style-oxygen_4.7.2-0r7_i386.deb \
libkdecorations4_4.7.2-0r7_i386.deb \
libkephal4abi1_4.7.2-0r7_i386.deb \
libkwineffects1abi2_4.7.2-0r7_i386.deb \
libkworkspace4_4.7.2-0r7_i386.deb \
libplasmagenericshell4_4.7.2-0r7_i386.deb \
libtaskmanager4abi2_4.7.2-0r7_i386.deb \
libplasmaclock4abi2_4.7.2-0r7_i386.deb \
libksgrd4_4.7.2-0r7_i386.deb \
libplasma-geolocation-interface4_4.7.2-0r7_i386.deb \
libsolidcontrol4abi2_4.7.2-0r7_i386.deb \
libprocesscore4abi1_4.7.2-0r7_i386.deb \
libweather-ion6_4.7.2-0r7_i386.deb \
libkscreensaver5_4.7.2-0r7_i386.deb \
libprocessui4a_4.7.2-0r7_i386.deb \
libsolidcontrolifaces4abi2_4.7.2-0r7_i386.deb \
kde-workspace_4.7.2-0r7_all.deb \
ksysguard_4.7.2-0r7_i386.deb \
freespacenotifier_4.7.2-0r7_i386.deb \
libksignalplotter4_4.7.2-0r7_i386.deb \ 
ksysguardd_4.7.2-0r7_i386.deb

I'll just finish this post here. The rest is just more of the same. The final list of packages a compile in right order is:

kde4libs
meta-kde
soprano
kde-runtime
kdebase
akonadi
prison
kdepimlibs
kde-wallpapers
kde-workspace

All in all, this took some 10 hours of finding deps, compiling and installing, and 6.4 GiB between original .tar.gz files, compilation dirs and generated packages.


[1] It's nice that I don't need to compile libqt4*, those take ages[5].

[2] Later lisandro from qkd pointed me to the dependency graph that confirms my guess.

[3] I did all this while running a KDE SC 4.6.5 session, and even writing this post in kate. Thanks to Linux and Debian not much actually broke, I only lost the ability to browse with konqueror after I installed the first batch of packages because KIO could not instantiate an HTTP/HTTPS ioslave anymore.

[4] Note that the dependency graph two notes above is actually a build dependency graph.

[5] Not that this compilation was fast. kdelibs without parallelism took some 3h and kdepimlibs and kde-workspace are two huge beasts.


debian kde

Posted Sun 27 Nov 2011 11:18:21 PM CET Tags:

After several months thinking about it, and just two requests, I finally decided to publish satyr's code. I decided to use github because I already switched to satyr from hg to git, mainly for testing and understanding it. I think I can live with hg, althought branch management in git seems to be given more thought and a good implementation.

So, without further ado: satyr in github

Remember, it's still a test software, by no means polished or ready for human consumption, and with very low development force. Still, I think it has some nice features, like interchangeable skins and a well defined backend, D-Bus support, quick tag edition, reasonable collection managment, and, thanks to Phonon, almost-gapless playback and things like «stop after playing the current this file» (but not «after any given file» yet).

In Debian Sid it mostly works only with the GStreamer backend; I haven't tried the xine one and I know VLC does not emit a signal needed for queueing the next song, so you have to press «next» after each song. AFAIK this is fixed upstream.

satyr pykde python

Posted Sat 03 Sep 2011 02:59:43 PM CEST Tags:

Note: this is a translation of an old post. I decided to translate it because now I'm looking for a SysAdmin position (tell your friends!) and I would like this post to show how I work.

Last Saturday I received an email from one of the guys from work with the subject «urgennnnnnnnt: heeeeeeeeeelp»[sic]. He says he was idling on Friday night when his machine stopped emiting sound via the soundcard and then it behaved erratically. When he tried rebooting it, it didn't boot anymore. «It says something about disk not bootable...».

Monday morning I go to work and go to see the machine. Precisely, it said something about «disk not bootable». I boot with a USB key with GRML and I find that the disk has no partitions.

Panic.

The guy is doing a PostDoc in something astronomical (literally) and all his work is in that machine. No backups, as usual, so I prepare myself to rescue the partitions.

In that same USB key I have a system with parted. I boot with it and I try using parted's rescue tool. Nothing. I ask the guy how the disk was partitioned, etc. He tells me that he only installed Kubuntu clicking 'Next'. Kubuntu by default creates a swap partition and an ext3 partition for / and that's it, which made what was coming relatively easy.

I reboot in GRML and I use hexdump -C /dev/sda | more to see the disk's content. This is not the first time that I juggle with partitions and MBRs, but last time I did it, I used a tool that is now discontinued (the tool was called DiskEdit, included in The Norton Utilities), which had special edit modes for MBRs, FATs, and a lot of useful things... in MS universe.

First I confirm that, yes, the first sector is a MBR (at least it has the 0x55aa signature at the end), and that the whole partition table is empty, but in the second sector of the disk there seems to be a copy. I take pen and paper, write down what I found, but it turns out not only I have half the data, the partition I thought I found was too small.

So I decide to look for the partition by hand. To do that I needed to find out first how does the ext3 kernel code know wether a partition is ext3 or not. I knew it would be some kind of magic signature, but I had no idea which. So I installed the sources for 2.6.29 in my laptop and started to look at ext3's code. After going around a lot, including reading the code that is excuted when you mount a filesystem of type ext3, where we can see that it uses a magic signature[3] and the structure of the ext3 superblock, where we can see the magic's offset is 0x38.

So the problem of finding an ext3 partition is reduced to the problem of finding 0x53ef (damn little endian) at a sector's offset 0x38 in the disk. Luckily more has a find tool, so I sit down to search every occurrence of 53 ef, hoping that the address at the left ends in 30 and that they would be the 9th and 10th bytes in the line (damn 0 based offsets).

A few 'next' after, I get my first candidate. It looks good, because I was also comparing my findings with a similar dump from my USB key (which I have formatted as ext2, and luckily ext2 and ext3 share those structures), and also I spot something that looks like a uuid.

This candidate's address is 0x80731038. I substract 0x38 and I get the address 0x80731000, a nice round number for a superblock. Converted to decimal that's 2.155.024.384, some 2GiB from the disk's begginning. Looks really good! The swap partition could be before the root one, and could have that size.

I use fdisk /dev/sda to see the (still empty) partition table. It says there's 16.065 sectors per cylinder, times 512 bytes per sector, equals 8.225.280 bytes per cylinder. Almost all distros (actually I think all of them) partition disks at cylinder boundaries[1], so if the sector I found is right at the beginning of a cylinder...

I divide 2.155.024.384/8.225.280=...

(suspense pause)[2]

262.000124494...

¡Damn! I almost had it... Hmm, how much is the factional part? (262.000124494-262)*8.225.280=... ¡1024! ¿Is it that...?

I run strace debugfs -R show_super_stats /dev/sdb1 (the partition in my USB key) and I see that it actually seeks 1024 bytes within the partition for reading the superblock!

This is it. With 262 in my head, I fire fdisk /dev/sda and I create two partitions: swap in cylinders 1-261 and root from cylinder 262 till the end. I save, cross my fingers and I run debugfs -R show_super_stats /dev/sda1. It fails! What's wrong? I reboot and I try again, just in case the kernel did not re-read correctly the partition table. It fails again. WTF?

Ah, duh, it's sda2, where do I have my head... Ok, debugfs -R show_super_stats /dev/sda2... it works, the sonofabitch works! I can't believe it. I risk it: fsck -n /dev/sda2. «Filesystem is clean». Damn, I try harder: fsck -n -f /dev/sda2...

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda2 etc etc...

It's fine! But the MBR doesn't have GURB installed, so I do the usual GRUB reinstall process, I reboot...

It boots like nothing has happened, and it finishes in a beautiful login. Satisifed, I pat myself in the back, pack my things and I start my weekend.

sysadmin rescue


[1] ... wasting some 8MiB between the MBR and the first partition.

[2] The sharp ones reading this will notice that this can not give an integer by no means.

[3] Reiser magics are funny. Looks like he started the fad that now AdOlEsCeNtS use.

Posted Thu 07 Apr 2011 11:46:15 PM CEST Tags: tags/rescue
09

After several months thinking about it, and just two requests, I finally decided to publish satyr's code. I decided to use github because I already switched to satyr from hg to git, mainly for testing and understanding it. I think I can live with hg, althought branch management in git seems to be given more thought and a good implementation.

So, without further ado: satyr in github

Remember, it's still a test software, by no means polished or ready for human consumption, and with very low development force. Still, I think it has some nice features, like interchangeable skins and a well defined backend, D-Bus support, quick tag edition, reasonable collection managment, and, thanks to Phonon, almost-gapless playback and things like «stop after playing the current this file» (but not «after any given file» yet).

In Debian Sid it mostly works only with the GStreamer backend; I haven't tried the xine one and I know VLC does not emit a signal needed for queueing the next song, so you have to press «next» after each song. AFAIK this is fixed upstream.

satyr pykde python

Posted Sat 03 Sep 2011 02:59:43 PM CEST Tags:
Posted Wed 06 Apr 2011 08:28:18 PM CEST
01
Posted Wed 06 Apr 2011 08:28:18 PM CEST
05
Posted Wed 06 Apr 2011 08:28:18 PM CEST
08
Posted Wed 06 Apr 2011 08:28:18 PM CEST
Posted Wed 06 Apr 2011 08:28:18 PM CEST
06
Posted Wed 06 Apr 2011 08:28:18 PM CEST
03

Last year and a half I was working in research. This position was about, among other things, to port a programming language (two, actually, Hop and Bigloo) to the Antroid platform. I already had written something about it, but this time I want to show my high level impressions of the platform. What follows is part of a report I wrote at the end of that job, which includes the things I wanted to say.

The Android port can be viewed as four separate sub-tasks. Hop is software developed in the Scheme language; more particularly, it must be compiled with the Bigloo Scheme compiler, which in turn uses gcc for the final compilation. That means that we also needed to port Bigloo first to the platform, not because we were planning to use it in the platform, but because we need the Bigloo runtime libraries ported to Android, as Hop and any other program compiled with Bigloo uses them. The other three subtasks, which are discussed later, are porting Hop itself; developing libraries to access devices and other features present in the platform; and, we'll see later the reasons, make the port work with threads.

When we started to investigate how to port native code to the platform we found that there wasn't much support. At fisrt the only documentation we could find was blog posts of people trying to do it by hand. They were using the compiler provided in Android's source code to compile static binaries that could be run on the platform. Because Bigloo uses dinamic libraries to implement platform dependent code and modules, we aimed to find a way to compile things dinamically. After 3 or 4 weeks we found a wrapper written in Ruby that managed all the details of calling gcc with the proper arguments. With this we should be able to port anything that uses gcc as the compiler, just like Bigloo does. At the same time, the first version of Android's NDK (Native Development Kit) appeared, but it wasn't easy to integrate in our build system.

(Note: Actually I think most of the problems we faced doing this port stem from this. The NDK forces you to write a new set of Makefiles, but our hand-made build system and build hierarchy made such an effort quite big. Also, that mean supporting a parallel build system, while it should not be so crazy to spect a cleaner way to integrate the toolchain into an existing build system, not only in hand-made like in this case, but also the most common ones, like autotools, cmake, etc.)

Even having the proper compiler, we found several obstacles related to the platform itself. First of all, Bigloo relies heavily on the C and pthread libraries to implement lowlevel functionality. Bigloo can use both glibc, GNU's implementation, or µlibc, an implementation aimed for embedded aplications. Bigloo also relies on Boehm's Garbage Collector (GC) for its memory management. The C library implementation in Android is not the glibc or the µlibc, but an implementation developed by Google for the platform, called Bionic. This version of the C library is tailored to the platform's need, with little to no regards to native application development.

The first problem we found is that GC compiled fine with Bionic, but the apllications that used GC did not link: there was a missing symbol that normally is defined in the libc, but that Bionic did not. We tried cooperating with the GC developers, we tried inspecting a Mono port to Android, given that this project also uses GC, trying to find a solution that could be useful for everyone, but at the end we just patched our sources to fake such symbol with a value that remotely made sense.

We also found that Bionic's implementation of pthreads is not only incomplete, but also has some glitches. For instance, in our build system, we test the existence of a function like everybody else: we compile a small test program wich uses it. With this method we found at least one function that is declared but never defined. That means that Bionic declares that the function exists, but then it never implements it. Another example is the declaration and definition of a function, but the lack of definition of constants normally used when calling this function.

Also, because most of the tests also must be executed to inform about the peculiarities of each implementation, we had to change our build system to be able to execute the produced binaries in the Android emulator.

Google also decided to implement their own set of tools, again, trimmed down to the needs of the platform, instead of using old and proven versions, like Busybox. This means that some tools behave differently, with no documentation about it, so we mostly had to work around this differences everytime a new one apperared.

All in all, we spent two and a half months just getting Bigloo to run in Android, dismissing the problem that Boehm's GC, using its own build system, detected that the compiler declared to not support threads, and refused to compile with threads enabled. This meant that Bigloo itself could not be compiled with pthreads support.

With this caveat in mind, we tackled the second subtask, porting Hop itself. This still raised problems with the peculiarities of the platform. We quickly found that the dinamic linker wasn't honoring the LD_LIBRARY_PATH environment variable, which we were trying to use to tell the system where to find the dynamic libraries.

The Android platform installs new software using a package manager. The package manager creates a directory in the SD card that it's only writable by the applilcation being installed. Within this directory the installer puts the libraries declared in the package. Bigloo, besides the dinamic libraries, requieres some aditional files that initialize the global objects. This files are not extracted by the installer, so we had to make a frontend in Java that opens the package and extract them by hand. But the installer creates the directory for the libraries in such a way that the application later cannot write in it.

Also, we found that the dinamic linker works for libraries linked at runtime, but does not for dlopen()'ing them, so we also had to rewrite a great part of our build system for both Bigloo and Hop to produce static libraries and binaries. This also needed disabling the dynamic loading of libraries, and with them, their initialization, so we had to initialize them by hand.

To add more unsuspected work, the Android package builder, provided with the SDK, ignores hidden files, which Bigloo uses to map Scheme module names to dynamic libraries. We had to work around this feature in the unpacking algorithm.

Then we moved to improve the friendliness of the frontend. So far, we could install Hop in the platform, either in a phone or in the emulator, but we could only run it in the emulator, because we were using a shell that runs as root on the emulator, but that runs as a user in a real device. This user, for the reasons given above, cannot even get into Hop's install dir. Even when Android has a component interface that allows applications to use components from other apps, none of the terminal apps we found at that time declared the terminal itself as a reusable component. We decided to use the code from the most popular one, which was based on a demo available on Android's source code, but not installed in actual devices. We had to copy the source code and trimm it down to our needs.

Having a more or less usable Hop package for Android, we decided to try and fix the issue we mentioned before: GC didn't compile with threads enabled. This means that we can't use the pthreads library, which is very useful for Hop. Hop uses threads to attend several requests at the same time. Bigloo implements two threads APIs, one based on pthreads and another which implements fair threads. Hop is able to use 5 different request schedulers, but works better with the one based on pthreads.

For these reasons we decided to focus in getting GC to use threads with the Android platform. GC's build system tests the existence of a threading platform checking the thread model declared by gcc. The gcc version provided with Android's SDK declares to have a 'single thread model', but we couldn't find what does this mean in terms of the code produced by gcc or how this could affect to GC's execution.

(Note: we didn't manage to make GC compile with threads.)

With a threadless Hop running, we had to add code to the server so we could talk between the server and the frontend while at the same time it is attending the requests from a web client. After several attempts to attack this problem, we decided that the best solution was to make this interface another service served by Hop. This meant less modifications to Hop itself, but a bigger one to the frontend we already had.

During these changes we found out a problem with JNI. The terminal component we imported into our code uses a small C library for executing the application inside (normaly a shell, in the original code, but Hop in our case) which is accessed from Java using JNI. The original Term application exported this class as com.android.term.Exec, but our copy exported it as fr.inria.hop.Exec. Even with this namespace difference JNI got confused and tried to use the Exec class from the original Term app. This is just another example how the platform is hard to work with. We found that the community support is more centered around Java and that very few people know about JNI, the NDK or any other native related technologies. We couldn't find an answer to this problem, so we worked around this by renaming the class.

So that's it. I can provide all the technical details for most the assertions I postulated above, but that would make this post unreadbal for its length. If you have any question about them, just conact me.

android

Posted Sat 26 Mar 2011 12:12:39 AM CET Tags:
Posted Wed 06 Apr 2011 08:28:18 PM CEST
02
Posted Wed 06 Apr 2011 08:28:18 PM CEST
12

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:

Yesterday[1] I wrote: «Then I found out that it wouldn't be easy to use the tiles [generated by CloudMade] with marble». How wrong was I.

Here you can read how to access the tiles in CM's servers. The only thing you need is an API key, which you can get once you registered (it practically begs you to get one :). Another piece of the puzzle is marble itself. I was specting to copy OSM's service description file and bend it and twist it until it worked with CM's tiles. But things are, luckily, much more simple.

In marble you simply do «File», «Create a new map...», then select «Online map providing indexed tiles», which clearly describes CM. Next, you have to provide and URL template, which you can simply derive from the doc in CM's site. Mine's like this:

http://tile.cloudmade.com/5f806ad32bb44b38a464020fa2223193/51084/256/{zoomLevel}/{x}/{y}.png

You, of course, will have to change the API key and the style number for yours. One thing I found is that the @2x doesn't seem to work with marble; without it it works just fine. Maybe it's a User-Agent thing. Then you keep answering what marble asks you in the wizard. By the end of it, you'll have marble working with your tiles! It's amazing.

One extra hack was to add support for hillshading. These are tiles that are alpha-blended with OSM's, wich gives you the impression of viewing a 3D map (bah, at least in the covered regions with mountains). The service files (.dgml files) are actually XML files quite easy to edit if you defocus and just look at the big picture ot if. So I just simply opened both files (OSM's is in /usr/share/kde4/apps/marble/data/maps/earth/openstreetmap/openstreetmap.dgml, the one you created in $HOME/.local/share/marble/maps/earth/foo/foo.dgml), copied the texture tag that describes hillshading, pasted it in my .dgml file, saved, reopened marble and... voilà! Even better, as hillshading is actually another tile server, marble can share it for OSM's, OSMaRenderer and yours.

The final result is simply amazing. You can see a screenshot here.

More on my use of marble and OSM pretty soon.


[1] When I say "yesterday", I actually mean "almost 7 months ago". That's what took me to fix my glob and come back to writing, finishing this post.


openstreetmap marble

Posted Wed 25 Jul 2012 03:35:42 PM CEST Tags:

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:
Posted Thu 29 Dec 2011 10:35:03 PM CET

Ever since it came out in November 2009 I wanted a Nokia N900. Many reasons made me buy one, some reasons made me wait until this month. Maemo, it's OS, is almost exactly what I wanted, a real GNU/busybox/Linux. I can do a lot of stuff with it because it's, I think, the closest to a mini computer with an integrated GPS, camera, cellular network modem, WiFi and can act both as an USB master or a slave. I don't know about the N950 or the N9, but really this is a tinkerer's dream.

So far I haven't done much with it, most of my personal time is being invested in something else, but I've been using it mainly as an in-car computer and a WiFi-to-LAN router. The latter will be the subject to another post, but I want to concentrate on the former in this post.

As an in-car computer right now I want two things: navigation and audio player. The audio player is done; so far the media player that comes in the... phone? computer? I'll say computer. So far the media player that comes with the computer is enough for me. Maybe sometime I'll do a satyr skin for Maemo and port the beast to it, just for the heck of it.

Also it has the OVI maps, the default app for navigation, which from what I have heard, is very good, but it needs a data connection to work. I don't want to buy a data plan yet, and also there are places where there is no good data coverage. You might not even get enough signal to make a call or even send an SMS. So I want something that can cache maps locally in the computer.

The answer is actually somewhat complex: marble, monav and OpenStreetMap. Marble is the KDE map viewing application. It can show several types of maps, even from several celestial bodies. It can talk to a GPS to record and save tracks, and to monav, which is a routing application that uses OSM data. Of course, Marble can use OSM's tiles. And the good thing about both is that they can use data cached on the computer, which means no data link needed, at least not all the time.

Installing them in Maemo is as simple as selecting them in the App manager (although I suggest to install and use Faster Application Manager, which is no so far from a mix between a smartphone store and, let's say, aptitude). Both are in the extras repo.

Once they're installed, the next logical step is to donwload monav's data. For that you open marble; from its menu you select «Routing», then «Configure». There you can select Monav, and «Configure». There you can download routing maps per country. Download all the ones you want, but remember it takes space and some time to uncompress.

As for the maps themselves, you can select the tileset you want (so far the common Mapnik and OSMaRenderer are available). Marble will start downloading and caching tiles from OSM's servers. You can even use «Download Region...» to download the tiles for a certain region or route in an interval of zooms. Becareful about this, because OSM's servers will ban you for a little while if you pull too much info.

So now I have an almost complete offline map and routing solution. I still have to investigate how to add voices (so far it's mute). I also found a couple of problems with this setup. The most "simple" to solve are UI ones: the navigation instructions have too much text and too small icons, and while having the navigation widget on, you lose the zoom controls. Luckily my dock leaves the volume/zoom hardware buttons free, so I can zoom with them (who needs multitouch :).

But the most prominent one is the map itself. Both Mapnik and OSMaRenderer are too confusing for in-car navigation, they have too much detail, and minor streets are not clearly distinguished from the blocks themselves. Mapnik in particular has a very low contrast, which also makes it hard to read in plain daylight.

Enter CloudMade. I still don't get much what CloudMade is exactly for, but one thing I know: it's very easy to create tilesets from OSM data. First I browsed the styles already in the site and I found one called «Almost Ordinance Survey 25k». I took it and modified it heavily. I named it «daylight in-car navigation».

Then I found out that it wouldn't be easy to use the tiles with marble. At least I got to find the right settings. Next chapter will be about using OSMaRenderer locally.

openstreetmap marble maemo

Posted Fri 30 Dec 2011 09:58:34 PM CET Tags:
Posted Thu 29 Dec 2011 10:34:58 PM CET
Posted Wed 06 Apr 2011 08:28:18 PM CEST