Today I arrived at work and I was shoved to a scrambled war room. Inside, we the two
sysadmins working with C*, our inmediate boss, the DBA, the PHP developer involved
in this first migration proyect (from MySQL, this is important) and the project
leader replacing the one on vacations. Yesterday before I left I saw a similar
but more informal gathering around the other sysadmin who's testing the migration
in our preproduction environment. As I was busy in the other corner of the office
with my backup tasks (I'm still strugling with a lots of constraints, but I
think I finally tackled it down, as in down to the floor. I hope it will just stay
still for production... but I digress), so I was unaware the reason of the meeting,
except for the title in the mail: "Problem with the counter".

If you followed this history closely, you have all the clues to know where I'm
going[^1]. We're replacing the smallest of our databases, the avatar database,
which might be small (~50GiB of data), but is has a great impact, because not only
a lot of our pages use it, but also our customers and/or partners. Each image has
an unique ID implemented with a MySQL auto increment key. Because of the impact,
we couldn't simply go for UUIDs.

Now, we knew that even when C* implements counter columns since v0.8, there is no
guarantee that the counter changes are atomic from a cluster point of view.
What really surprised us is that this no-guarantee also holds for only one node.
In other words, simultaneous changes to a counter (incremens by 1, in our case)
are not atomic even when they're done in the same node.

To make sure, I put on my hazmat suit[^3] and plunged
head first, shit shovel in hand (at the end they were not needed, the code is
very readable), into C*'s code. After some maneuvering (it's not
a straight route, there was some back-and-forth) I got
[the piece of code](https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Table.java#L384)
that I was looking for. Basically it says that if there are no indexes to update
and no deletes, the update is done concurrently without any locks. And you cannot
index counters, of course, as it makes no sense.

Clearly the subject of atomic counters is not something that C* plans to fix any
time soon[^2], and given the difficulty of it, I can understand that desition.
But I would expect atomicity at node level, so if one desires so, one can shoot
his own foot "implementing" atomic counters just writing the updates 
in only one node (and then some medium/heavy infra to implement HA).

One more guy, 90 minutes, 5 possible solutions (including
[snowflake](https://github.com/twitter/snowflake)) later, we desided to temporarily
still use MySQL for the counter (remember, we're going online next week) and look
closer to more permanent solutions later (this other guy, a sysadmin in another
group, has been on-and-off fighting to even make snowflake start in his own
machine), including of course, the long term goal of getting
rid of this kind of IDs.

[^1]: Well, if the title didn't gave it away from the beggining.

[^2]: If you feel curious, start [here](https://issues.apache.org/jira/browse/CASSANDRA-721).

[^3]: 3 years in France has made me somewhat careful... Dammit!