Cassandra counters are not atomic

Today I arrived at work and I was shoved to a scrambled war room. Inside, we the two sysadmins working with C*, our inmediate boss, the DBA, the PHP developer involved in this first migration proyect (from MySQL, this is important) and the project leader replacing the one on vacations. Yesterday before I left I saw a similar but more informal gathering around the other sysadmin who's testing the migration in our preproduction environment. As I was busy in the other corner of the office with my backup tasks (I'm still strugling with a lots of constraints, but I think I finally tackled it down, as in down to the floor. I hope it will just stay still for production... but I digress), so I was unaware the reason of the meeting, except for the title in the mail: "Problem with the counter".

If you followed this history closely, you have all the clues to know where I'm going1. We're replacing the smallest of our databases, the avatar database, which might be small (~50GiB of data), but is has a great impact, because not only a lot of our pages use it, but also our customers and/or partners. Each image has an unique ID implemented with a MySQL auto increment key. Because of the impact, we couldn't simply go for UUIDs.

Now, we knew that even when C* implements counter columns since v0.8, there is no guarantee that the counter changes are atomic from a cluster point of view. What really surprised us is that this no-guarantee also holds for only one node. In other words, simultaneous changes to a counter (incremens by 1, in our case) are not atomic even when they're done in the same node.

To make sure, I put on my hazmat suit3 and plunged head first, shit shovel in hand (at the end they were not needed, the code is very readable), into C*'s code. After some maneuvering (it's not a straight route, there was some back-and-forth) I got the piece of code that I was looking for. Basically it says that if there are no indexes to update and no deletes, the update is done concurrently without any locks. And you cannot index counters, of course, as it makes no sense.

Clearly the subject of atomic counters is not something that C* plans to fix any time soon2, and given the difficulty of it, I can understand that desition. But I would expect atomicity at node level, so if one desires so, one can shoot his own foot "implementing" atomic counters just writing the updates in only one node (and then some medium/heavy infra to implement HA).

One more guy, 90 minutes, 5 possible solutions (including snowflake) later, we desided to temporarily still use MySQL for the counter (remember, we're going online next week) and look closer to more permanent solutions later (this other guy, a sysadmin in another group, has been on-and-off fighting to even make snowflake start in his own machine), including of course, the long term goal of getting rid of this kind of IDs.


  1. Well, if the title didn't gave it away from the beggining. 

  2. If you feel curious, start here

  3. 3 years in France has made me somewhat careful... Dammit!