"My maps do not render". It's always a simple question like this. The error message was quite cryptic:

Failed to parse color: "D27E01"

This makes sense, that is not a color; #D27E01 is. I thought "I might have made a typo". Searching through my map style gave nothing. Hmm,m that's weird. Maybe the database? I pick a few colors from the data, f.i. for bus or subte/metro/subway/underground lines. Nope, not that either. So where is it coming from?

I fire gdb and thanks to automatic debug symbol loading I get more info that I would otherwise get. First rock I stumble upon: the function where that error is raised is called for every parsed color. In a map style there are gazillion, and remember I'm also parsing some coming from the db. So to set a conditional break point. It's easy, right? Just break parse_color if [condition] and done!

Not so fast. To begin with, strings don't exist in C¹, it's the arrays and \0. Also, arrays don't exist in C, it's pointers and wishful thinking. condition will have to involve strcmp() and == 0. But the parameter is actually a std::string const& str, so it's in the C++ realm. After asking around, guinevere#gdb@libera.chat suggested break parse_color if $_streq(str.data(), "D27E01"), which worked.

The next step was to make sense of the code. mapnik, the library I'm debugging, is the most inscrutable code I have ever seen. Here's a full backtrace of the moment I hit the bug:

#0  mapnik::parse_color (str="D27E01") at ./src/color_factory.cpp:32
#1  0x00007ffff5921295 in mapnik::color::color (this=this@entry=0x7fffffffc0d0, str="D27E01", premultiplied=premultiplied@entry=false) at ./src/color.cpp:38
#2  0x00007ffff614051d in mapnik::evaluate_expression_wrapper<mapnik::color>::operator()<mapbox::util::variant<mapnik::value_null, bool, long, double, icu_76::UnicodeString, mapnik::attribute, mapnik::global_attribute, mapnik::geometry_type_attribute, mapbox::util::recursive_wrapper<mapnik::unary_node<mapnik::tags::negate> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::plus> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::minus> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::mult> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::div> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::mod> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::less> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::less_equal> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::greater> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::greater_equal> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::equal_to> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::not_equal_to> >, mapbox::util::recursive_wrapper<mapnik::unary_node<mapnik::tags::logical_not> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::logical_and> >, mapbox::util::recursive_wrapper<mapnik::binary_node<mapnik::tags::logical_or> >, mapbox::util::recursive_wrapper<mapnik::regex_match_node>, mapbox::util::recursive_wrapper<mapnik::regex_replace_node>, mapbox::util::recursive_wrapper<mapnik::unary_function_call>, mapbox::util::recursive_wrapper<mapnik::binary_function_call> >, mapnik::feature_impl, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mapnik::value_adl_barrier::value, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, mapnik::value_adl_barrier::value> > > > (this=<optimized out>, expr=..., feature=..., vars=...) at ./include/mapnik/symbolizer.hpp:284
#3  mapnik::extract_value<mapnik::color>::operator() (this=<optimized out>, expr=...) at ./include/mapnik/symbolizer.hpp:342
#4  apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:332
#5  0x00007ffff61405e8 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#6  0x00007ffff6140640 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#7  0x00007ffff61406a0 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#8  0x00007ffff6140700 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#9  0x00007ffff6140760 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#10 0x00007ffff61407c0 in apply<mapnik::detail::strict_value const&, mapnik::extract_value<mapnik::color> > (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#11 0x00007ffff61407eb in visit<mapnik::extract_value<mapnik::color>, mapnik::detail::strict_value const&> (v=..., f=...) at /usr/include/mapbox/variant.hpp:838
#12 0x00007ffff6140837 in mapnik::util::apply_visitor<mapnik::extract_value<mapnik::color>, mapnik::detail::strict_value const&> (v=..., f=...) at ./include/mapnik/util/variant.hpp:42
#13 0x00007ffff614090b in mapnik::get<mapnik::color, (mapnik::keys)9> (sym=..., feature=..., vars=std::unordered_map with 0 elements) at ./include/mapnik/symbolizer.hpp:335
#14 mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4>::process (this=0x7fffffffc890, sym=..., feature=..., prj_trans=...) at ./src/agg/process_line_symbolizer.cpp:95
#15 0x00007ffff597d682 in mapnik::process_impl<true>::process<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4>, mapnik::line_symbolizer, mapnik::feature_impl, mapnik::proj_transform> (ren=..., 
    sym=..., f=..., tr=...) at ./include/mapnik/symbolizer_dispatch.hpp:43
#16 0x00007ffff597d6e1 in mapbox::util::detail::dispatcher<void, mapnik::point_symbolizer, mapnik::line_symbolizer, mapnik::line_pattern_symbolizer, mapnik::polygon_symbolizer, mapnik::polygon_pattern_symbolizer, mapnik::raster_symbolizer, mapnik::shield_symbolizer, mapnik::text_symbolizer, mapnik::building_symbolizer, mapnik::markers_symbolizer, mapnik::group_symbolizer, mapnik::debug_symbolizer, mapnik::dot_symbolizer>::apply<mapbox::util::variant<mapnik::point_symbolizer, mapnik::line_symbolizer, mapnik::line_pattern_symbolizer, mapnik::polygon_symbolizer, mapnik::polygon_pattern_symbolizer, mapnik::raster_symbolizer, mapnik::shield_symbolizer, mapnik::text_symbolizer, mapnik::building_symbolizer, mapnik::markers_symbolizer, mapnik::group_symbolizer, mapnik::debug_symbolizer, mapnik::dot_symbolizer> const&, mapnik::symbolizer_dispatch<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> > > (
    v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#17 0x00007ffff597d6f9 in mapbox::util::variant<mapnik::point_symbolizer, mapnik::line_symbolizer, mapnik::line_pattern_symbolizer, mapnik::polygon_symbolizer, mapnik::polygon_pattern_symbolizer, mapnik::raster_symbolizer, mapnik::shield_symbolizer, mapnik::text_symbolizer, mapnik::building_symbolizer, mapnik::markers_symbolizer, mapnik::group_symbolizer, mapnik::debug_symbolizer, mapnik::dot_symbolizer>::visit<mapnik::symbolizer_dispatch<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >, mapbox::util::variant<mapnik::point_symbolizer, mapnik::line_symbolizer, mapnik::line_pattern_symbolizer, mapnik::polygon_symbolizer, mapnik::polygon_pattern_symbolizer, mapnik::raster_symbolizer, mapnik::shield_symbolizer, mapnik::text_symbolizer, mapnik::building_symbolizer, mapnik::markers_symbolizer, mapnik::group_symbolizer, mapnik::debug_symbolizer, mapnik::dot_symbolizer> const&, mapnik::point_symbolizer const&, void> (v=..., f=...) at /usr/include/mapbox/variant.hpp:838
#18 0x00007ffff597d712 in mapnik::util::apply_visitor<mapnik::symbolizer_dispatch<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >, mapbox::util::variant<mapnik::point_symbolizer, mapnik::line_symbolizer, mapnik::line_pattern_symbolizer, mapnik::polygon_symbolizer, mapnik::polygon_pattern_symbolizer, mapnik::raster_symbolizer, mapnik::shield_symbolizer, mapnik::text_symbolizer, mapnik::building_symbolizer, mapnik::markers_symbolizer, mapnik::group_symbolizer, mapnik::debug_symbolizer, mapnik::dot_symbolizer> const&> (f=..., v=...) at ./include/mapnik/util/variant.hpp:42
#19 0x00007ffff598611a in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::render_style (this=<optimized out>, p=..., style=0x1cb2e60, rc=..., 
    features=std::shared_ptr<mapnik::Featureset> (use count 3, weak count 0) = {...}, prj_trans=...) at ./include/mapnik/feature_style_processor_impl.hpp:592
#20 0x00007ffff59869af in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::render_material (this=this@entry=0x7fffffffc890, mat=..., p=...)
    at ./include/mapnik/feature_style_processor_impl.hpp:552
#21 0x00007ffff5987994 in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::render_submaterials (this=this@entry=0x7fffffffc890, parent_mat=..., p=...)
    at ./include/mapnik/feature_style_processor_impl.hpp:453
#22 0x00007ffff598c1e0 in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::apply (this=this@entry=0x7fffffffc890, scale_denom=<optimized out>, 
    scale_denom@entry=0) at ./include/mapnik/feature_style_processor_impl.hpp:148
#23 0x00007ffff6d9d858 in agg_renderer_visitor_1::operator()<mapnik::image<mapnik::rgba8_t> > (this=<optimized out>, pixmap=...) at src/mapnik_python.cpp:220
#24 0x00007ffff6dabeea in mapbox::util::detail::dispatcher<void, mapnik::image<mapnik::rgba8_t>, mapnik::image<mapnik::gray8_t>, mapnik::image<mapnik::gray8s_t>, mapnik::image<mapnik::gray16_t>, mapnik::image<mapnik::gray16s_t>, mapnik::image<mapnik::gray32_t>, mapnik::image<mapnik::gray32s_t>, mapnik::image<mapnik::gray32f_t>, mapnik::image<mapnik::gray64_t>, mapnik::image<mapnik::gray64s_t>, mapnik::image<mapnik::gray64f_t> >::apply<mapnik::image_any&, agg_renderer_visitor_1> (v=..., f=...) at /usr/include/mapbox/variant.hpp:332
#25 0x00007ffff6dabf28 in mapbox::util::detail::dispatcher<void, mapnik::image<mapnik::null_t>, mapnik::image<mapnik::rgba8_t>, mapnik::image<mapnik::gray8_t>, mapnik::image<mapnik::gray8s_t>, mapnik::image<mapnik::gray16_t>, mapnik::image<mapnik::gray16s_t>, mapnik::image<mapnik::gray32_t>, mapnik::image<mapnik::gray32s_t>, mapnik::image<mapnik::gray32f_t>, mapnik::image<mapnik::gray64_t>, mapnik::image<mapnik::gray64s_t>, mapnik::image<mapnik::gray64f_t> >::apply<mapnik::image_any&, agg_renderer_visitor_1> (v=..., f=...) at /usr/include/mapbox/variant.hpp:336
#26 0x00007ffff6dabf89 in mapbox::util::variant<mapnik::image<mapnik::null_t>, mapnik::image<mapnik::rgba8_t>, mapnik::image<mapnik::gray8_t>, mapnik::image<mapnik::gray8s_t>, mapnik::image<mapnik::gray16_t>, mapnik::image<mapnik::gray16s_t>, mapnik::image<mapnik::gray32_t>, mapnik::image<mapnik::gray32s_t>, mapnik::image<mapnik::gray32f_t>, mapnik::image<mapnik::gray64_t>, mapnik::image<mapnik::gray64s_t>, mapnik::image<mapnik::gray64f_t> >::visit<agg_renderer_visitor_1, mapnik::image_any&, mapnik::image<mapnik::null_t>&, void> (v=..., f=...) at /usr/include/mapbox/variant.hpp:838
#27 0x00007ffff6dabfa2 in mapnik::util::apply_visitor<agg_renderer_visitor_1, mapnik::image_any&> (f=..., v=...) at /usr/include/mapnik/util/variant.hpp:42
#28 0x00007ffff6da4e41 in render (map=..., image=..., scale_factor=1, offset_x=0, offset_y=0) at src/mapnik_python.cpp:316
#29 0x00007ffff6dac584 in boost::python::detail::invoke<int, void (*)(mapnik::Map const&, mapnik::image_any&), boost::python::arg_from_python<mapnik::Map const&>, boost::python::arg_from_python<mapnik::image_any&> > (f=<optimized out>, 
    ac0=..., ac1=<synthetic pointer>...) at /usr/include/boost/python/detail/invoke.hpp:79
#30 boost::python::detail::caller_arity<2u>::impl<void (*)(mapnik::Map const&, mapnik::image_any&), boost::python::default_call_policies, boost::mpl::vector3<void, mapnik::Map const&, mapnik::image_any&> >::operator() (
    this=<optimized out>, args_=<optimized out>) at /usr/include/boost/python/detail/caller.hpp:233
#31 0x00007ffff6fdf3ae in boost::python::objects::py_function::operator() (this=0xec3670, args=0x7fffe7caf400, kw=<optimized out>) at ./boost/python/object/py_function.hpp:147
#32 boost::python::objects::function::call (this=0xec3ac0, args=0x7fffe7caf400, keywords=0x0) at libs/python/src/object/function.cpp:221
#33 0x00007ffff6fdf62c in boost::python::objects::(anonymous namespace)::bind_return::operator() (this=<optimized out>) at libs/python/src/object/function.cpp:581
#34 boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke (function_obj_ptr=...) at ./boost/function/function_template.hpp:193
#35 0x00007ffff6fe465b in boost::function0<void>::operator() (this=<optimized out>) at ./boost/function/function_template.hpp:771
#36 boost::python::detail::exception_handler::operator() (this=<optimized out>, f=...) at libs/python/src/errors.cpp:74
#37 0x00007ffff6da9b67 in boost::python::detail::translate_exception<std::runtime_error, void (*)(std::runtime_error const&)>::operator() (this=<optimized out>, handler=..., f=..., 
    translate=0x7ffff6d9d0f0 <runtime_error_translator(std::runtime_error const&)>) at /usr/include/boost/python/detail/translate_exception.hpp:39
#38 boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::runtime_error const&)> >::operator()<bool, boost::python::detail::translate_exception<std::runtime_error, void (*)(std::runtime_error const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=<optimized out>, f=..., a=<synthetic pointer>...) at /usr/include/boost/bind/bind.hpp:368
#39 boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::runtime_error, void (*)(std::runtime_error const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::runtime_error const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=<optimized out>, a1=..., a2=...) at /usr/include/boost/bind/bind.hpp:1298
#40 boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::runtime_error, void (*)(std::runtime_error const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::runtime_error const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (function_obj_ptr=..., a0=..., a1=...)
    at /usr/include/boost/function/function_template.hpp:137
#41 0x00007ffff6da9bc7 in boost::python::detail::translate_exception<mapnik::value_error, void (*)(mapnik::value_error const&)>::operator() (this=<optimized out>, handler=..., f=..., 
    translate=0x7ffff6d9d250 <value_error_translator(mapnik::value_error const&)>) at /usr/include/boost/python/detail/translate_exception.hpp:39
#42 boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(mapnik::value_error const&)> >::operator()<bool, boost::python::detail::translate_exception<mapnik::value_error, void (*)(mapnik::value_error const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=<optimized out>, f=..., a=<synthetic pointer>...) at /usr/include/boost/bind/bind.hpp:368
#43 boost::_bi::bind_t<bool, boost::python::detail::translate_exception<mapnik::value_error, void (*)(mapnik::value_error const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(mapnik::value_error const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=<optimized out>, a1=..., a2=...) at /usr/include/boost/bind/bind.hpp:1298
#44 boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<mapnik::value_error, void (*)(mapnik::value_error const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(mapnik::value_error const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (function_obj_ptr=..., a0=..., a1=...)
    at /usr/include/boost/function/function_template.hpp:137
#45 0x00007ffff6da9c27 in boost::python::detail::translate_exception<std::out_of_range, void (*)(std::out_of_range const&)>::operator() (this=<optimized out>, handler=..., f=..., 
    translate=0x7ffff6d9d120 <out_of_range_error_translator(std::out_of_range const&)>) at /usr/include/boost/python/detail/translate_exception.hpp:39
#46 boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::out_of_range const&)> >::operator()<bool, boost::python::detail::translate_exception<std::out_of_range, void (*)(std::out_of_range const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=<optimized out>, f=..., a=<synthetic pointer>...) at /usr/include/boost/bind/bind.hpp:368
#47 boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::out_of_range, void (*)(std::out_of_range const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::out_of_range const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=<optimized out>, a1=..., a2=...) at /usr/include/boost/bind/bind.hpp:1298
#48 boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::out_of_range, void (*)(std::out_of_range const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::out_of_range const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (function_obj_ptr=..., a0=..., a1=...) at /usr/include/boost/function/function_template.hpp:137
#49 0x00007ffff6da9c87 in boost::python::detail::translate_exception<std::exception, void (*)(std::exception const&)>::operator() (this=<optimized out>, handler=..., f=..., 
    translate=0x7ffff6d9d150 <standard_error_translator(std::exception const&)>) at /usr/include/boost/python/detail/translate_exception.hpp:39
#50 boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::exception const&)> >::operator()<bool, boost::python::detail::translate_exception<std::exception, void (*)(std::exception const&)>, boost::_bi::rrlist2<boost::python::detail::exception_handler const&, boost::function0<void> const&> > (this=<optimized out>, f=..., a=<synthetic pointer>...) at /usr/include/boost/bind/bind.hpp:368
#51 boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::exception, void (*)(std::exception const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::exception const&)> > >::operator()<boost::python::detail::exception_handler const&, boost::function0<void> const&> (this=<optimized out>, a1=..., a2=...) at /usr/include/boost/bind/bind.hpp:1298
#52 boost::detail::function::function_obj_invoker2<boost::_bi::bind_t<bool, boost::python::detail::translate_exception<std::exception, void (*)(std::exception const&)>, boost::_bi::list3<boost::arg<1>, boost::arg<2>, boost::_bi::value<void (*)(std::exception const&)> > >, bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::invoke (function_obj_ptr=..., a0=..., a1=...) at /usr/include/boost/function/function_template.hpp:137
#53 0x00007ffff6fe4511 in boost::function2<bool, boost::python::detail::exception_handler const&, boost::function0<void> const&>::operator() (this=<optimized out>, a0=..., a1=...) at ./boost/function/function_template.hpp:771
#54 boost::python::detail::exception_handler::handle (this=<optimized out>, f=...) at ./boost/python/detail/exception_handler.hpp:41
#55 boost::python::handle_exception_impl (f=...) at libs/python/src/errors.cpp:24
#56 0x00007ffff6fdc3c6 in boost::python::handle_exception<boost::python::objects::(anonymous namespace)::bind_return> (f=...) at ./boost/python/errors.hpp:29
#57 boost::python::objects::function_call (func=<optimized out>, args=<optimized out>, kw=<optimized out>) at libs/python/src/object/function.cpp:622
#58 0x0000000000543b8b in _PyObject_MakeTpCall (tstate=0xa7d510 <_PyRuntime+283024>, callable=0xec3ac0, args=<optimized out>, nargs=2, keywords=<optimized out>) at ../Objects/call.c:242
#59 0x000000000055f191 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at ../Python/generated_cases.c.h:813
#60 0x000000000064db6c in _PyEval_EvalFrame (tstate=0xa7d510 <_PyRuntime+283024>, frame=0x7ffff7fb2020, throwflag=0) at ../Include/internal/pycore_ceval.h:119
#61 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, tstate=0xa7d510 <_PyRuntime+283024>, func=0x7ffff7497740, locals=<optimized out>) at ../Python/ceval.c:1814
#62 PyEval_EvalCode (co=0xbcf8d0, globals=<optimized out>, locals=<optimized out>) at ../Python/ceval.c:604
#63 0x000000000066da21 in run_eval_code_obj (tstate=0xa7d510 <_PyRuntime+283024>, co=0xbcf8d0, globals=0x7ffff7434a00, locals=0x7ffff7434a00) at ../Python/pythonrun.c:1381
#64 0x000000000066988c in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff7434a00, locals=0x7ffff7434a00, flags=<optimized out>, arena=<optimized out>, interactive_src=0x0, generate_new_source=0)
    at ../Python/pythonrun.c:1466
#65 0x0000000000682983 in pyrun_file (fp=fp@entry=0xb1dcd0, filename=filename@entry=0x7ffff744aa30, start=start@entry=257, globals=globals@entry=0x7ffff7434a00, locals=locals@entry=0x7ffff7434a00, closeit=closeit@entry=1, 
    flags=0x7fffffffd3a8) at ../Python/pythonrun.c:1295
#66 0x0000000000682283 in _PyRun_SimpleFileObject (fp=fp@entry=0xb1dcd0, filename=filename@entry=0x7ffff744aa30, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd3a8) at ../Python/pythonrun.c:517
#67 0x00000000006820be in _PyRun_AnyFileObject (fp=0xb1dcd0, filename=0x7ffff744aa30, closeit=1, flags=0x7fffffffd3a8) at ../Python/pythonrun.c:77
#68 0x0000000000680ef1 in pymain_run_file_obj (program_name=0x7ffff7434b30, filename=0x7ffff744aa30, skip_source_first_line=0) at ../Modules/main.c:410
#69 pymain_run_file (config=0xa4fc08 <_PyRuntime+96392>) at ../Modules/main.c:429
#70 pymain_run_python (exitcode=0x7fffffffd39c) at ../Modules/main.c:697
#71 Py_RunMain () at ../Modules/main.c:776
#72 0x000000000063d6eb in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at ../Modules/main.c:830
#73 0x00007ffff7c90ca8 in __libc_start_call_main (main=main@entry=0x63d640 <main>, argc=argc@entry=3, argv=argv@entry=0x7fffffffd5d8) at ../sysdeps/nptl/libc_start_call_main.h:58
#74 0x00007ffff7c90d65 in __libc_start_main_impl (main=0x63d640 <main>, argc=3, argv=0x7fffffffd5d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd5c8) at ../csu/libc-start.c:360
#75 0x000000000063cab1 in _start ()

What I get from it is that the color is not coming from the style of the db but it's the result of an expression; see frame #3. Thing is, those parameters have been optimized out into CPU registers and there is no easy way to inspect them as is:

#3  mapnik::extract_value<mapnik::color>::operator() (this=<optimized out>, expr=...) at ./include/mapnik/symbolizer.hpp:342
342     in ./include/mapnik/symbolizer.hpp
(gdb) print this
$1 = <optimized out>
(gdb) print expr
$2 = <optimized out>

So, what options do I have? Well, one is to compile mapnik (and python3-mapnik, because I'm driving it from a Python program; you can see it in the stacktrace since frame #29²) and I tried this first, but even compiling the released code on my Debian failed in ways I could not fix myself without potentially introducing more bugs.

But luckily and ofcoursely³ the sources from Debian did compile, compiled the debs using CXXFLAGS='-g -O0' CFLAGS='-g -O0' dpkg-buildpackage -us -uc⁴, installed them by hand and now I can do my debug session normally.

Unluckily for this post the error was in the data. When I searched for it I used the wrong key for the Postgres HStore (color vs colour; damn Webster!). It's also a lucky hit, because I was reaching the point were I would had to untangle a mess of C++ templating and I was running out of time. mapnik is definitely the hardest code I had ever had to follow.

Wow, fist written (not dictated) post in a long time. Let's see if I can finish it.

Given the fact that Amazon is shutting down downloading your purchased ebooks, I decided to download them all before the cut in a couple of days. To me, the definitve gide comes in this toot:

rem@front-end.social, Feb 18

The first link is a tool that will automatically download the books for you; otherwise, you have to download them by hand/clickops. If your collection is big that can be tedious. The first problem is that that tool is developed in a obscure programming language called bun¹, which of course gets installed by the classic curl | bash method which... sigh. Let's just say it's not the ideal. At least the options they use in curl are not bad, but it would be nice if they used the --long-options so I don't have to peruse curl's man page to see if they're not doing anything even more crappy, like ignoring bad SSL certs or something worse.

So instead of blindly doing that, I open the script and start reading. The usual crap, but it includes this gem:

install_env=BUN_INSTALL
bin_env=\$$install_env/bin

install_dir=${!install_env:-$HOME/.bun}
bin_dir=$install_dir/bin
exe=$bin_dir/bun

In a roundabout way, it's saying that it will honor the BUN_INSTALL envvar as the root of the installation, defaulting to $HOME/.bun, which is what I was looking for; I don't want more trash in my home directory. I also run the script with tracing to make sure it doesn't do anything ugly.

mdione@ioniq:~/src/system/fsck-amazon$ BUN_INSTALL=$(pwd) bash -x ./install

I should have read it more. The bloody thing helpfully adds these lines you the .bashrc:

# bun
export BUN_INSTALL="$HOME/src/system/fsck-amazon"
export PATH="$BUN_INSTALL/bin:$PATH"

sigh So I undo that and only set the envvars on the shell I run for all this.

Ok, now to follow the bulkk tool's install instructions. The step that most time takes is:

🚚 typescript...

sigh. To be slightly fair, my internet has been crappy for a while now. I blame the PLC network between my ISP router and my home made AP.

So far only (!!!) 121MiB have been used. sigh Let's see how much will it be at the end, because the next step is:

bunx puppeteer browsers install chrome

See that last one? This is because the ony things that can use the web lately are only full fledged browsers that include a fsck-ton amount of technologies, so this thing is going to drive Amazon's site with a full fledged Chrome browser. And of course it escapes my feeble attempt to give it a jail:

mdione@ioniq:~/src/system/fsck-amazon/amazon-kindle-bulk-downloader$ bunx puppeteer browsers install chrome
chrome@133.0.6943.98 /home/mdione/.cache/puppeteer/chrome/linux-133.0.6943.98/chrome-linux64/chrome

sigh

mdione@ioniq:~/src/system/fuck-amazon/amazon-kindle-bulk-downloader$ du -sm /home/mdione/.cache/puppeteer/
585     /home/mdione/.cache/puppeteer/

So 706MiB so far.

eyeroll

To not write the auth on the disk in plain text, I just run the system with manual auth:

mdione@ioniq:~/src/system/fsck-amazon/amazon-kindle-bulk-downloader$ bun run start --manualAuth
$ bun run src/index.ts --manualAuth
✔ Enter the Amazon base URL … https://www.amazon.com/
✔ Press enter once you've logged in … yes
Got auth
[...]
Downloading complete. You can find your books in the 'downloads' folder.

An that's it. After this I cleaned up the two diredtories, making sure not to delete the downloads the tool created.

Next step is to download the Calibre plugin (I already have Calibre via the OS packages) and follow the install instructions. One of the step is to write a list of serials for the ebooks you have. I my case, it was on drop-from-the-top menu -> All Settings -> Device Options -> Device Info -> Serial Number; it's a 4 groups of 4 characters string, WWWW XXXX YYYY ZZZZ. Also in my case, the config files was ~/.config/calibre/plugins/dedrm.json. You can write it by hand or use the GUI to add new serials. For the latter, it's Calibre -> Preferences -> Plugins -> File type -> DeDRM -> Customize plugin -> Kindle eInk ebooks -> + (add serial). Just notice it wants the serial as a single 16 char string.

Last step: convert them all:

mdione@ioniq:~/src/system/fuck-amazon$ for file in downloads/; do calibredb add $file --with-library=library; done

Notice two things: when I said "making sure not to delete the downloads the tool created", I meant moving that directory into this root and deleting everything else. Second, this just DeDRMs the files, and leaves them in a different directory (library), but Calibre won't see them until you really add them.

Note: due to my new setup, this post came out almost in raw form. It is now heavily edited as it should have been, but at least forced me to finish it :)

I started using Mastodon some three years ago, and since the beginning I started having issues adapting to how timelines work and how you actually should use the platform. I listed several things that I didn't like, and I always thought that at some point I would just sit down and write my own client. This weekend looked like it would rain all weekend, so it was the perfect time to do it.

As with any project that uses a database, I usually go for an ORM. I have already used SQLAlchemy on 3 or 4 projects, so it was natural for me to use it again. One of them is probably dead, but the rest get repeated usage; not daily, but many times a year for many years. Through all those years, most had the data layer written once and forgot about, because the data model is quite simple; and this case too. The problem is that every time I mostly have to relearn how to use it, specially since even when SQLAlchemy has evidently stayed quite backwards compatible, it has also evolved.

Version 2.0 has a very different interface. With the old version, if you wanted to fetch an object, a row from the database, you did something like this:

image = session.query(Image).filter_by(name=filename)[0]

But with the new version, they have exposed a thin abstraction of the SQL language; the previous exmaple becomes:

image = session.scalars(select(Image).where(User.name == filename))[0]

I love the magick¹ they are using to convert that expression into something they can later render as SQL, which involves rewriting __getattribute__() to return another object that implements __eq__() to capture the value.

It's like writing SQL in Python, right? And in some ways, it makes a lot of sense, because one of the things people usually complain about ORMs is that, at some point doing things that are easy in SQL they become too convoluted on ORMs. It's like you are bending yourself backwards just to try to kind of scratch your left ear using your right arm, but under your left leg.

So, putting a slight veneer of Python on top of SQL kind of makes sense, and I like that someone has explored that idea to the point that it has become the new way of doing things. And with that, I mean, I guess the old API still works, again, all the code I have written so far still works, and I still use it still works, but the new way looks better. But then another thing that over and sometimes too, and in particular, SQL actually seems to do too, is to have two levels of data returned. You may have, you may get objects, which is basically you go and fetch stuff from the database and you get objects in return, and it's the objects you are thinking of in terms of your application model and not in terms of your tables and other stuff, which is amazing because you can do one-to-end and end-to-end relationships, and it's just natural, because let's say a person has several addresses, you just say person.addresses.add, and you add an address there, and it just works. You don't have to think about anything there, you just add stuff and it will go in the database. Before you have an end-to-end relationship and you can go from photos to tags and from tags to photos, and it just works, right? You just get, you fetch a tag and say, give me all the photos, and it just gives you the photos and you completely forget about the intermediary table, and that's just amazing. It just works and it's fantastic. Then there's the other API where you don't get objects, you just get data from the database. You get roles, you get tuples with data inside, which has no structure and anything else. After writing this Friday night, what I did in Friday night was mostly to look at the responses from a mastern, servers, see what they look like, and try to make data model based on that. I am not going to complete the model accounts or toots, because they have way more information than I want to, to display in my client, but that's all I did on Friday night. So the morning I started writing the model in SQL Alchemy, and it was just fine, just mapping JSON fields to tables and columns, and then to classes and objects and attributes. I started writing all the code and it was just fine, everything was kind of obvious in that sense. But at some point I hit a wall where I was going to, I was fetching all the, I was fetching accounts based on an ID, and instead of returning me objects, it was returning me tuples. So when I wanted to say, okay, I have a new toot, and this toot was tooted by this, toot or this account, if I don't know the account, I create a new one and then I create a toot with this new account, and I just push it to that list. But in the case of the account already existing and me getting a tuple instead of an object, when I was trying to say, okay, toot.author is this object, I was not putting an object, I was putting just a string of the ID, because that's all I had, that's all what the ORM was giving me. And of course later the ORM says, no, this doesn't work because I expect an object and you'll give me just an ID. And I spent like 24 hours on that, I mean, from Saturday afternoon to this morning, okay, not 24 hours, maybe like 10 hours on it. And asking everywhere and I didn't get a very good answer. So I just switched to SQLite 3. I just had to do it. So before my workflow was, if I needed to add a new column to a table, I only had to add a new attribute to the class and then destroy the database and create it again. One of the good decisions I did at the beginning was to store all the original information in raw, just getting the JSON and storing it in the database. So that means that if I make any changes to the data structure that I will use later, I can just read everything back from the database, create the new objects and save them in the database again. And this is something I could use to replace database structure migrations. I just nuke all the secondary tables and just create all the data again from scratch, yeah, from the original data. So I can do these migrations very easily now. So with SQL Alchemy, I could just drop the tables, create everything again and just feed, create all the objects again and that worked. But now I don't have that anymore. So what do I do? Well, my workflow now includes SQLite browser, which is a GUI application for creating, for manipulating SQLite files, right? And I just had to add a new column to the Tooth table, so I did just that in the GUI. And what I'm going to do at some point is to make a dump of the structure and keep it in a SQL file. So if I ever have to do this migration again, which I have to do at some point, I will just take that file, just executed it from scratch when I have to create the tables again and that's it. I will just do it that way. I really wish our, in this case, SQLite, SQL Alchemy would have been just transparent for me. But I guess that's exactly the kind of things people say about ORAMS, the type of criticism that ORAMS usually get, that 90% of the work is really transparent, but then at some point it is no longer so transparent and you start doing weird shit just to accomplish what you want. So what I'm going to do in the future, I'm not really sure. I will see how this raw SQLite works. This project is rather simple. I have very few tables, like I have two tables for the original raw content, one not 24 hours, maybe like 10 hours on it. And asking everywhere and I didn't get a very good answer. So I just switched to SQLite 3. I just had to do it. So before my workflow was, if I needed to add a new column to a table, I only had to add a new attribute to the class and then destroy the database and create it again. One of the good decisions I did at the beginning was to store all the original information in raw, just getting the JSON and storing it in the database. So that means that if I make any changes to the data structure that I will use later, I can just read everything back from the database, create the new objects and save them in the database again. And this is something I could use to replace database structure migrations. I just nuke all the secondary tables and just create all the data again from scratch, yeah, from the original data. So I can do these migrations very easily now. So with SQL Alchemy, I could just drop the tables, create everything again and just feed, create all the objects again and that worked. But now I don't have that anymore. So what do I do? Well, my workflow now includes SQLite browser, which is a GUI application for creating, for manipulating SQLite files, right? And I just had to add a new column to the Tooth table, so I did just that in the GUI. And what I'm going to do at some point is to make a dump of the structure and keep it in a SQL file. So if I ever have to do this migration again, which I have to do at some point, I will just take that file, just executed it from scratch when I have to create the tables again and that's it. I will just do it that way. I really wish our, in this case, SQLite, SQL Alchemy would have been just transparent for me. But I guess that's exactly the kind of things people say about ORAMS, the type of criticism that ORAMS usually get, that 90% of the work is really transparent, but then at some point it is no longer so transparent and you start doing weird shit just to accomplish what you want. So what I'm going to do in the future, I'm not really sure. I will see how this raw SQLite works. This project is rather simple. I have very few tables, like I have two tables for the original raw content, one for the actual raw content and one for the raw toot content. One of the things is when you boost another toot, you basically generate a toot that has a toot nested into it. So the original raw content includes all that information, but to me, for my client, the most important thing is the original toot itself. So I have another table that only has the original toots, not the boosters, and the client will just work from that. And then I have toots, tooters, and tacks so far. I'm not really sure if I will do a table with bookmarks and filters and all that shit, because those things usually stay on the server in your instance, and I should just query them all the time. Press also think of having an offline version where you can do stuff, and if you cannot send them immediately to the instance, you just queue them for later, for when you have a connection, or when you want to use a connection. So that's it. It's very, very simple data model. If this goes right, but later I have a more complex data model, I might try other over ends, especially simpler over ends that are less magic, a little bit more transparent, but yeah, we'll see when the time comes.

Just a quick note. Many years ago I started rendering my maps in batches. Then I found out that to increase speed, I could use metatiles. I found that 8 was a good size.

Now I developed a rendering tile server, which means I can query for tiles, and if they hagve not been redered yet, they can be rendered on the fly. But now the requirements changed. When rendering batches, I can keep the rendering pipeline busy because there's always another metatile to render (unless I'm at the end of the batch), so all the threads are always busy. But with such a big metatile, the latency is terrible. It's not a problem because I'm, not consuming the tiles immediately.

With the rendering tile server, I do care about latency. Those big metatiles can take several seconds to render, and as a user I don't want to wait that long to see the map. Also, because the client can see at most 4 metatiles, but usually just 1 or 2, the rest of the threads are sitting idle. So by reducing the metatile size to 2, I can reduce the latency by parallelyzing more. Granted, each individual tile takes more time to render in average, but the service becomes more snappy. So far a metatile size of 2 has proven good enough; I don't think I'll ever try with 1.

Look at the following Ansible task:

- name: Prometheus Exporters
  - name: Extend caps for ping - File
    template:
      src:   "roles/server/files{{ item }}"
      dest:  "{{ item }}"
      owner: prometheus
      group: root
      mode:  0600
    loop:
      - /etc/systemd/system/prometheus-blackbox-exporter.service.d/override.conf
    when: "'blackbox' in exporters"
    register: foo
    tags: config, monitoring
    notify:
    # - Reload systemd
    - Reload Blackbox Exporter

For a while I have been wishing to be able to write this like this:

class PrometheusExporters:
    def extend_caps_for_ping_File(self):
        if 'blackbox' in exporters:
            foo = []
            for exporter in exporters:
                # tags: config, monitoring
                foo.append(template(f"roles/server/files{exporter}", exporter, 'prometheus', 'root', 0o600, 
                           notify=(reload_blackbox_exporter,)))

Notice that these are not 100% equivalent; the Python version uses exporter as the loop_variable, but in the Ansible code I never set that because it's so cumbersome.

Why do I prefer that notation? Because:

• Even if Ansible aims to be a declarative language, it has many programming languages features like when (if), loop (for) and block/recover (try/except).
• Yes, the register equivalent I wrote there is not really nice, but alternatives are, I think, worse.
• When using certain modules I almost always use the same params. Notice that a proper translation would be template(src=f"roles/server/files{exporter}", dest=exporter, owner='prometheus', group='root', mode=0o600), and that this is very close to the inline module invocation a.k.a. free form arguments¹: template: src="roles/server/files{{ exporter }}" dest="{{ exporter }}" owner=prometheus group=root mode=0600.
• If the compiler would be clever enough, I could declare functions that would work as templates for tasks.
• As a programmer, this order makes more sense.

But I don't have to wait to have enough energy to write such compiler myself; I can more or less already change the order:

- name: Prometheus Exporters
  - name: Extend caps for ping - File
    when: "'blackbox' in exporters"
    loop:
      - /etc/systemd/system/prometheus-blackbox-exporter.service.d/override.conf
    register: foo
    template:
      src:   "roles/server/files{{ item }}"
      dest:  "{{ item }}"
      owner: prometheus
      group: root
      mode:  0600
    notify:
    # - Reload systemd
    - Reload Blackbox Exporter
    tags: config, monitoring

Not 100% where I want it (setting loop_variable is still ugly), but in my head it's easier to read. I don't use free form arguments because it's not clear that I can split the line like I did on the Python code.

Two days ago https://norcal.social/@superball asked about generating smart quotes on Linux. I never got to understand what they meant with 'smart quotes', but I answered with a reference to the US intl with dead keys keyboard layout. I have been using it for ages. This allowed me to write proper Spanish with a US layout and it just mimicked my experience when learning to type with a Olivetti Lexicon 80.

I knew for a long while that I gloat, yes, gloat, about being able to write 'weird' letters, but it seems like I never really wrote them all. So today I sat down to waste, yes, waste a couple of hours, as in more than two, compiling the following table:

The biggest surprises about it are:

• Overdot is the most used writable diacritic, even when 5 of the letters use a different diacritic.
• Underdot and caron get more than I expected.
• Only one letter with horn? Was it worth it? Maybe I don't know how to use it?
• Ogonoek is the best diacritic name. A shame is so underused :)
• If it wasn't for the overdot, x would be the single letter that can't be combined.
• Best symbol? I'm partial to ř because it's used to represent a Czech sound that is also present in the West North West part of my country, one ocean and one Equator apart. See https://www.youtube.com/watch?v=uDpVPj49R8w and, wow, https://www.youtube.com/watch?v=9cZSKnfeigI.
• But also ẋ and ơ for their different uniqueness.

Let me tell you about that last video. The guy is from Córdoba, from Cruz del Eje, so he has a mix of Cordobese accent but the ř sound from Traslasierra ('behind the mountains', referencing the Sierras Grandes) and La Rioja. Riojan people have another accent.

https://www.openstreetmap.org/relation/153536?mlat=-30.7336&mlon=-64.7943#map=8/-30.188/-66.621.

Of course, these are not all diacritics (see https://en.wikipedia.org/wiki/Diacritic), and probably not all the possible combinations. Not to mention that they can be accumulated; see both glyphs noted with ¹, but also https://en.wikipedia.org/w/index.php?title=Zalgo_text.

Finally, diacritics are not the only thing you can write with this layout. These symbols are also writable without combining:

¡¹²³¤£€¼½¾‘’¥×÷ äåé®™þüúíóö«» áßðëïœø¶° æ·©¢ñµ±ç¿

and ^ combined with digits writes them superscripted:

¹²³⁴⁵⁶⁷⁸⁹⁰

Definitely a very versatile layout. If you want to fully explore your keyboard layout, install tastenbrett¹⁸ and take a look.

Ah! And smart quotes are not actual glyphs but a feature (I completely forgot about it because I always deactivate them):

https://en.wikipedia.org/w/index.php?title=Quotation_marks_in_English#Smart_quotes.

Yesterday¹ I did something that I don't know how to qualify. At the beginning I was just playing a silly thing where I registered a new domain on a dynamic domain service. I set up Apache on my home server to answer this domain with a single 211 bytes index.html file, and wrote a toot with a link to that site. What I wanted to see is the mastodon effect, which is similar to the slashdot effect, but supposedly more automatic. The idea is that your followers' Fediverse servers would try to obtain a preview of the page to present the users. And because they might boost that toot to their followers, you can easily either get DDOS'ed by this, or get a huge bill for network traffic.

But for reasons that will be apparent soon, this post is going to be meta. What happened with the toot? Many things.

It has been more than 48 hours¹ since I posted the link and I have more than 2.4k boosts. It's definitely way, way more than I expected. My now second most popular toot² was about an antipodes map³, and it had 200 boosts or so. A couple of months ago, there was this European petition about taxing the rich. That petition needed a million signatures all over Europe. I tried to boost and write about it a lot, but I didn't get much traction, my toots about it only got 6 boosts. Those toots, in contrast to the experiment or the antipodes, were the ones that to me were more important to be spread. And yet, what gets more around from me are amusing toots.

And that is something I have been thinking about for a while. The people that I follow who have lots of followers, boosts and replies are mostly people that are mostly focused on one or maybe two main topics. My feed is all over the place: development, replacing cars with ebikes, 15m cities, things that amuse me, science, (astro)photography, languages, maps, history and god knows what else. And I came to the conclusion that I will never try to focus on one or two topics, that I want to keep my feed like that and that's fine, just don't get down if your interactions are more limited. If I want something more I will have to find it elsewhere (and my psychologist agrees :)

The second thing I want to talk about is that the toot has 418 likes. That's really interesting. I don't know why people liked the post, the post itself, maybe because I explained a little bit the idea behind the post, and of course that's why it has been boosted a lot, but why likes? I have expressed sometimes how I use replying, boosting, liking, and bookmarking myself. Replying means I think I have something to add or ask, or maybe just a joke, which I do often. Boosting is for things that are really, really interesting, things I want to spread, and that I probably don't know how to write about or have the time to do it about myself. Bookmarking is for either reading later (which most of the time never happens), save something on the phone to reply in the computer or vice versa (the phone is awkward to type in, but it has a camera); or I really want to keep it as reference.

To me likes mean two things. The first one is, of course, I like what you did that I don't consider that important enough to boost it, I just want to encourage you to do more of that. For instance, I like when people say that they have added some stuff into OpenStreetMap, made a little better something we all can enjoy, use and contribute. Sometimes I boost, but most of the time I just like, because I feel like I'm preaching to the choir here. Finally, I might like instead of saying 'Thanks'.

Third thing was that 56 people¹ followed me after the experiment toot. Getting extra 20% of followers in 48h feels weird. I don't know why people follow me, but this is definitely unexpected. I did not expect people to follow me just because I am doing this experiment; maybe they just saw something else in my feed and decided to follow :shrug:.

Fourth thing that this experiment did was to break my notifications. I mentioned that I use Phanpy, which luckily does not really send any notifications, it just adds a dot to the bell icon, so there has been no real intrusion in my life, but it means that now that dot by the bell is not useful for me anymore⁴ :) It just sits there because people keep boosting it. I have Phanpy configured such that I have four columns: Notifications, my home screen, the local feed for my server, and bookmarks (see above). I can see when notifications are arriving in that left column, but the icon doesn't work anymore. I also use Phanpy on my mobile phone (no, no dedicated app, I don't consider Mastodon a critical enough service to require my attention when something happens), where I only have one column and the bell icon is completely useless there⁴. I'm not sure if other Mastodon clients have this feature, but Phanpy has the option to only show notifications that have mentions, meaning replies. Phanpy also has had grouping notifications such as boosts and likes into a single element for a long time. I think newer Mastodon versions have this too. These three features have allowed me to keep effectively using this Mastodon account in these last two⁵ days.

Finally, people started commenting about it. Someone wondered if this was GDPR compliant. I am no GDPR expert. I know that it's all about what do you do with personal information. I don't have much personal information from the people that have boosted the toot. I could maybe ask the Mastodon API, but this is not the reason for the experiment. See the last paragraph for more details. The only thing I have is the Apache log, which indicates, among other things, the IP of the client and the user agent. The latter I will use to distinguish Fediverse services from actual people following the link. Then in 15 or few more days logrotate will rotate these logfiles out of existence. There is no permanent storage whatsoever, this is just a home server basic setup, nothing fancy. There is not even a log aggregator or nothing that I can use to massage log lines apart from what I already explained.

People started asking about the results. Sorry people, you will ave to be patient with me. I didn't expect to write this, but now that I am doing it I think it's more important that the raw numbers. They say that they have bookmarked the post so they can come back later and see what happened. What I usually do in these cases is to edit the original post so anybody who has interacted with the tooth will get a notification. But that is fine when it's just a few boosts and likes. Notifying more than 2.4k people where only a handful has expressed interest on the outcome of it sounds like too much. So I will do exactly what they suggest, just reply my own toot with the link to this post, and later another with the future technical post.

One person started by sending a private message telling another person who had evidently boosted the toot, telling them in Dutch that it was probably a scam and suggesting them to unboost the tooth. I answered them that this is no scam, that it is just an idiot trying to do an experiment that went out of hand. And they told me the experiment has been done a million times, which they're totally right, and that I am "profiting from friendly people who risk getting muted or blocked because I don't want to be bothered by experiments in my timeline". My answer to that was, literally, "I would have to think about this" and this is what spurred this post. I mean, I already did some meta commentary before where I talked about how things that I felt really important and pressing got mostly ignored while silly stuff like antipodes or a stupid experiment got more attention.

I can understand the part of where something amusing gets a lot of attention. I understand people that are using Mastodon to amuse themselves, mostly because I do that too. But "profiting from friendly people", I never thought it that way. I mean, maybe this person is an idiot, but I try to think when my actions have unintended consequences. And despite the tone, it made me think, which is good. "Profiting", I kind of am, because I'm asking for a favor from people I don't know. But I didn't force anybody to do what they did. Yes, initially the toot did not explain what it was for, but I edited it not even 15 minutes later mentioning what it was for, and I think that's when it really took off. People have been genuinely boosting this on their own accord. I have no leverage to force anybody to do it; if they do it, they do it because they want. And if that makes this person mute or block people because of that, it's on them. But it took me the whole day to figure this out.

Another person said several things. One was, the domain name suggest that I'm trying to DDoS Mastodon. Yeah, the name was very poorly chosen. It was a puny name in the sense that I just replaced the "do" in Mastodon with "DDoS", and becomes something like "mastodohsn't", which completely smashes the meaning of everything. It's a crappy domain name, I agree. He mentions not visiting ddns.net domains. I didn't think about that. I had had a dynamic domain between 2003 and 2012 for my home server. Back then it was the easiest way to get a domain name. Now I use one that my ISP provides me. But for this experiment I set up a completely different domain name, mostly because I didn't want to have my personal domain name posted publicly. So far I have maintained a very low profile about it. Definitely I have never posted a link to my home server on Mastodon. He mentions that chain letters are already being extensively researched. This is not what I'm doing. I'm interested on how much impact posting a link on the Fediverse has on web servers. "Either you have criminal intent or you are not a scientific researcher". I am not a scientific researcher, but I am not a criminal either. He mentions Sami Kamkar. Sami Kamkar created and released in 2005 the fastest-spreading videos of all time, the MySpace Warms Sami, and was subsequently raided by the United States Secret Service under the Patriot Act. My answer to him was it's a very poorly devised project that went out of hand, which is a very honest and accurate answer. I am definitely very far from being anything similar to Sami Kamkar. Again, I'm just going to peruse some logs. I'm going to try to figure out how to monitor those logs. And that's all. There is no other hidden agenda. There is no other interest. This is my home server. The logs were not going to be retained for any long period of time.

So, in conclusion, to whoever has boosted this, thank you. This is what I asked you to do and this is what you did, good job :) To whoever liked it, thank you too, I guess. To whoever has followed me because of that toot, welcome to a random guy's eclectic feed, I hope you stay. To those who have sent me messages because of the experiment or the toot or some other technical stuff, thank you for being so interested. Thank you to those who made me doubt myself, because it's one of the ways I grow, by double checking myself and deciding whether what I did was right or wrong and I should apologize.

And I think that's it. I hope to write the analysis soon. Right now I have around one hits per minute, so probably for tomorrow night when I will have the time to sit down and do stuff on this.

Just a quick one: dpkg -S does not handle symlinks, and now that the usrmerge is almost passed (I think?), it's getting more annoying:

mdione@ioniq:~$ dpkg -S /lib/x86_64-linux-gnu/libgexiv2.so.2.14.3
dpkg-query: no path found matching pattern /lib/x86_64-linux-gnu/libgexiv2.so.2.14.3

mdione@ioniq:~$ dpkg -L libgexiv2-2:amd64
/usr/lib/x86_64-linux-gnu/libgexiv2.so.2.14.3

mdione@ioniq:~$ namei -lx /lib/x86_64-linux-gnu/libgexiv2.so.2.14.3
f: /lib/x86_64-linux-gnu/libgexiv2.so.2.14.3
Drwxr-xr-x root root /
lrwxrwxrwx root root lib -> usr/lib
drwxr-xr-x root root   usr
drwxr-xr-x root root   lib
drwxr-xr-x root root x86_64-linux-gnu
-rw-r--r-- root root libgexiv2.so.2.14.3

/lib has been moved to /usr/lib and symlinked for compatibility reasons. There's 22yo (!!!) bug about it, which was deemed important 8ya. I hope it gets fixed soon :)

One of the parts of having my own map style with hypsometric contour lines is that I have to generate those contour lines. There's a tool in GDAL, particularly the one that actually does everything based on DEM files, called gdaldem that can generate shapefiles with contour lines that mapnik can read. But since my source files are 1x1° files, I will have to generate one layer for each shapefile and that doesn't scale very well, especially at planet size.

So what I do is I convert those shapefiles to SQL files and then I inject them into my database one by one, and then I can use mapnik's own support for filtering by bbox when it's rendering, so that should be faster⁴.

I put the SQL files in my file system, and then I import them by hand as I need them, and I'm running out of space again. A few years ago I had a 1TB disk, and that was enough, and now I am at the 2TB disk, and it's getting small. I have the impression that the new DEMs I am using are bigger, even if I streamlined every layer so it uses as less space as possible.

One of the things I'm doing is converting my processing script into a Makefile, so I can remove intermediary files. My process goes from the original DEM files, that are in LatLon, I project them to WebMerkator. This file becomes the source for the terrain files, which gives the hypsometric tints, and I generate the contours from there, and then I do a compensation for slope shade and hill shade. Notice that I get two intermediary files that I can easily remove, which are first, the reprojected file, because once I have the terrain and contour files, I can remove it, I don't care anymore; and also the compensated file, I don't need it anymore once I have the shade files. The Makefile is covering that part, once the files are generated, the intermediary files are gone.

Going back to the SQL files, I don't inject SQL data directly into my database, because I don't have space for that. So, I just generate this SQL file and I compress it, so it's not using so much space, because SQL is really a lot of text. I've been using xz as the compressor, and I have been blindly using its highest compression level, CL 9. What do I mean with blindly? I noticed it actually takes a lot of time. I just measured it with one tile, and it took 451 seconds. That's 7.5 minutes per degree tile, which is a lot. So I asked myself, what's the compression ratio to time spent ratio?

I took a single file and I compressed it with all the compression levels between 1 and 9, and I took the time and the space in the final file. I made a scatter graph, and it looks like this pretty weird Z figure²:

Here's the raw data¹:

I'm not going to explain the graph or table, except to point to the two obvious parts: the jump from CL 3 to 4, where it's not only the first and only noticeable space gain, it also takes less time; and the fact that compressions levels 1-3 and 4-9 have almost no change in space gained. So I either use CL 1 or 4. I'll go for 1, until I run out of space again.

All this to say: whenever you make an optimization, measure all the dimensions, time, space, memory consumption, and maybe you have other constraints like, I don't know, heat produced, stuff like that. Measure and compare.