ayrton: an update

«So, by simply copying that one and ast.py, modifying the latter so it uses my modified syntax checker, and modifying the setup.py file, I should be done». Famous last words.

I was completely wrong. The _ast module imported by ast.py is not implemented in Python/ast.c but in Python/Python-ast.c. Understanding this, I reread this post I linked to last time and finally1 understood that changing the syntax would not be so easy. Another eye-opener was the stack trace I got when ast.c's ast_for_call() (the function I'm planning to change) is called when a function call is found in the source. Here, have a look:

#0  ast_for_call (c=0x7fffffffa300, n=0x7ffff700d918, func=0x9ccf40) at Python/ast.c:2470
#1  0x00000000005bfe73 in ast_for_trailer (c=0x7fffffffa300, n=0x7ffff700db98, left_expr=0x9ccf40) at Python/ast.c:2105
#2  0x00000000005c03c0 in ast_for_power (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2217
#3  0x00000000005c0b01 in ast_for_expr (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2406
#4  0x00000000005c1108 in ast_for_testlist (c=0x7fffffffa300, n=0x7ffff6ca4030) at Python/ast.c:2562
#5  0x00000000005c118e in ast_for_expr_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:2584
#6  0x00000000005c3e84 in ast_for_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:3586
#7  0x00000000005bc94c in PyAST_FromNodeObject (n=0x7ffff6face18, flags=0x7fffffffa4e0, filename=0x7ffff6d56370, arena=0x8c8780) at Python/ast.c:709
#8  0x000000000050c9e8 in PyParser_ASTFromStringObject (s=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, arena=0x8c8780) at Python/pythonrun.c:2160
#9  0x000000000050c6be in Py_CompileStringObject (str=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, optimize=-1) at Python/pythonrun.c:2068
#10 0x00000000004c6e54 in builtin_compile (self=0x7ffff70f9c28, args=0x7ffff6ca04f8, kwds=0x0) at Python/bltinmodule.c:671

The compile() builtin (#10) passes through a couple of obscure functions till it reaches the AST build/syntax check code (all the ast_for_*() functions), and we quickly reach it from ast.parse():

def parse(source, filename='<unknown>', mode='exec'):
    return compile(source, filename, mode, PyCF_ONLY_AST)

In a desperate move, I though of hacking a modified version of just that function and do some LD_PRELOAD jedi moves, but the Python interpreter binary (/usr/bin/python3) is statically linked against libpython3. Future versions2 of the Python interpreter main() function3will be really simple:

/* Minimal main program -- everything is loaded from the library */

#include "Python.h"

#ifdef __FreeBSD__
#include <floatingpoint.h>
#endif

int
main(int argc, char **argv)
{
        /* 754 requires that FP exceptions run in "no stop" mode by default,
         * and until C vendors implement C99's ways to control FP exceptions,
         * Python requires non-stop mode.  Alas, some platforms enable FP
         * exceptions by default.  Here we disable them.
         */
#ifdef __FreeBSD__
        fp_except_t m;

        m = fpgetmask();
        fpsetmask(m & ~FP_X_OFL);
#endif
        return Py_Main(argc, argv);
}

So it's not so crazy to think of creating a new interpreter that uses my version of ast_for_call(), but I haven't found the linking command that makes that possible. So this part of the project is in the freezer.

Meanwhile, I spent a couple of hours with the code and managed to implement piping and redirection, even with some tests! You can have an idea of how it works reading the README4. Also, expect a release soon.

As for replacing sh, no efforts (besides burning some glucose and firing some neurons) have been done.


  1. Also famous last words. 

  2. Python-3.3 has a more complex version which has to handle making two copies of argv while dealing with the system's encoding. 

  3. It's weird, the source code of the executable resides in Modules/python.c. Go figure. 

  4. Some day I'll convert that file to reST, make it the tutorial, and have it with the rest of the documentation.