«So, by simply copying that one and ast.py, modifying the latter so it uses my modified syntax checker, and modifying the setup.py file, I should be done». Famous last words.

I was completely wrong. The _ast module imported by ast.py is not implemented in Python/ast.c but in Python/Python-ast.c. Understanding this, I reread this post I linked to last time and finally[1] understood that changing the syntax would not be so easy. Another eye-opener was the stack trace I got when ast.c's ast_for_call() (the function I'm planning to change) is called when a function call is found in the source. Here, have a look:

#0  ast_for_call (c=0x7fffffffa300, n=0x7ffff700d918, func=0x9ccf40) at Python/ast.c:2470
#1  0x00000000005bfe73 in ast_for_trailer (c=0x7fffffffa300, n=0x7ffff700db98, left_expr=0x9ccf40) at Python/ast.c:2105
#2  0x00000000005c03c0 in ast_for_power (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2217
#3  0x00000000005c0b01 in ast_for_expr (c=0x7fffffffa300, n=0x7ffff6ca4238) at Python/ast.c:2406
#4  0x00000000005c1108 in ast_for_testlist (c=0x7fffffffa300, n=0x7ffff6ca4030) at Python/ast.c:2562
#5  0x00000000005c118e in ast_for_expr_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:2584
#6  0x00000000005c3e84 in ast_for_stmt (c=0x7fffffffa300, n=0x7ffff6facfa8) at Python/ast.c:3586
#7  0x00000000005bc94c in <span class="createlink">PyAST FromNodeObject</span> (n=0x7ffff6face18, flags=0x7fffffffa4e0, filename=0x7ffff6d56370, arena=0x8c8780) at Python/ast.c:709
#8  0x000000000050c9e8 in <span class="createlink">PyParser ASTFromStringObject</span> (s=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, arena=0x8c8780) at Python/pythonrun.c:2160
#9  0x000000000050c6be in <span class="createlink">Py CompileStringObject</span> (str=0x7ffff6ca37a0 "foo (a=3, b)", filename=0x7ffff6d56370, start=257, flags=0x7fffffffa4e0, optimize=-1) at Python/pythonrun.c:2068
#10 0x00000000004c6e54 in builtin_compile (self=0x7ffff70f9c28, args=0x7ffff6ca04f8, kwds=0x0) at Python/bltinmodule.c:671

The compile() builtin (#10) passes through a couple of obscure functions till it reaches the AST build/syntax check code (all the ast_for_*() functions), and we quickly reach it from ast.parse():

def parse(source, filename='<unknown>', mode='exec'):
    return compile(source, filename, mode, PyCF_ONLY_AST)

In a desperate move, I though of hacking a modified version of just that function and do some LD_PRELOAD jedi moves, but the Python interpreter binary (/usr/bin/python3) is statically linked against libpython3. Future versions[2] of the Python interpreter main() function[3] will be really simple:

/* Minimal main program -- everything is loaded from the library */

#include "Python.h"

#ifdef __FreeBSD__
#include <floatingpoint.h>
#endif

int
main(int argc, char **argv)
{
        /* 754 requires that FP exceptions run in "no stop" mode by default,
         * and until C vendors implement C99's ways to control FP exceptions,
         * Python requires non-stop mode.  Alas, some platforms enable FP
         * exceptions by default.  Here we disable them.
         */
#ifdef __FreeBSD__
        fp_except_t m;

        m = fpgetmask();
        fpsetmask(m & ~FP_X_OFL);
#endif
        return Py_Main(argc, argv);
}

So it's not so crazy to think of creating a new interpreter that uses my version of ast_for_call(), but I haven't found the linking command that makes that possible. So this part of the project is in the freezer.

Meanwhile, I spent a couple of hours with the code and managed to implement piping and redirection, even with some tests! You can have an idea of how it works reading the README[4]. Also, expect a release soon.

As for replacing sh, no efforts (besides burning some glucose and firing some neurons) have been done.


[1] Also famous last words.

[2] Python-3.3 has a more complex version which has to handle making two copies of argv while dealing with the system's encoding.

[3] It's weird, the source code of the executable resides in Modules/python.c. Go figure.

[4] Some day I'll convert that file to reST, make it the tutorial, and have it with the rest of the documentation.


python ayrton