Python – How to make decorators as powerful as macros

language-designlisppattern matchingpython

Quick background: I am designing a Pythonic language that I want to be as powerful as Lisp while remaining easy to use. And by "powerful", I mean "flexible and expressive".

I've just been introduced to Python function decorators, and I like them. I found this question that suggests decorators are not as powerful as Lisp macros. The third answer to this question says that the reason they're not is because they can only operate on functions.

But what if you could use decorators on arbitrary expressions? (In my language, expressions are first-class, just like functions and everything else, so doing that would be fairly natural.) Would that make decorators as powerful as Lisp macros? Or would I have to use syntax rules, like Racket and Honu, or some other technique?

EDIT

I was asked to provide an example of the syntactic construct. For an "expression decorator", it would simply be a decorator on an expression (where "$(…)" creates an expression object):

@Decorator("Params")
$(if (true):
    printf("Was true")
else:
    printf("Wasn't true"))

Also, if I do need to use syntax rules, as mentioned above, it would use a pattern matching syntax:

# Match a bool expression, a comma, and a string.
macro assert(expr:bool lit string) { e , s }:
    if (!e):
        error s
    else:
        printf("All good")

The above would match this:

assert i == 0, "i should have been 0"

That's a rough sketch, but it should give you an idea.

Best Answer

Short answer: You can't.

Not in Python or anything close to what it is today, at any rate. The reason is straightforward: Functions are objects. Expressions are not. Expressions describe activities--the activity of operating on objects and producing a result. While the input and output of expressions are objects, the expressions themselves are not. Because they are not objects, they cannot be decorated.

This limitation on Python metaprogramming is pretty baked into the language. It's not impossible to change Python semantics. Modules like forbiddenfruit let you monkeypatch built in objects in ways that are officially "impossible." Modules like MacroPy, karnickel, and the now apparently defunct MetaPython do provide some macro facilities through AST rewriting and related techniques. None of these is certified for production use, however, and most of them pop up, support one or two language iterations, and then evaporate. The double-back-flip-with-twist moves necessary to make them ever work at all just are not sustainable over any period of time, in any generality, or in the context of "code you can depend on."

There are modules that successfully use introspection and reflection to inspect what is currently happening in a program. My say module, for example, works well across many Python 2 and Python 3 versions, and even alternate Python implementations like PyPy. But few Python meta-modules can actually change program semantics. GvR has been pretty resistant to metaprogramming facilities. The with statement semantics, for example, were constrained for the particular purpose of limiting how much metaprogramming it can support.

But, that's Python as it exists today. You might imagine your Python++ language having arbitrary code blocks (the mythical multi-line lambda, often mentioned but never actually seen in Python) that can be decorated. Lambdas are objects, and code blocks are very object-ify-able. You probably would not want the overhead of making every expression into an object, but it's not hard to imagine an environment in which any expression that was interesting to encode in a block could be done so, then decorated.

Two standard (i.e. non-Lispy) and quite Pythonic languages that have worked to integrate metaprogramming and advanced macro facilities are Julia and Nim (formerly known as Nimrod). The way they address program representation and transformation are very powerful, and would be good study points for your design.

Related Topic