Python – Creating a class within a function and access a function defined in the containing function’s scope

Edit:

See my full answer at the bottom of this question.

tl;dr answer: Python has statically nested scopes. The static
aspect can interact with the implicit variable declarations, yielding non-obvious results.

(This can be especially surprising because of the language's generally dynamic nature).

I thought I had a pretty good handle on Python's scoping rules, but this problem has me thoroughly stymied, and my google-fu has failed me (not that I'm surprised – look at the question title 😉

I'm going to start with a few examples that work as expected, but feel free to skip to example 4 for the juicy part.

Example 1.

>>> x = 3
>>> class MyClass(object):
...     x = x
... 
>>> MyClass.x
3

Straightforward enough: during class definition we're able to access the variables defined in the outer (in this case global) scope.

Example 2.

>>> def mymethod(self):
...     return self.x
... 
>>> x = 3
>>> class MyClass(object):
...     x = x
...     mymethod = mymethod
...
>>> MyClass().mymethod()
3

Again (ignoring for the moment why one might want to do this), there's nothing unexpected here: we can access functions in the outer scope.

Note: as Frédéric pointed out below, this function doesn't seem to work. See Example 5 (and beyond) instead.

Example 3.

>>> def myfunc():
...     x = 3
...     class MyClass(object):
...         x = x
...     return MyClass
... 
>>> myfunc().x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in myfunc
  File "<stdin>", line 4, in MyClass
NameError: name 'x' is not defined

This is essentially the same as example 1: we're accessing the outer scope from within the class definition, just this time that scope isn't global, thanks to myfunc().

Edit 5: As @user3022222 pointed out below, I botched this example in my original posting. I believe this fails because only functions (not other code blocks, like this class definition) can access variables in the enclosing scope. For non-function code blocks, only local, global and built-in variables are accessible. A more thorough explanation is available in this question

One more:

Example 4.

>>> def my_defining_func():
...     def mymethod(self):
...         return self.y
...     class MyClass(object):
...         mymethod = mymethod
...         y = 3
...     return MyClass
... 
>>> my_defining_func()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in my_defining_func
  File "<stdin>", line 5, in MyClass
NameError: name 'mymethod' is not defined

Um…excuse me?

What makes this any different from example 2?

I'm completely befuddled. Please sort me out.
Thanks!

P.S. on the off-chance that this isn't just a problem with my understanding, I've tried this on Python 2.5.2 and Python 2.6.2. Unfortunately those are all I have access to at the moment, but they both exhibit the same behaviour.

Edit
According to http://docs.python.org/tutorial/classes.html#python-scopes-and-namespaces: at any time during execution, there are at least three nested scopes whose namespaces are directly accessible:

• the innermost scope, which is
  searched first, contains the local
  names
• the scopes of any enclosing
  functions, which are searched
  starting with the nearest enclosing
  scope, contains non-local, but also
  non-global names
• the next-to-last scope contains the
  current module’s global names
• the outermost scope (searched last)
  is the namespace containing built-in
  names

#4. seems to be a counter-example to the second of these.

Edit 2

Example 5.

>>> def fun1():
...     x = 3
...     def fun2():
...         print x
...     return fun2
... 
>>> fun1()()
3

Edit 3

As @Frédéric pointed out the assignment of to a variable of the same name as it has in the outer scope seems to "mask" the outer variable, preventing the assignment from functioning.

So this modified version of Example 4 works:

def my_defining_func():
    def mymethod_outer(self):
        return self.y
    class MyClass(object):
        mymethod = mymethod_outer
        y = 3
    return MyClass

my_defining_func()

However this doesn't:

def my_defining_func():
    def mymethod(self):
        return self.y
    class MyClass(object):
        mymethod_temp = mymethod
        mymethod = mymethod_temp
        y = 3
    return MyClass

my_defining_func()

I still don't fully understand why this masking occurs: shouldn't the name binding occur when the assignment happens?

This example at least provides some hint (and a more useful error message):

>>> def my_defining_func():
...     x = 3
...     def my_inner_func():
...         x = x
...         return x
...     return my_inner_func
... 
>>> my_defining_func()()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in my_inner_func
UnboundLocalError: local variable 'x' referenced before assignment
>>> my_defining_func()
<function my_inner_func at 0xb755e6f4>

So it appears that the local variable is defined at function creation (which succeeds), resulting in the local name being "reserved" and thus masking the outer-scope name when the function is called.

Interesting.

Thanks Frédéric for the answer(s)!

For reference, from the python docs:

It is important to realize that scopes
are determined textually: the global
scope of a function defined in a
module is that module’s namespace, no
matter from where or by what alias the
function is called. On the other hand,
the actual search for names is done
dynamically, at run time — however,
the language definition is evolving
towards static name resolution, at
“compile” time, so don’t rely on
dynamic name resolution! (In fact,
local variables are already determined
statically.)

Edit 4

The Real Answer

This seemingly confusing behaviour is caused by Python's statically nested scopes as defined in PEP 227. It actually has nothing to do with PEP 3104.

From PEP 227:

The name resolution rules are typical
for statically scoped languages […]
[except] variables are not declared.
If a name binding operation occurs
anywhere in a function, then that name
is treated as local to the function
and all references refer to the local
binding. If a reference occurs before
the name is bound, a NameError is
raised.

[…]

An example from Tim Peters demonstrates the potential pitfalls of
nested scopes in the absence of declarations:

i = 6
def f(x):
    def g():
        print i
    # ...
    # skip to the next page
    # ...
    for i in x:  # ah, i *is* local to f, so this is what g sees
        pass
    g()

The call to g() will refer to the variable i bound in f() by the for
loop. If g() is called before the loop is executed, a NameError will
be raised.

Lets run two simpler versions of Tim's example:

>>> i = 6
>>> def f(x):
...     def g():
...             print i
...     # ...
...     # later
...     # ...
...     i = x
...     g()
... 
>>> f(3)
3

when g() doesn't find i in its inner scope, it dynamically searches outwards, finding the i in f's scope, which has been bound to 3 through the i = x assignment.

But changing the order the final two statements in f causes an error:

>>> i = 6
>>> def f(x):
...     def g():
...             print i
...     # ...
...     # later
...     # ...
...     g()
...     i = x  # Note: I've swapped places
... 
>>> f(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in f
  File "<stdin>", line 3, in g
NameError: free variable 'i' referenced before assignment in enclosing scope

Remembering that PEP 227 said "The name resolution rules are typical for statically scoped languages", lets look at the (semi-)equivalent C version offer:

// nested.c
#include <stdio.h>

int i = 6;
void f(int x){
    int i;  // <--- implicit in the python code above
    void g(){
        printf("%d\n",i);
    }
    g();
    i = x;
    g();
}

int main(void){
    f(3);
}

compile and run:

$ gcc nested.c -o nested
$ ./nested 
134520820
3

So while C will happily use an unbound variable (using whatever happens to have been stored there before: 134520820, in this case), Python (thankfully) refuses.

As an interesting side-note, statically nested scopes enable what Alex Martelli has called "the single most important optimization the Python compiler does: a function's local variables are not kept in a dict, they're in a tight vector of values, and each local variable access uses the index in that vector, not a name lookup."