Classes as objects
Before understanding metaclasses, you need to master classes in Python. And Python has a very peculiar idea of what classes are, borrowed from the Smalltalk language.
In most languages, classes are just pieces of code that describe how to produce an object. That's kinda true in Python too:
>>> class ObjectCreator(object):
... pass
...
>>> my_object = ObjectCreator()
>>> print(my_object)
<__main__.ObjectCreator object at 0x8974f2c>
But classes are more than that in Python. Classes are objects too.
Yes, objects.
As soon as you use the keyword class
, Python executes it and creates
an object. The instruction
>>> class ObjectCreator(object):
... pass
...
creates in memory an object with the name ObjectCreator
.
This object (the class) is itself capable of creating objects (the instances),
and this is why it's a class.
But still, it's an object, and therefore:
- you can assign it to a variable
- you can copy it
- you can add attributes to it
- you can pass it as a function parameter
e.g.:
>>> print(ObjectCreator) # you can print a class because it's an object
<class '__main__.ObjectCreator'>
>>> def echo(o):
... print(o)
...
>>> echo(ObjectCreator) # you can pass a class as a parameter
<class '__main__.ObjectCreator'>
>>> print(hasattr(ObjectCreator, 'new_attribute'))
False
>>> ObjectCreator.new_attribute = 'foo' # you can add attributes to a class
>>> print(hasattr(ObjectCreator, 'new_attribute'))
True
>>> print(ObjectCreator.new_attribute)
foo
>>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable
>>> print(ObjectCreatorMirror.new_attribute)
foo
>>> print(ObjectCreatorMirror())
<__main__.ObjectCreator object at 0x8997b4c>
Creating classes dynamically
Since classes are objects, you can create them on the fly, like any object.
First, you can create a class in a function using class
:
>>> def choose_class(name):
... if name == 'foo':
... class Foo(object):
... pass
... return Foo # return the class, not an instance
... else:
... class Bar(object):
... pass
... return Bar
...
>>> MyClass = choose_class('foo')
>>> print(MyClass) # the function returns a class, not an instance
<class '__main__.Foo'>
>>> print(MyClass()) # you can create an object from this class
<__main__.Foo object at 0x89c6d4c>
But it's not so dynamic, since you still have to write the whole class yourself.
Since classes are objects, they must be generated by something.
When you use the class
keyword, Python creates this object automatically. But as
with most things in Python, it gives you a way to do it manually.
Remember the function type
? The good old function that lets you know what
type an object is:
>>> print(type(1))
<type 'int'>
>>> print(type("1"))
<type 'str'>
>>> print(type(ObjectCreator))
<type 'type'>
>>> print(type(ObjectCreator()))
<class '__main__.ObjectCreator'>
Well, type
has a completely different ability, it can also create classes on the fly. type
can take the description of a class as parameters,
and return a class.
(I know, it's silly that the same function can have two completely different uses according to the parameters you pass to it. It's an issue due to backward
compatibility in Python)
type
works this way:
type(name, bases, attrs)
Where:
name
: name of the class
bases
: tuple of the parent class (for inheritance, can be empty)
attrs
: dictionary containing attributes names and values
e.g.:
>>> class MyShinyClass(object):
... pass
can be created manually this way:
>>> MyShinyClass = type('MyShinyClass', (), {}) # returns a class object
>>> print(MyShinyClass)
<class '__main__.MyShinyClass'>
>>> print(MyShinyClass()) # create an instance with the class
<__main__.MyShinyClass object at 0x8997cec>
You'll notice that we use MyShinyClass
as the name of the class
and as the variable to hold the class reference. They can be different,
but there is no reason to complicate things.
type
accepts a dictionary to define the attributes of the class. So:
>>> class Foo(object):
... bar = True
Can be translated to:
>>> Foo = type('Foo', (), {'bar':True})
And used as a normal class:
>>> print(Foo)
<class '__main__.Foo'>
>>> print(Foo.bar)
True
>>> f = Foo()
>>> print(f)
<__main__.Foo object at 0x8a9b84c>
>>> print(f.bar)
True
And of course, you can inherit from it, so:
>>> class FooChild(Foo):
... pass
would be:
>>> FooChild = type('FooChild', (Foo,), {})
>>> print(FooChild)
<class '__main__.FooChild'>
>>> print(FooChild.bar) # bar is inherited from Foo
True
Eventually, you'll want to add methods to your class. Just define a function
with the proper signature and assign it as an attribute.
>>> def echo_bar(self):
... print(self.bar)
...
>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})
>>> hasattr(Foo, 'echo_bar')
False
>>> hasattr(FooChild, 'echo_bar')
True
>>> my_foo = FooChild()
>>> my_foo.echo_bar()
True
And you can add even more methods after you dynamically create the class, just like adding methods to a normally created class object.
>>> def echo_bar_more(self):
... print('yet another method')
...
>>> FooChild.echo_bar_more = echo_bar_more
>>> hasattr(FooChild, 'echo_bar_more')
True
You see where we are going: in Python, classes are objects, and you can create a class on the fly, dynamically.
This is what Python does when you use the keyword class
, and it does so by using a metaclass.
Metaclasses are the 'stuff' that creates classes.
You define classes in order to create objects, right?
But we learned that Python classes are objects.
Well, metaclasses are what create these objects. They are the classes' classes,
you can picture them this way:
MyClass = MetaClass()
my_object = MyClass()
You've seen that type
lets you do something like this:
MyClass = type('MyClass', (), {})
It's because the function type
is in fact a metaclass. type
is the
metaclass Python uses to create all classes behind the scenes.
Now you wonder "why the heck is it written in lowercase, and not Type
?"
Well, I guess it's a matter of consistency with str
, the class that creates
strings objects, and int
the class that creates integer objects. type
is
just the class that creates class objects.
You see that by checking the __class__
attribute.
Everything, and I mean everything, is an object in Python. That includes integers,
strings, functions and classes. All of them are objects. And all of them have
been created from a class:
>>> age = 35
>>> age.__class__
<type 'int'>
>>> name = 'bob'
>>> name.__class__
<type 'str'>
>>> def foo(): pass
>>> foo.__class__
<type 'function'>
>>> class Bar(object): pass
>>> b = Bar()
>>> b.__class__
<class '__main__.Bar'>
Now, what is the __class__
of any __class__
?
>>> age.__class__.__class__
<type 'type'>
>>> name.__class__.__class__
<type 'type'>
>>> foo.__class__.__class__
<type 'type'>
>>> b.__class__.__class__
<type 'type'>
So, a metaclass is just the stuff that creates class objects.
You can call it a 'class factory' if you wish.
type
is the built-in metaclass Python uses, but of course, you can create your
own metaclass.
In Python 2, you can add a __metaclass__
attribute when you write a class (see next section for the Python 3 syntax):
class Foo(object):
__metaclass__ = something...
[...]
If you do so, Python will use the metaclass to create the class Foo
.
Careful, it's tricky.
You write class Foo(object)
first, but the class object Foo
is not created
in memory yet.
Python will look for __metaclass__
in the class definition. If it finds it,
it will use it to create the object class Foo
. If it doesn't, it will use
type
to create the class.
Read that several times.
When you do:
class Foo(Bar):
pass
Python does the following:
Is there a __metaclass__
attribute in Foo
?
If yes, create in-memory a class object (I said a class object, stay with me here), with the name Foo
by using what is in __metaclass__
.
If Python can't find __metaclass__
, it will look for a __metaclass__
at the MODULE level, and try to do the same (but only for classes that don't inherit anything, basically old-style classes).
Then if it can't find any __metaclass__
at all, it will use the Bar
's (the first parent) own metaclass (which might be the default type
) to create the class object.
Be careful here that the __metaclass__
attribute will not be inherited, the metaclass of the parent (Bar.__class__
) will be. If Bar
used a __metaclass__
attribute that created Bar
with type()
(and not type.__new__()
), the subclasses will not inherit that behavior.
Now the big question is, what can you put in __metaclass__
?
The answer is something that can create a class.
And what can create a class? type
, or anything that subclasses or uses it.
The syntax to set the metaclass has been changed in Python 3:
class Foo(object, metaclass=something):
...
i.e. the __metaclass__
attribute is no longer used, in favor of a keyword argument in the list of base classes.
The behavior of metaclasses however stays largely the same.
One thing added to metaclasses in Python 3 is that you can also pass attributes as keyword-arguments into a metaclass, like so:
class Foo(object, metaclass=something, kwarg1=value1, kwarg2=value2):
...
Read the section below for how Python handles this.
The main purpose of a metaclass is to change the class automatically,
when it's created.
You usually do this for APIs, where you want to create classes matching the
current context.
Imagine a stupid example, where you decide that all classes in your module
should have their attributes written in uppercase. There are several ways to
do this, but one way is to set __metaclass__
at the module level.
This way, all classes of this module will be created using this metaclass,
and we just have to tell the metaclass to turn all attributes to uppercase.
Luckily, __metaclass__
can actually be any callable, it doesn't need to be a
formal class (I know, something with 'class' in its name doesn't need to be
a class, go figure... but it's helpful).
So we will start with a simple example, by using a function.
# the metaclass will automatically get passed the same argument
# that you usually pass to `type`
def upper_attr(future_class_name, future_class_parents, future_class_attrs):
"""
Return a class object, with the list of its attribute turned
into uppercase.
"""
# pick up any attribute that doesn't start with '__' and uppercase it
uppercase_attrs = {
attr if attr.startswith("__") else attr.upper(): v
for attr, v in future_class_attrs.items()
}
# let `type` do the class creation
return type(future_class_name, future_class_parents, uppercase_attrs)
__metaclass__ = upper_attr # this will affect all classes in the module
class Foo(): # global __metaclass__ won't work with "object" though
# but we can define __metaclass__ here instead to affect only this class
# and this will work with "object" children
bar = 'bip'
Let's check:
>>> hasattr(Foo, 'bar')
False
>>> hasattr(Foo, 'BAR')
True
>>> Foo.BAR
'bip'
Now, let's do exactly the same, but using a real class for a metaclass:
# remember that `type` is actually a class like `str` and `int`
# so you can inherit from it
class UpperAttrMetaclass(type):
# __new__ is the method called before __init__
# it's the method that creates the object and returns it
# while __init__ just initializes the object passed as parameter
# you rarely use __new__, except when you want to control how the object
# is created.
# here the created object is the class, and we want to customize it
# so we override __new__
# you can do some stuff in __init__ too if you wish
# some advanced use involves overriding __call__ as well, but we won't
# see this
def __new__(upperattr_metaclass, future_class_name,
future_class_parents, future_class_attrs):
uppercase_attrs = {
attr if attr.startswith("__") else attr.upper(): v
for attr, v in future_class_attrs.items()
}
return type(future_class_name, future_class_parents, uppercase_attrs)
Let's rewrite the above, but with shorter and more realistic variable names now that we know what they mean:
class UpperAttrMetaclass(type):
def __new__(cls, clsname, bases, attrs):
uppercase_attrs = {
attr if attr.startswith("__") else attr.upper(): v
for attr, v in attrs.items()
}
return type(clsname, bases, uppercase_attrs)
You may have noticed the extra argument cls
. There is
nothing special about it: __new__
always receives the class it's defined in, as the first parameter. Just like you have self
for ordinary methods which receive the instance as the first parameter, or the defining class for class methods.
But this is not proper OOP. We are calling type
directly and we aren't overriding or calling the parent's __new__
. Let's do that instead:
class UpperAttrMetaclass(type):
def __new__(cls, clsname, bases, attrs):
uppercase_attrs = {
attr if attr.startswith("__") else attr.upper(): v
for attr, v in attrs.items()
}
return type.__new__(cls, clsname, bases, uppercase_attrs)
We can make it even cleaner by using super
, which will ease inheritance (because yes, you can have metaclasses, inheriting from metaclasses, inheriting from type):
class UpperAttrMetaclass(type):
def __new__(cls, clsname, bases, attrs):
uppercase_attrs = {
attr if attr.startswith("__") else attr.upper(): v
for attr, v in attrs.items()
}
return super(UpperAttrMetaclass, cls).__new__(
cls, clsname, bases, uppercase_attrs)
Oh, and in Python 3 if you do this call with keyword arguments, like this:
class Foo(object, metaclass=MyMetaclass, kwarg1=value1):
...
It translates to this in the metaclass to use it:
class MyMetaclass(type):
def __new__(cls, clsname, bases, dct, kwargs1=default):
...
That's it. There is really nothing more about metaclasses.
The reason behind the complexity of the code using metaclasses is not because
of metaclasses, it's because you usually use metaclasses to do twisted stuff
relying on introspection, manipulating inheritance, vars such as __dict__
, etc.
Indeed, metaclasses are especially useful to do black magic, and therefore
complicated stuff. But by themselves, they are simple:
- intercept a class creation
- modify the class
- return the modified class
Since __metaclass__
can accept any callable, why would you use a class
since it's obviously more complicated?
There are several reasons to do so:
- The intention is clear. When you read
UpperAttrMetaclass(type)
, you know
what's going to follow
- You can use OOP. Metaclass can inherit from metaclass, override parent methods. Metaclasses can even use metaclasses.
- Subclasses of a class will be instances of its metaclass if you specified a metaclass-class, but not with a metaclass-function.
- You can structure your code better. You never use metaclasses for something as trivial as the above example. It's usually for something complicated. Having the ability to make several methods and group them in one class is very useful to make the code easier to read.
- You can hook on
__new__
, __init__
and __call__
. Which will allow you to do different stuff, Even if usually you can do it all in __new__
,
some people are just more comfortable using __init__
.
- These are called metaclasses, damn it! It must mean something!
Now the big question. Why would you use some obscure error-prone feature?
Well, usually you don't:
Metaclasses are deeper magic that
99% of users should never worry about it.
If you wonder whether you need them,
you don't (the people who actually
need them to know with certainty that
they need them and don't need an
explanation about why).
Python Guru Tim Peters
The main use case for a metaclass is creating an API. A typical example of this is the Django ORM. It allows you to define something like this:
class Person(models.Model):
name = models.CharField(max_length=30)
age = models.IntegerField()
But if you do this:
person = Person(name='bob', age='35')
print(person.age)
It won't return an IntegerField
object. It will return an int
, and can even take it directly from the database.
This is possible because models.Model
defines __metaclass__
and
it uses some magic that will turn the Person
you just defined with simple statements
into a complex hook to a database field.
Django makes something complex look simple by exposing a simple API
and using metaclasses, recreating code from this API to do the real job
behind the scenes.
The last word
First, you know that classes are objects that can create instances.
Well, in fact, classes are themselves instances. Of metaclasses.
>>> class Foo(object): pass
>>> id(Foo)
142630324
Everything is an object in Python, and they are all either instance of classes
or instances of metaclasses.
Except for type
.
type
is actually its own metaclass. This is not something you could
reproduce in pure Python, and is done by cheating a little bit at the implementation
level.
Secondly, metaclasses are complicated. You may not want to use them for
very simple class alterations. You can change classes by using two different techniques:
99% of the time you need class alteration, you are better off using these.
But 98% of the time, you don't need class alteration at all.
Best Answer
Use a Metaclass
I would recommend Method #2, but you're better off using a metaclass than a base class. Here is a sample implementation:
Or in Python3
If you want to run
__init__
every time the class is called, addto the
if
statement inSingleton.__call__
.A few words about metaclasses. A metaclass is the class of a class; that is, a class is an instance of its metaclass. You find the metaclass of an object in Python with
type(obj)
. Normal new-style classes are of typetype
.Logger
in the code above will be of typeclass 'your_module.Singleton'
, just as the (only) instance ofLogger
will be of typeclass 'your_module.Logger'
. When you call logger withLogger()
, Python first asks the metaclass ofLogger
,Singleton
, what to do, allowing instance creation to be pre-empted. This process is the same as Python asking a class what to do by calling__getattr__
when you reference one of it's attributes by doingmyclass.attribute
.A metaclass essentially decides what the definition of a class means and how to implement that definition. See for example http://code.activestate.com/recipes/498149/, which essentially recreates C-style
struct
s in Python using metaclasses. The thread What are some (concrete) use-cases for metaclasses? also provides some examples, they generally seem to be related to declarative programming, especially as used in ORMs.In this situation, if you use your Method #2, and a subclass defines a
__new__
method, it will be executed every time you callSubClassOfSingleton()
-- because it is responsible for calling the method that returns the stored instance. With a metaclass, it will only be called once, when the only instance is created. You want to customize what it means to call the class, which is decided by it's type.In general, it makes sense to use a metaclass to implement a singleton. A singleton is special because is created only once, and a metaclass is the way you customize the creation of a class. Using a metaclass gives you more control in case you need to customize the singleton class definitions in other ways.
Your singletons won't need multiple inheritance (because the metaclass is not a base class), but for subclasses of the created class that use multiple inheritance, you need to make sure the singleton class is the first / leftmost one with a metaclass that redefines
__call__
This is very unlikely to be an issue. The instance dict is not in the instance's namespace so it won't accidentally overwrite it.You will also hear that the singleton pattern violates the "Single Responsibility Principle" -- each class should do only one thing. That way you don't have to worry about messing up one thing the code does if you need to change another, because they are separate and encapsulated. The metaclass implementation passes this test. The metaclass is responsible for enforcing the pattern and the created class and subclasses need not be aware that they are singletons. Method #1 fails this test, as you noted with "MyClass itself is a a function, not a class, so you cannot call class methods from it."
Python 2 and 3 Compatible Version
Writing something that works in both Python2 and 3 requires using a slightly more complicated scheme. Since metaclasses are usually subclasses of type
type
, it's possible to use one to dynamically create an intermediary base class at run time with it as its metaclass and then use that as the baseclass of the publicSingleton
base class. It's harder to explain than to do, as illustrated next:An ironic aspect of this approach is that it's using subclassing to implement a metaclass. One possible advantage is that, unlike with a pure metaclass,
isinstance(inst, Singleton)
will returnTrue
.Corrections
On another topic, you've probably already noticed this, but the base class implementation in your original post is wrong.
_instances
needs to be referenced on the class, you need to usesuper()
or you're recursing, and__new__
is actually a static method that you have to pass the class to, not a class method, as the actual class hasn't been created yet when it is called. All of these things will be true for a metaclass implementation as well.Decorator Returning A Class
I originally was writing a comment but it was too long, so I'll add this here. Method #4 is better than the other decorator version, but it's more code than needed for a singleton, and it's not as clear what it does.
The main problems stem from the class being it's own base class. First, isn't it weird to have a class be a subclass of a nearly identical class with the same name that exists only in its
__class__
attribute? This also means that you can't define any methods that call the method of the same name on their base class withsuper()
because they will recurse. This means your class can't customize__new__
, and can't derive from any classes that need__init__
called on them.When to use the singleton pattern
Your use case is one of the better examples of wanting to use a singleton. You say in one of the comments "To me logging has always seemed a natural candidate for Singletons." You're absolutely right.
When people say singletons are bad, the most common reason is they are implicit shared state. While with global variables and top-level module imports are explicit shared state, other objects that are passed around are generally instantiated. This is a good point, with two exceptions.
The first, and one that gets mentioned in various places, is when the singletons are constant. Use of global constants, especially enums, is widely accepted, and considered sane because no matter what, none of the users can mess them up for any other user. This is equally true for a constant singleton.
The second exception, which get mentioned less, is the opposite -- when the singleton is only a data sink, not a data source (directly or indirectly). This is why loggers feel like a "natural" use for singletons. As the various users are not changing the loggers in ways other users will care about, there is not really shared state. This negates the primary argument against the singleton pattern, and makes them a reasonable choice because of their ease of use for the task.
Here is a quote from http://googletesting.blogspot.com/2008/08/root-cause-of-singletons.html: