Usage of iterator() on django queryset

djangodjango-querysetiterator

I came across some strange behaviour recently, and need to check my understanding.

I'm using a simple filter in the model and then iterating over the results.

e.g.

allbooks = Book.objects.filter(author='A.A. Milne')

for book in allbooks:
   do_something(book)

oddly, it was returning only a partial list of books.

However, when using the same code and using iterator(), this seems to work well.

i.e.

for book in allbooks.iterator():
    do_something(book)

Any idea why?

p.s. I did look through the Django documentation, but can't see how the queryset would be cached already anywhere else…

iterator()
Evaluates the QuerySet (by performing the query) and returns an iterator over the results. A QuerySet typically caches its results internally so that repeated evaluations do not result in additional queries; iterator() will instead read results directly, without doing any caching at the QuerySet level. For a QuerySet which returns a large number of objects, this often results in better performance and a significant reduction in memory

Note that using iterator() on a QuerySet which has already been evaluated will force it to evaluate again, repeating the query.

Best Answer

oddly, it was returning only a partial list of books.

That's not how the queryset must work. Iterating over queryset should give you every record returned by your database. Debug your code. You'll find the error, otherwise debug it again.

It's easy to check in the REPL. Run manage.py shell:

from app.models import Model
for o in Model.objects.filter(fieldname="foo"): print o

#Let's see DB query
from django.db import connection
print(connection.queries)

Related Solutions

Python – Build a basic Python iterator

Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and __next__().

The __iter__ returns the iterator object and is implicitly called at the start of loops.
The __next__() method returns the next value and is implicitly called at each loop increment. This method raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.

Here's a simple example of a counter:

class Counter:
    def __init__(self, low, high):
        self.current = low - 1
        self.high = high

    def __iter__(self):
        return self

    def __next__(self): # Python 2: def next(self)
        self.current += 1
        if self.current < self.high:
            return self.current
        raise StopIteration


for c in Counter(3, 9):
    print(c)

This will print:

This is easier to write using a generator, as covered in a previous answer:

def counter(low, high):
    current = low
    while current < high:
        yield current
        current += 1

for c in counter(3, 9):
    print(c)

The printed output will be the same. Under the hood, the generator object supports the iterator protocol and does something roughly similar to the class Counter.

David Mertz's article, Iterators and Simple Generators, is a pretty good introduction.

How to combine two or more querysets in a Django view

Concatenating the querysets into a list is the simplest approach. If the database will be hit for all querysets anyway (e.g. because the result needs to be sorted), this won't add further cost.

from itertools import chain
result_list = list(chain(page_list, article_list, post_list))

Using itertools.chain is faster than looping each list and appending elements one by one, since itertools is implemented in C. It also consumes less memory than converting each queryset into a list before concatenating.

Now it's possible to sort the resulting list e.g. by date (as requested in hasen j's comment to another answer). The sorted() function conveniently accepts a generator and returns a list:

result_list = sorted(
    chain(page_list, article_list, post_list),
    key=lambda instance: instance.date_created)

If you're using Python 2.4 or later, you can use attrgetter instead of a lambda. I remember reading about it being faster, but I didn't see a noticeable speed difference for a million item list.

from operator import attrgetter
result_list = sorted(
    chain(page_list, article_list, post_list),
    key=attrgetter('date_created'))

Best Answer

Related Solutions

Python – Build a basic Python iterator

How to combine two or more querysets in a Django view

Related Topic