Simple two way Paging Solution for AppEngine

One of the challenges on AppEngine is to get get paging right. Here is what I did. It is supposed to be generic and easy to implement, but it will not give you page numbering.

Googles AppEngine team suggest using a unique property to sort on. Using that technique you will need a unique property on your Model, I will use sort_key.

For example:
Choose sort_key to be a composite key from the pub_date and slug of a Post entity to look like YYYYMMDDHHMMSS:slug (e.g. 20090119120000:simple-paging)

The idea is to grab N+1 entities from the datastore where N is the number of entities supposed to be on one page and use the existance of the last entity as indicator for a next page.

But how do we know if a previous page exists. If we get a bookmark we have been referred from some page (or bookmark is forged – which is of no interest to us).

:::python
def pager(qs, qs_asc, qs_desc, per_page=10, bookmark=None):
    """compiles a list of entities and signals if a next
       or prev selection exist for the given query"""

    # macro
    def fetcher(qs):
        """fetches ``per_page+1`` entities,
           return ``per_page`` entities and
           a flag indicating if more entities exist"""
        more = False
        e = qs.fetch(per_page+1)
        if len(e) == per_page+1:
            more = True
            e = e[:per_page]
        return (e, more)

    # At initiation the assumtion is that we can display all
    # entities returned from the queryset ``qs``.

    prev = None
    next = None

    # based on the ``bookmark`` we fetch the entities from
    # the datastore. If the ``bookmark`` is not None we
    # assume that a page in the opposite paging direction
    # exists.

    if bookmark is None:
        entities, next = fetcher(qs)
    else:
        if not bookmark.startswith('-'):
            prev = True
            entities, next = fetcher(qs_asc(bookmark))
        else:
            next = True
            entities, prev = fetcher(qs_desc(bookmark[1:]))
            entities.reverse()
    return (prev, entities, next)

To use the pager we need to have a regular queryset and two functions that take the bookmark as parameter and return a queryset: an ascending sorted queryset and a decending sorted queryset.

Assuming the bookmark from the querystring is stored in bookmark we can use to following code to compute the entities and prev,next flags.

:::python
prev, entites, next = pager(Model.all().order('sort_key'),
      lambda bm: Post.all().order('sort_key').filter('sort_key <', bm),
      lambda bm: Post.all().order('-sort_key').filter('sort_key >', bm),
      bookmark=bookmark)

To complete this post I’ll show the code for a Jinja template.

:::html
<ul>
  {% for entity in entities %}
  <li>{{ entity }}</li>
  {% endfor %}
  {% if prev %}
  {% set f = entities|first %}
  <a href="?bookmark=-{{ f.sort_key }}">&larr; previous</a>
  {% endif %}
  {% if next %}
  {% set l = entities|last %}
  <a href="?bookmark={{ l.sort_key }}">next &rarr;</a>
  {% endif %}
</ul>

Conclusion

The above code shows how to do simple paging on appengine without using the method described by google. If one really needs page numbering, this is bound to become quite a bit harder on the datastore. If you simply need the total count of the entities in the datastore you could implement a sharded as it has been described in the linked video quite nicely.

Feel free to use this code as you like.