ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Python News

Stackless Reincarnate

02/14/2002

Stackless was a controversial modification to Python, separating its execution stack from the C execution stack, the C-stack. With Stackless you could set up multiple execution chains, switch between them, change them, or restart them. Uncoupled from the C-stack, you could capture the control flow of your Python program and manipulate it any way you wanted to.

Manipulating the execution stack is the basic idea behind continuations, a building block for many other obscure but powerful programming structures. With Python as written, with it's own execution frame connected directly to the C stack, you just can't do it. Tied to the C stack, Python doesn't have full control over its execution chain. The C stack also puts limits on recursion. You can only recurse so deep before you run out of room on the stack.

As reported in A Stackless PEP, Stackless's developer, Christian Tismer hoped Stackless would be accepted as a part of Python's core. That's what was controversial. With Python's focus on simplicity, some thought such high power techniques didn't belong in Python. Tismer's changes also wouldn't work with Jython. They were deep magic, few understood them, and fewer wanted to have to maintain them. There may be something to that last argument. Maintaining Stackless was a challenge for Tismer, which is partly why he wanted to see it in the core. Stackless's supporters figured, despite the resistance, it would be accepted. Last year, in A Python Roadmap writer Cameron Laird predicted that Python would go Stackless in version 2.1. That didn't happen.

"I tried to convince the core developer group that they should re-design Python completely, to get rid of this unhealthy pair," writes Tismer. "The Python stack is a chain of linked frames, which could be easily convinced to switch each other, but the C stack stands there like a rock, mimicking every nesting of Python frames. That made switching impossible. Well, not today. I turned bad into good again and did the forbidden thing."

Over a piece of cake, a number of cups of coffee, and some deep thinking about the future, Tismer decided to abandon his old code and all hope of being accepted into the core. He decided instead to manipulate the C stack itself. In 1999, before the days of Stackless, Sam Rushing had created a coroutine package that directly switched C stacks. Python's developers agreed this would never be allowed in core Python. It would be too difficult to port, as every platform and every compiler handles the C-stack differently. To be a part of Python, manipulating the C-stack was forbidden. Tismer's incomprehensible Stackless code was the result of his trying to do the impossible, to make continuations work inside Python without completely rewriting Python and without ever touching the C-stack itself. But Stackless wasn't going to make it into the core either way. "Stackless is dead. Long live Stackless!"

In the new approach Tismer intercepts the eval_frame function, routing it to a wrapper function. On the first call to eval_frame, the current C-stack stop is saved by way of a bit of assembly code. This becomes the reference point for all later calls. The assembly code then copies to and from the stack. It ferets away parts of the stack when it grows too large or adjusts the stack by coping on a different microthread or coroutine. To the core Python code, this is all invisible. It just keeps pushing things on and popping them off the stack as always, never needing to know exactly how the stack is being manipulated behind its back.

Related Reading

Python Standard Library
By Fredrik Lundh

The changes to Python are small, mostly done with macros that won't need to change as Python or Stackless changes. The contents of those macros are kept in a single stackless.h file, which Tismer can easily maintain. The assembly code, however, is platform dependent and will need to be hand coded for every platform. Tismer has written assembly code for Microsoft Windows, and he says Linux on Intel should be easy, with just a few syntactic changes for gcc. The OS does not play as much of a role as the processor and compiler. Tismer does not plan to port to other platforms, though, unless someone loans him a machine and pays him for a week of his time.

Some have worried that the death of Stackless might mean the death of Python on the Palm. Because of Palm's memory contraints, it's crucial to limit the stack size. Stackless has been the foundation of Python ports so far. I asked Tismer if the new approach would work on the Palm. "I'd say even better," he replied. "I tested Stackless with reducing its overall stack needs below 2K of stack. The new technique turns out to be great since it doesn't switch to another hardware stack. It just grows and shrinks the existing stack, like normal C functions always do." He says a Palm port of Stackless is in the works.

For now, Tismer has turned to the more interesting work of writing stacklessmodule.c, which will define basic operations for coroutines and microthreads. Since Python now has generators, he won't reimplement them in Stackless, and for now, he won't be supporting first class continuations.

Despite the lack of broad platform support, the reincarnation of Stackless Python is mostly good. It's simpler and more powerful. The code that does the work need be written only once for each platform, and it doesn't need to change with each new version of Python. And there is no longer any restriction on using C extensions with Stackless. If you need microthreads or coroutines and your platform is supported, it's a beautiful new day.

Stephen Figgins administrates Linux servers for Sunflower Broadband, a cable company.


Read more Python News columns.

Return to the Python DevCenter.

Copyright © 2009 O'Reilly Media, Inc.