MI5T proudly presents
last update: 02-Mar-04
A Python Implementation That Does Not Use The C Stack
© 1999-2004 Christian Tismer, Mission Impossible 5oftware Team
|Sponsorship <- starting a Stackless Python Alliance!||How you can help to make Stackless succeed|
|Downloads <- uploaded 02-Mar-04||Here you can find the latest available downloads|
|CVS access||Get the recent CVS files|
|Mailing Lists||Subscribe to the Stackless discussion or checkins list|
02-Mar-04: Bugfix Release While destructing tasklets, it can be necessary to create a new main tasklet just for the destruction. This did not work when an exception was pending. Added support for that to 2.2 and 2.3.
01-Mar-04: Bugfix Release A while ago, we made tasklets garbage collected objects. By that, we introduced a bug, which acts like a time bomb. During garbage collection, the current tasklet is visited, when it has no valid frame in its frame field. A simple fix made sure that a tasklets frame is either NULL or valid. This patch was applied to both 2.3 and the (still) maintained 2.2 version.
Many thanks to Seung Chan Lim for reporting this bug.
|24-Feb-04: Stackless Sprint in Berlin A sprint on Stackless Python will be done In Berlin, March 10-14. We are planning to work on auto-scheduling, more softswitching support, documentation, examples, Zope, refactoring Stackless into configurable layers, and more. See the announcement.|
|24-Feb-04: Small corrections for Python 2.3.3 There were some pickling problems which caused IDLE to work no longer. This has been fixed. At the same time, an error in standard Python's cPickle showed up. Patch submitted.|
|15-Feb-04: Stackless 3.0 for Python 2.3.3 is ready After a longer search for some final bug wihich applied to both Stackless for Python 2.2 and 2.3, I am releasing a so far final version of Stackless 3.0. There are a couple fo enhancements planned, of course. Some of them will be the theme of the upcoming Sprint on Stackless Python in March 2004|
|29-Jan-04: Huge steps towards Python 2.3.3 What can I tell, things are just moving on very quickly. Bob is doing a very good job enhancing stuff and moving Stackless further towards Python 2.3.3. I'm very grateful for his tremendous efforts. A couple of people are actively discussing Stackless' future, interface changes and design goals on the Stackless list. People are sending money to support Stackless. This is great!|
25-Jan-04: Again towards Python 2.3.3
Bob Ippolito wrote the pickling code for the new iterator types in Python 2.3, today. Many thanks!
I still have some problems with tasklet finalization which I don't understand These apply to both 2.2 and 2.3 . I had a working solution for 2.2, which I changed thoughtlessly, but since I also don't understand why that worked different, I have to carry on :-)
19-Jan-04: Small fixes, on the way towards Python 2.3.3
Some problems came up with the Mac OSX port. Bob Ippolito adjusted the registers to be saved during a stack switch, and it works fine, now.
I spent the last weeks to develop a port to Python 2.3.3, which is almost ready now. The cvs tree is prepared. WHen the new version is done, I just have to change a symlink. People who would like to have access to all versions might drop me a note.
08-Jan-04: Small fixes, heading towards CSP
Samit Patel found a finalization bug that dumped core when channels were not empty on destruction. Bob Ippolito observed some incompatibilities on Mac OS X, and pointed me to some scripts which needed to be adjusted to the new structure. Tom Locke is building CSP support for Stackless. While doing so, he stumbled over related problems with tasklet finalization. Repaired these and actualized the binaries.
06-Jan-04: A Huge Change in the Code Base
For the last two+ weeks, I was hacking on the internals
of stackless, in order to make things cleaner, smaller,
more natural and more complete.
As a major enhancement, the system now keeps track of all tasklets which are using a non-trivial cstate, ensuring that all these tasklets will receive a kill() signal if they do not finish normally before thread termination.
""" This was by far the largest check-in since months! f_tstate variables are removed from frames. cstack objects are removed from frames. All cstate is now kept in the tasklets. Tasklets with serious cstate are now killed automatically on thread deallocation. Tasklet soft-switching is now secured against repeated entry from "outside": A version variable is tracked, which makes sure that "main" is always left with the most recent version of initial_stub. Hey, this was two weeks of fulltime work! """
Please, give this code a heavy load of testing! With a few changes, this should be the code base for porting Stackless to Python 2.3.3. Especially, I'm planning to drop the version variable and use a reference to initial_stub, instead.
18-Dec-03: Almost Last Update of this Website The transfer of this website is still in the works. I'm gradually moving things over. There was a lot of changes and enhancements to Stackless in the last months. I will try to extract a web page from the CVS logs.
One first thing you might want to try is a Demo of Stackless Zope. This is nothing fancy, the point is the completely different paradigm of web programming.
|02-Jul-03: Reorganizing the Website This website is going to be re-organized completely. Part of it will be moved into a Wiki, which will show up in a few days.|
|27-Jun-03: Stackless again at EuroPython After another long period of silence, but being very vary busy to get Stackless 3.0 ready, I had a 90 minutes presentation at EuroPython 2003. The graphical highlight of the talk was my presentation of the EVE online game, which is now sold since a couple of weeks. Unfortunately there were a few hardware problems with my desktop, and I need to buy a better laptop for such opportunities. The software highlight was the new outstanding feature of Stackless 3.0: Thread Pickling is supported from now on and for all times.|
|Again a long break But don't worry. This time, I am preparing a huge overhaul of all of Stackless. There is a technology merge under the way which is very hard to do: Combine the elegance and small footprint of Stackless 1.0 with the power of Stackless 2.0. This is again a complete re-implementation of Stackless, re-introducing ideas from the first implementation, but without continuations. Tasklets and channels will stay the basic building blocks of Stackless. There is almost no change in the user interface: Stackless 3.0 is just better, smaller, faster and more versatile, but software should run almost unchanged.|
|15-Oct-02: Ported to Python 2.2.2 One
day after the release of Python 2.2.2, Stackless has been ported to it.
I hope to be quicker next time. There were a couple of changes to typeobject.c
which were backported from 2.3, and I had to make many adjustments (basically
thowing stuff away). My changes to the type implementation are now very
small and will be re-submitted as a proposed patch.
The PC binaries have been upgraded to 2.2.2 and include _tkinter as well.
|10-Oct-02: Scheduler Object introduced In order to gather more flexibility for deriving your own scheduler, I provide a scheduler object, which can be overridden by the user. The primary reason for this is the very different scheduler implementation of IronPort, which needs to be supported. Right now, the object is almost empty, but it will be filled with methods as the porting progresses.|
|07-Oct-02: Support for PPC Unix and S390 Thanks to Gustavo Niemeyer for his additional platform support. I'm thinking to start a porting contest, in order to get all the rest of platforms prepared for Stackless :-)|
|15-Sep-02: Support for Sparc and Cygwin
On July, 16, Gerd Woetzel sent
me a patch that supports Sun Sparc. It took this long to get back to it,
due to all the other changes. Many thanks, Gerd! Stackless now supports:
Windows, Mac OS X, Linux-X86, Cygwin, SunOS Sparc with gcc.
Sorry, multiple-channel waits had to wait for this to get ready. Will come rather soon now...
|15-Sep-02: Stackless as External The Stackless core code and the stackless module application are now split apart. Gettig the Linux build to work again took me quite some days, but now everything is in good shape. Stackless with tasklets and channels is now an external module. The PC binaries now include stackless.pyd as extension module.|
|13-Sep-02: Stackless with Flextype In order to be able to have an efficient, clean way for inheritance from my internal types, I invented flextype.c, which supports virtual method tables in type objects. This leads to very efficient virtual methods with no time penalty when a method is not overridden. Virtual method resolution is done when a class is instantiated. The source tree is now split into many small files, and the project is reorganized quite much.|
|02-Sep-02: New Beta with raise_exception Quite a lot has happened. Tasklets support now raise_exception, whether they are waiting on a channel or not. A lot of changes have been made. The whole schedule() implementation is re-worked, and the tasklet flag handling as well. Please try it out. The PC binaries have been updated. A complete rework of the channel protocol, together with multiple channel waits, will appear in a few days.|
|25-Aug-02: Stackless Beta with PC Binaries Stackless has been improved reasonably in the last weeks. Please get the latest binaries and try it out.|
|01-Jul-02: Stackless Ported to Power PC Thanks to Just van Rossum, who gave me an account on his PPC machine, I was able to finalize the PPC port of Stackless. It seems to work great. Please give it some testing and send me feedback. Note that I have no idea about PowerPC assembly. I just used what I learned from ICON's swtich implementation, and from Armin Rigo's patch for the x86-Unix version.|
|26-Jun-02: Stackless At EuroPython After hacking on the pickler stuff until the morning of this first EuroPython day, I was quite sure that this will work. I used the last three hours before my talk to throw some 20 slides together, and was able to show first results during the talk. It worked out great. The conference was also a big success, with over 250 attendees, and many interesting talks, a lot of Python VIPs...|
|23-Jun-02: Thread Pickling? I got an email from Arman Bostani, concerning the thread pickler. Trying to answer his email, I happened to fall into hacking mode and begun to implement what is necessary to pickle threads which have ctsack objects attached to frames. I ended up with doing an analysis of compiler map files, source/code/assembly listing, instead of preparing my slides for EuroPython.|
|21-Jun-02: TCL problem again solved so far Jeff corrected some of his own bugs. It still crashes when one thread uses slicing, but it works with a single thread. Fortunately, the Tcl developers informed me that they are planning to remove the stack references completely in Tcl 2.4, so we decided to finalize the solution when this version is available.|
|17-Jun-02: TCL problems again Jeff Senn reported crashes with Tcl and multiple (real) threads, when running the interpreter loop in a seperate thread. I could not really track it down. Jeff also used a patched Stackless with his own scheduler built in, so there were quite a few new variables in the play.|
|26-May-02: TCL problem solved so far After lots of debugging and hair-pulling, it seems to be solved. The simple idea is to block the stack slicing feature in the context of tkinter calls. Waiting for user feedback, if this solution is complete.|
|22-May-02: TCL becoming a problem During my stay at IronPort, I happend to step on some problems with tkinter. Tcl/Tk does keep global data on the C stack and expects it to be accessible during recursions into the Python/Tcl interpreter. This is really bad. When being "stackless", stack slices are removed at every n (=8 or so) recursion levels.|
18-May-02: Limbo Dancing works fine! Yet some more thinking for weeks, and a few days of implementation. Now, the first implementation of channels is available. It isn't complete, but works just fine. The stackless module now supports to create channels by function channel(). A channel has the methods send() and receive(). Try it yourself! You might either want to build your own Stackless, checking out the module stackless from :pserver:firstname.lastname@example.org:/home/cvs , or, for Windows users, just take the pre-built binary, extract it into some directory, and start Python or PythonWin from there. I will for sure build an installer again, after I have got some feedback.
This implementation tries to follow the concepts of the OCCAM/ALEF/LIMBO languages by supporting the channel concept as the central communication/blocking/control feature. Tasks do not communicate directly, but through channels.
You might have guessed it: Stackless will try to implement/support Hoare's CSP, finally. But before, I have to understand more of it. This will happen in the next few weeks.
This release is an alpha release. There are most probably a lot of omissions. Some are obvious:
- There isn't yet a scheduler, like there was one for the uthread module. This is trivial to implement and will be added in almost no time.
- There isn't yet an ALT/PRI ALT construct like in OCCAM/ALEF/LIMBO. This is most sophisticated and very hard to implement correctly! I will try to find out how to do this the best way, during my stay at IronPort, from May 18th to May 29. If somebody from the US would like to contact me by phone, please use the opportunity and call me via IronPort. I am employed by this company.
- You shouldn't use channels from the interpreter prompt, but in scripts, only. The machinery doesn't check for changed interactive sessions yet and will most likely crash, soon. Despite of that, it seems to work nicely with PythonWin and extensions.
|April 02: Stackless goes Limbo After the thinking pause, I am now busy implementing the new design and writing some paper about it. I decided to adopt the concepts of Inferno's programming language Limbo. You might want to read about their concepts in Brian W. Kernighan's A Descent into Limbo. To give a quick idea, just this: Tasklets are scheduled preemptively. All data and control logic between tasklets is performed by channels. Tasklet synchronisation and blocking is performed on reads and writes via a channel.|
|16-Mar-02: Hired for Stackless Stackless got the best sponsorship ever: I was hired part-time by IronPort, a company near San Francisco with a great messaging product that is based upon Stackless! It is a special delight to be in the same company with Sam Rushing. I'm working for them in Germany, coming over from time to time. Really great!|
|06-Mar-02: Thread Pickler It is incredible! Arman Bostani from TwinSun told me that they have implemented a thread pickler! Their only reason to use the old stackless was its freedom from the C stack, and they implemented in fact such an animal. They are going to donate it to the Open Source. The big drawback: My new version is no longer C stack independant. I promised them to do everything necessary to make this work with the new Stackless, too.|
|March 02: I need a pause for thinking After some discussions on py-dev and a lot of private emails, I became quite undecided how to continue the system design. I wasn't able to continue coding until I was completely sure what to code.|
|24-Feb-02: Further Simplification There is now only one single stack switching function left, which contains just four lines of assembly code. Recursion elimination has been mapped onto two internal transfer calls.|
|22-Feb-02: 1,000,000 Switches in 0.6 seconds I was stunned when I measured switching speed on an AMD Athlon XP 1500+ machine. A python script switched tasklets 1,000,000 times. The usual Python call overhead was included. I measured this without switching and got about 0.3 seconds. That means: Context switching by an automated, internal scheduler without the interpreter overhead comes at the same cost as a Python function call.|
|22-Feb-02: 100,000 Tasklets in 80 MB Right now I'm designing tasklets, which are the interface objects for coroutines, microthreads, however you name them. A tasklet wraps a chain of Python frames and a piece of the C stack -- the piece that will be needed to restart the tasklet. Today, I tested a Win32 release build and created 100,000 tasklets. The application's size grew to 80 MB. That is: A tasklet doesn't cost much more than 800 bytes.|
14-Feb-02: O'Reilly Network: Stackless Reincarnate Please read what a nice article Steven Figgins created again from an interview.
21-Jan-02: Stackless is in CVS. I created a CVS repository on tismer.com and checked a full Python 2.2 source tree in, with the vendor tag python_2_2. The module name is, guess, "stackless". Here the login data:
Additionally, there is a mailing list which fills your inbox with all my patches: http://www.stackless.com/mailman/listinfo/stackless-checkins/
19-Jan-02 Stackless Python is DEAD! Long live Stackless Python
Here the message that I sent out, declaring the end and the beginning of a new era. Stackless now breaks its primary rule, never to touch the C stack. Without this rule, we get the best implementation ever, with unlimited switching. Stackless will not make it into the core after this change, but who cares. The maintenance work is from now on a cake walk.
|A long, long break. Is Stackless dead? It was, nearly. An attempt to port to 2.1 didn't work out. I got bored to have to continously try to catch up with the rasant development of Python, for a Stackless that would probably never get complete, and that would never make it into the Python distribution, causing me to maintain this incomprehensible code for life.|
14-May-01: Sourcecode Debug update: Some minor changes to the project files, ceval.c and continuationmodule.c, in order to support a DEBUG build. There is also a "pyexe20" project supplied for building the debug Python shell. On behalf of Aaron Brancotti from First Thought, Italy.
04-May-01: My Apologies: This bugfix is ready since two weeks and I didn't upload it. Sorry, this shall not happen again.
Stackless Python 2.1 will be available, soon. I already have an internal version, but I want to to some harder changes to the kernel before releasing the code.
|18-Apr-01: Probably the last bugfix for 2.0: Bernd Rinn reported of a memory leak in March. It took weeks to nail that bug down. For heaven's sake, I wanted to get the 2.1 port done, but fixing this bug was of higher priority.|
|06-Mar-01: The Award is there: Thanks to all who voted for me. I got the trophy on the reception of SPAM9. Pictures will be added, soon.|
23-Feb-01: Yet another bugfix: Changjune Kim reported a funny bug in interactive mode. This is solved, the bug was due to an unitialized error variable. People who don't like to download the source again need to do just this: Insert "err = 0;" after line 1477 of ceval.c:
22-Feb-01: Nomination for the Programmers' Choice for Python: ActiveState
|05-Feb-01: Stackless Python RPM Files available: Bernd Rinn has prepared Linux RPM files which can be downloaded here. Many Thanks, Bernd!|
04-Feb-01: Another minor Bug-fix for Stackless 2.0: print >> sys.stout, 1, 2, 3 now works as expected.
|02-Feb-01: Stackless Python and Korea: The
Korean Python Users Group held its second seminar on Python. Changjune
Kim held a wonderful speech about Stackless Python. I was invited to
join and gave some extra information.
The Korean's gave me a very warm welcome. I will join the next meeting and
I'm trying to learn Hangul now.
Here my first attempt to write "Stackless Python": (corrected by Changjune)
|15-Jan-01: Minor Bug-fix for Stackless 2.0: Wolfgang Lipp reported that uthreads were now executed sequentially. Please get the new version from the download section.|
|12-Jan-01: Tiny source-code update: Henk Jansen reported that again a "static" function crept in that wasn't static :-)|
|10-Jan-01: Stackless Python 2.0 source-code released. It should be easy to build under Windows. For Unix, some hand work will be needed. Thanks in advance to those who will be supporting this.|
|07-Jan-01: Stackless Python 2.0 released. In addition to the port to Python 2.0 and some minor enhancements for 1.5.2+ as well, a couple of example scripts are now included. Most important may be the current microthread implementation from Will Ware, Just van Rossum and Mike Fletcher. The microthreads will be developed further during the next half year to run at full speed as C code. Until then, this continuation based implementation will stay as the state of the art. Most of this work was done at my parent's home, therefore it is the Kiel Edition.|
|15-Nov-00: Stackless Python gets Sponsorship! CCPGames, Inc., the creators of the EVE game, are supporting Stackless by some funding of my work. This is great, since it will allow me to put much more time into Stackless Python! Thanks to Hilmar Veigar Petursson of CCPGames.|
|06-Nov-00: "What is Stackless": A Stackless Tutorial by Gordon McMillan|
|17-Oct-00: "Python Roadmap": O'Reilly Article by Cameron Laird|
|11-Oct-00: "Programming Stackless Python": O'Reilly Article by Cameron Laird|
|10-Oct-00: Stackless Python 1.2 and Continuations 0.9 released. Major changes: Stackless now uses its own reference counting which results in less trouble when large tracebacks are generated. Exception handling has been improved, in order to avoid leaks: On every continuation jump, exceptions are cleared.|
|05-Oct-00: The Stackless Python Mailing List is now available. Please come and join|
|04-Oct-00: "Introduction to Stackless Python": O'Reilly Article by Cameron Laird|
|01-Oct-00: "Charming Python": Interview by David Mertz|
|23-Aug-00: "A Stackless PEP": O'Reilly Article by Stephen Figgins|
|29-Mar-00: Version 1.1.1 minor bug-fix release (eval/exec)|
|28-Mar-00: Microthreads are there. Much more speed and major design improvement.|
|31-Jan-00: Version 1.02 passes all tests. Added feature: safe deletion of deeply nested structures. PowerPoint sheets from IPC8 available.|
There a quite a lot people currently helping me. A couple of new modules are under way, and there are some contributions on examples and documentation. They will appear here, soon.
Stackless Python was developed and optimized under Win98 and Visual Studio 6.0. Supported directly by Chris
Successfully compiled by Jeff Collins (1.5.2+)
To come from Jeff, soon? Currently, there exists a Palm port of standard 1.5.2, and Stackless is planned.
Stackless Python and the continuation module have the same license policy as Standard Python for all users. It is free to use for everybody. Just he optimizations of the interpreter are restricted to Stackless Python. In order to apply them, Standard Python must become stackless.
Developing Stackless into a rock solid basis for applications is a lot of hard work and it needs much of my time. You can help me by sending money. The more money I get for working on Stackless, the more I can reduce work on other projects, since I have to feed a big family.
A number of people said that they would like to sponsor me with a little amount if it were easy to do. Therefore, I signed up with PayPal, where you can send your contribution quite easily.
Please enter a note whether you would like to be listed on the sponsorship page. Thanks in advance!
Stackless Python needs support by the companies that use it. The SPA should help to provide this, by supporting me by money and human resources. At the moment, almost everything happens in spare time, which is not enough for a big project like this. Also there need to be more people who can work on Stackless. Companies cannot depend on a single person like me. Please contact me if your company would like to help here.
“””There have been a few attempts to implement generators and coroutines in Python. These were either of low performance, since they were implemented using threads, or limited by Python's recursive interpreter layout, which prevented switching between frames liberally. Changing Python to become non-recursive was considered a major, difficult task. Actually it was true, and I had to change truth before I could continue. ”””
For further information, have a look into my paper for the 8th International Python Congress:
Article in HTML format, as PDF file, Word 2000 document, or everything in a Zip compressed file. Some people also might find some older text useful. I’d like to hear which style is better and what you would like to see in the final documentation.
Developers might also be interested in Guido’s thoughts about continuations from the Python Developers list. Also have a look at a thread based Coroutine implementation from Tim Peters that can be now be implemented with Stackless Python.
PowerPoint slides (315k) from my IPC8 presentation are available as well.
PowerPoint slides (243k) from the Korea presentation (not all were used). These mention some applications, briefly explain the differences between standard and Stackless Python, and give some thoughts about the pros and cons of continuations.
All documentation above is fairly out of date and will be augmented soon, hopefully.
PowerPoint slides (57k) from the IPC9 devday discussion. The results of that discussion partially caused the last longing break in development, which raised the question on the Python list, whether Stackless was dead.
PowerPoint slides (809k) (PDF version 1272k) from the EuroPython presentation. This talk was quite successful and gave much more positive feedback than I had hoped for. I used a few of the IPC9 slides (see above) to show the problems which had caused trouble, and then introduced and compared the completely new approach.
I'm beginning to collect applications which make use of Stackless Python. One exciting example is
EVE, a massively-multiplayer online game (MMPOG), whose kernel will be based upon Stackless Python. For further information see the FAQ 8.6.
The IronPort A50 Intelligent Messaging Gateway simplifies and reduces the cost of outbound email delivery. The system is implemented upon Stackless Python.
|python2.2.dll for Win32 release
md5 checksum script for the zip
These binary snapshots are built from the current CVS tree. This is the frozen final status for Python 2.2. This version is going to be deprecated in a small while.
|python2.2.dll for Win32 debug
md5 checksum script for the zip
|python2.3.dll for Win32 release
md5 checksum script for the zip
|These binary snapshots are built from the current CVS tree. This is the frozen final status for Python 2.3.|
|python2.3.dll for Win32 debug
md5 checksum script for the zip
To retrieve the most current CVS version of Stackless, use the following commands (Linux, bash):
CVSROOT=:pserver:email@example.com:/home/cvs export CVSROOT cvs login <-- empty password here cvs co stackless
Note: stackless is just a link to the current version. You can also use
cvs co slpdev/src/2.2/src cvs co slpdev/src/2.3/src
to obtain versions, explicitly.
|Stackless||The Stackless Python Mailing List|
|Stackless-checkins||Notifications of Stackless CVS checkins|
I wish to thank Michael Hudson,
Sam Rushing and Will Ware for their help on the Linux build and for their valuable
input. Just van Rossum and Mike Fletcher for working with Will on the Microthreads.
Stephen R. Figgins and David Mertz for their articles, and Cameron Laird for
preparing even an article series. Gordon McMillan for his nice introduction,
and for not yet writing the PEPs (you know what I mean :), CCPGames
for their ongoing sponsorship, IronPort
for hiring me part-time, Bernd Rinn for building RPM files, all the many others
which I forgot to add to this page....
... and Guido van Rossum for still not sending the Spanish Inquisition.
Christian Tismer 23-Jan-00– last update: 02-Mar-04