Stackless Python
Forgot your password?
A presentation given by Richard Tew at the Python Atlanta meetup group on the 20th March 2008.
You can download it here: Stackless-2008-03-20-PyAtl.zip (70 KB)
I decided to spend my lunch hour looking into a Stackless crash bug today (to do with threading and tasklet killing when the interpreter exits). But after not being able to reproduce it I ended up using some work time to port Stackless to the new release of 2.5, release candidate 2.
For those who wish to use it and will not be compiling it themselves from the source code in SVN, I have made available new Windows binaries.
-- Richard Tew
Stackless Python has finally caught up to the first release candidate of Python 2.5, and for those who wish to use it and will not be compiling it themselves from the source code in SVN, preliminary Windows binaries have been made available.
Grant Olson has posted to the Stackless mailing list about a tutorial he has written for Stackless Python.
Along with a general introduction to Stackless, it includes comparisons with thread-based approaches and PyGame examples.
The Stackless Python SVN repository has been moved from been moved from its temporary location at codespeak.net to its new location at svn.python.org.
Thanks for this to the PSF, Stephan Deibel and especially Martin v. Löwis who did most of the work.
Stackless Python has been ported to the recent release of Python 2.4.3 (final).
You can either obtain the source code from the SVN repository or download the precompiled windows binaries.
The sprint at PyCon 2006 was a success and the result is that Stackless Python is now available for Python 2.4.2.
You can either obtain the source code from the SVN repository or download the precompiled windows binaries.
There is only one known issue, related to pickling a certain type of iterator. And this should be rarely encountered.
The presentation that will be made on the use of Stackless Python in the development of EVE Online is available for download here.
The zip file (file size 550k) contains the presentation in both ppt and html formats.
(updated with spelling corrections, an additional page or two and some added facts)
You can now subscribe to an RSS feed to get notified about any news thats get posted here.
Stackless Sprint
There will be Stackless sprint at PyCon 2006.
The primary goal of the sprint is to port Stackless to the latest version of Python, which at this time is 2.4.2. This involves bringing the Stackless CVS up to date with all the changes made to Python since version 2.3.3 in order to maintain the history of stackless changes made.
Other ways you can contribute if you are interested are:
- Documenting Stackless / improving the web site.
- Making Stackless more approachable for new users.
- Writing examples.
- Writing tests.
- Improving the web site.
- Any other way you can think of that will benefit Stackless!
As Christian will also be attending the PyPy sprint, which is to be held in the same alloted sprint time (after the conference) his role will generally be to guide those that are there in doing the required work.
CCP Games Presentation
Kristján Jónsson from CCP Games will be doing a Stackless related talk. Here is the initial proposal he submitted (which he just let me know has been accepted):
Requested timeslot: 30 minutes
Suggested title: Stackless Python in EVE Online
Summary of proposed presentation:
The massively multiplayer online role-playing game Eve Online, developed by CCP games
in Reykjavík, Iceland, makes heavy use of Stackless Python to implement game features.
This presentation shows how Stackless Python has been used throughout the code to
create a seamless environment of cooperative multitasking in the game engine and how
Python code and C code interacts in a complex real-world environment.
Presentation Outline
- Introduction
- Eve Online
- Overview
- History
- Game architecture
- Stackless Python
- Tasklets, channels
- What does "stackless" mean?
- The current Stackless model.
- Stackless in Eve
- The main loop
- Example python code:
autoexec.py
- Extending into C
- Examples: How Stackless, Python and C interact:
Synchro
Netclient.dll
- Conclusion
Intended Audience: Advanced Users, CPython developers
The New Pickling Concept
Stackless pickling has got a complete re-write. There is not any single type patched any longer. Instead, all types have got a shadow type which is living in the stackless._wrap module.
The wrapper types create a subtype of the real type when invoked. On __setstate__, they set the real data (to avoid deep recursion), and they set the real type. Afterwards, just an innocent object of the target type exists, which is not changed at all.
This is just an intermediate step towards the final solution. I will now rewrite things, again, in a much more generic style. Types-to-be-pickled will just have to provide a __reduce__ function and a __new__ function. The framework will make sure that the __new__ actually takes place in the __setstate__ phase. With that concept, we will have no problems with recursive objects.
I think this is worth a lightning talk at Europython.
Lots of changes -- need more unittests
As you might have noticed by stackless-checkins, I have done a lot of changes in the dev branch. The soft-switch protocol has changed a lot, in order to be cleaner and to support a move towards thunks and greenlets. This was quite a bulk of work, and there is still a lot of things to do.
The pickling has become more compatible, since I don't rename types, any longer. Instead, they all end up in the builtins. I'm not sure if this was the best decision, but finally, it is somehow logical. I tried to circumvent this by involving copy_reg, but it didn't help much. I'm still patching the types to support pickling. I don't think this is too bad. One bad spot is the existance of the modict. I would like to remove it, if I can find a good alternative.
The problem is that module dicts need special treatment, and if I don't change their type, then pickle and cPickle will treat them like any dict, so I would have to patch cPickle and pickle.
Another news about cPickle. I added a way to overcome its recursive behavior, by simply using hard-switching. I think this kind of treatment is what it deserves. A better approach would be to rewrite cPickle in a non-recursive style, but that would need much more effort than the Stackless trick. Fortunately, unpickling is non-recursive, already. I recognized that late, after I had supplied a patch, already. :-) Well, shit happens.
Since I have unlimited pickling, now, I could drop all the changes to tracebacks and implement their pickling using simple recursion. I will probably simplify tasklet pickling the same way, soon, but this is less urgent, since tasklets are my own type.
I was very happy to see a new unittest from Stefan Drees, and I'd really appreciate to see some more, very soon.
There is also some lack in testing the Stackless API, which is meant to be used by extension writers. A special area is the case where Stackless internals are called without initializing Stackless. This should always work, but I need some decent tests for this, and this is only possible by an extension module. Maybe someone would like to give this a try?
cheers -- chris
There was a deeper bug in the Win32 task switching code, which was hidden for years. I did not explicitly touch the registers bx, si and di, although I thought so. Just "mentioning" is not enough, you have to change them. This is a bugfix relevant to all Windows users.
There are also some clean-ups and a fix for a possible bug in the cstack/tasklet relationship. Another one was that I forgot to add tasklet.run to the tasklet method table. This is relevant for C people, only.
New feature: The TaskletExit exception now inherits from SystemExit. This makes very much sense, since code that blocks all but SystemExit will now also let taskletExit get through.
Updates to the stable release
there were a few bugfixes in
- nonrecursive kill (bad condition killing current, thanks CCP)
- more fixes for power pc configuration (thanks Bob Ippolito)
- avoiding warnings (thanks Stefan Drees)
A major change was the introduction of a special implicit function parameter, to get rid of duplicated code for non-recursive versions of internal functions. By this, type objects are now again absolutely identical to Standard Python's, and so it should be.
Execfile and eval are stackless. Generators can be pickled when being run.
This is now considered the current stable release. dev and src are identical.
Next changes to the dev trunk
The latest CCP bugfix concerning tasklet.become() wiull now be reverted in the dev trunk, and the frame f_execute protocoll will be changed to take an explicit retval parameter.
Then I will also implement continuations, in a very limited, careful way, on a user request. There will be nothing but callcc, with almost no optimization, just let it work.
CCPgames informed me of a small problem with using a toplevel tasklet.become() call during an import. This showed me a bad consequence on some optimization that I did, months ago. The result was that a Py_UnwindToken was able to leak through slp_frame_dispatch. I was not aware of this condition.
Next time, I will undo this optimization, which doesn't work out, but increases the dependancy of a temporary value being stored in the thread state. I'll probably make tempval into an explicit parameter, very soon. This will not affect the API or anybody, unless (s)he is using the internal API. Patches are checked in, new binaries are uploaded.
Stackless has used some extra fields in the type objects, to support non-recursive calls. After dynamic classes evolved more and more, keeping track of such extra fields has become more and more difficult. In extent, I had to maintain quite a large number of function copies, which behaved slightly different. After all, this approach didn't give me all the results I expected.
Since yesterday, I have been working intensively on a new approach. It might be sligtly less efficient, but it seems to be very, very clean.
tp_call_nr and tp_iternext_nr have been removed from type objects. So far, type objects are unchanged, again.
Stackless instead now supports an implicit parameter for every function call. It would be great if we could make it into an explicit parametzer, but that would change the mplementation completely.
Information about the concrete implementation should probably go somewhere else. Just summarizing: Stackless passes all Python tests and all of its internal tests, the code base is reduced, again, and we have a couple of noew non-recursive calls:
- tasklet.raise_exception can run stackless,
- tasklet.kill can run stackless,
- generator.next can run stackless, like "for i in generator" could already, last time.
This version of stackless is not accessible via the stackless cvs. If you want to try it, you need to use
cvs co slpdev/src/2.3/dev
Will add a link to this to the public pages, soon.
as well, while it is running. Actually, I changed the whole Stackless
calling convention for non-recursive calls, and it paid off.
See the next message for details.
Stackless can now pickle a running generator. At the moment, this is only possible in a for_iter context. gen.next() will still be recursive, until I find out how to tell this to typeobject's wrappers...
Please try the new Windows binaries.
The cvs sprint repository was moved over to the src repository. The current sprint results are therefore accessible to all users. New binaries are uploaded as well.
The sprint trunk has been renamed to dev. After a number of changes which result in a stable state, again, the dev trunk will be copied over to the src trunk.
There is now a "Sprint" folder available on the site. in which the results of the sprint are gathered.
For now you can find there a (yet incomplete) list of sprint participants and their personal sprint summary. This includes pictures of the event. There will be more and updated content added in the future.
The sprint is over! One of the achievements is this new stackless web page. It is plone based, and we hope, it will be now easier for you and us to contribute how-to's, code examples and other wiki documentation. We also now have an issue tracker (you need an extra login), our old site you find here.
