Realizing Jython 2.5

June 20, 2008 at 09:43 PM | categories: Jython, 2.5.x | View Comments

Jython 2.5 is really, finally, unbelievably coming together. This is the next release of Jython, after last summer's 2.2. In a nutshell, we have completed all new language features using an Antlr parser, except for absolute imports. All bytecode generation work, now using an ASM backend, is done. Of course, there are many outstanding bugs. And Python is not just a core language; we need to support fully the fact that "batteries are included". But let's look at where we are. Through the prism of what's new in 2.3, 2.4, and 2.5, here's what working:

  • 2.3: sets (PEP 218), generators (255), source code encoding (263), universal newline (278), enumerate (279), logging (282), Boolean (285), distutils (301), new import hooks (302), pickle enhancements (307), extended slices, datetimes, optparse. Still to go: csv, removing a dictionary in builtin that ensures that interned strings don't get in GC'ed (pre-2.3 behavior!, it helps to read what's new). Also various string, Unicode, and regex changes are mostly done in a separate utf16 branch that I'm currently in the midst of merging against trunk.
  • 2.4: unifying long integers (237), generator expressions (289), string.Template (292, but also needs new utf16 work), decorators (318), reverse iteration (322), subprocess module (324), multi-line imports (328), removal of OverflowWarning, min & max with keyword support, sorted. But we still need partial import with sys.modules, and I'm sure some more stuff I forgot. Decimal and -m support are working in student branches, we just need to incorporate.
  • 2.5: conditional expressions (308), partial functional (309, but we're cheating with a pure-Python version), distutils metadata (314), unified try/except/finally (341), coroutines and other generator functionality (342), with-statement, including contextlib (343), any, all. But we haven't done the exceptions remapping to new-style classes, absolute and relative imports, or all of the context manager support, such as in file. ctypes was a proposed Google Summer of Code project, but apparently PyPy has some work that's 95% the way there; we will talk with them at EuroPython. We need to look into what is necessary to make ElementTree work. sqlite3 depends on ctypes. As I was writing this, I tried out wsgiref; it works and I just committed it to the asm branch. (At some point, we will repoint everything like this to CPythonLib, but for now we are mixing it up as we go. Bear with us!)

Even quit() and exit() now work; I don't know when these oh-so-major features were added. We even now support large string constants. And of course, who can forget our support for the GIL (global interpreter lock) in Jython, something that Tobias Ivarsson, my Google Summer of Code student who is now working on an advanced compiler, added to __future__ as an Easter egg:

>>> from __future__ import GIL
Traceback (most recent call last):
  (no code object) at line 0
  File "<stdin>", line 0
SyntaxError: Never going to happen!

I would imagine that's definitive, we go against Java's native threads and compile to Java bytecode. It would be hard to have a GIL, even if we wanted one.

However, we are just turning the corner. The The Antlr parser in the asm branch currently does not support partial parses, and this breaks not only interactive sessions but doctests. Until this is solved - and Frank Wierzbicki is working like mad on this - we can't merge this branch onto trunk. But that should happen very soon.

With few exceptions, we simply go against the standard Python unit tests. Straightforward, cunning, or devious, we have labored against these unit tests. And in others, we have used Python as our foil: we support the same 2.5 AST parse tree, and we know this by comparing our parses with CPython's for all of the standard library - including those unit tests.

There's a lot more going on. I can't say enough about the work done by Charlie Groves, Philip Jenvey, Alan Kennedy, Nicholas Riley, and others to make this happen. Leo Soto, my other GSoC student, is making amazing progress on supporting Django on Jython, while finding and fixing bugs in Jython itself. Supporting Django forces us to find those gaps in compatibility. Similar efforts are going on with Pylons, TurboGears 2 (Ariane Paola, GSoC), and Zope (Georgy Berdyshev, GSoC). I'm also working on greenlet/Stackless support and involved in a collaboration with Jeremy Siek and Joe Angell at the University of Colorado to add gradual typing] (yes types! but only when you want to) to Jython. We have a T2000 contributed by Sun to let us see how much concurrency - in this case 32 hardware threads, 64 GB of memory - Jython can take advantage of. And so on.

Back to work!

Updates - 2008-06-24: we have support for new-style exceptions, the parser is now usable (but there are a couple of bugs left there), and Unicode support has been updated to UTF-16. See this posting, Flipping the 2.5 Bit for Jython.

blog comments powered by Disqus