Front Range Pythoneering

Trac on Jython - Genshi Support

noreply@blogger.com (Jim Baker) — Sun, 12 Jul 2009 00:18:00 +0000

(I was going to add this as a comment to NixDev Open Source Blog but the commenting system for that blog is currently broken. So here's my response, and maybe it will get me to start blogging here again too.)

There has been less interest in Genshi from Jython development as of late, probably because Mako is a very good alternative for Turbogears 2 and Pylons. But you need Genshi for Trac, so here's what you might do:

expat. Jython 2.5 has an implementation of expat (Lib/xml/parses/expat.py) that wraps SAX sufficiently that all of the unit tests for ElementTree pass. The only problem is that it's somewhat slow, since the wrapper is in Python. I would see that as a starting point to selective rewriting in Java, which is much easier to do in Jython than in CPython. Perhaps with just a little more emulation work it will also work with Genshi?
AST. Jython 2.5 implements the standard _ast and ast (the latter actually part of 2.6, but we needed it!) modules; we do not have any support for the older compiler module. I don't know the status of changeset 31 to support AST, but if it has been incorporated, or can be, we can work with this part then.
CPython bytecode. Lastly Jython 2.5 implements a CPython bytecode VM. This has been tested by compiling the entire regression test suite into CPython byte code (with CPython, we don't yet have a compiler for this path!), and except for some cases around code introspection and minor differences in floating point representation (which is an artifact of using CPython for the compilation process), it passes. So you should be able to generate bytecode and just have it run. Look at Lib/test/test_pbcvm.py for some details here.

Good luck! Feel free to ask any questions on the jython-dev mailing list or #jython on IRC.

Flipping the 2.5 Bit for Jython

noreply@blogger.com (Jim Baker) — Tue, 24 Jun 2008 16:59:00 +0000

Something worth pointing out; as of 8 AM this morning (MDT) in rev 4748, Frank Wierzbicki flipped the bits and pronounced this about the ASM branch:

jbaker:~/jythondev/asm jbaker$ dist/bin/jython
Jython 2.5a0+ (asm:4750, Jun 24 2008, 10:56:16)
[Java HotSpot(TM) Client VM ("Apple Computer, Inc.")] on java1.5.0_13
Type "help", "copyright", "credits" or "license" for more information.
>>>

Yesterday there were easily the most commits we have seen in the Jython project. The real threshold was reached when we incorporated the UTF-16 and new-style exception branches into this branch, fixed the grammar to support most incremental parses, while repointing the standard library to CPythonLib 2.5. Along with a flurry of other fixes!

There's a lot more to go, but this should be an encouraging sign for everyone interested in Jython!

Adopting UTF-16

noreply@blogger.com (Jim Baker) — Tue, 24 Jun 2008 16:13:00 +0000

Jython 2.5 standardizes on Java 5 as the base version for its implementation. Jython has always mapped both unicode and str types to java.lang.String, but the semantics of String changed as of Java 5. Instead of encoding characters as UCS-2, that is just the basic multlingual plane of 65536 code points, Java - like .Net - adopted the UTF-16 encoding. UTF-16 can represent all 1114112 Unicode code points (U+0 to U+10FFFF), except for isolated surrogates (U+D800 to U+DFFF). These surrogates act as escape characters in the UTF-16 encoding.

This makes things somewhat more complicated, to put it mildly. And this is without even considering combining characters!

Instead of a simple uniform encoding that we see in the narrow (UCS-2) or wide (UCS-4) builds of CPython, we get a variable-length encoding. And unlike UTF-8, it's usually not too efficient. In addition, we lose the ability to represent the isolated surrogates. Finally, because UTF-16 is so very close to UCS-2, it's prone to bugs.

Here's the implementation strategy we adopted. In supporting the unicode type with PyUnicode, we first determine if it's in the basic plane or not:


private enum Plane {
  UNKNOWN, BASIC, ASTRAL
}

private volatile Plane plane = Plane.UNKNOWN;

public boolean isBasicPlane() {
  if (plane == Plane.BASIC) {
      return true;
  } else if (plane == Plane.UNKNOWN) {
      plane = (string.length() == getCodePointCount()) ?
           Plane.BASIC : Plane.ASTRAL;
  }
  return plane == Plane.BASIC;
}

getCodePointCount is in turn implemented using String#codePointCount. Like other code point methods, it decodes any surrogate pairs.

String immutability means we can cache the result in the volatile field plane; idempotence of this operation ensures consistency. This allows us to equate code units (char) to code points (int), and use the implementations provided by PyString. As it turns out, this was always done before, the only difference between str and unicode was in the encoding rules.

In the rather rare case it isn't, we read with our SubsequenceIteratorImpl (which does a decode and then moves forward in the string, rather useful) or String#codePointAt and write with StringBuilder#appendCodePoint using iterators. A seemingly good alternative would be to use String#offsetByCodePoints. Too bad it doesn't reliably work. So instead we have our iterator implementations, lots and lots of them. And sometimes crazy stuff like this, seen in the implementation of PyUnicode#unicode_strip:

        return new PyUnicode(new ReversedIterator(
        new StripIterator(sep,
            new ReversedIterator(
                new StripIterator(sep,
                    newSubsequenceIterator())))));

If strip method was used extensively on strings that weren't in the basic plane, it might make sense to rewrite this to decode to an int[] buffer. But that's not likely to be case.

That's also the reason we avoid making the basic plane test unless we have to. There are many situations where Unicode can pass in and out of Jython - specifically to/from Java - without us caring about what planes its characters are drawn from. We assume some overhead from boxing with PyUnicode (although HotSpot mitigates the indirection cost), but we don't have to overdo it by computing this test on construction.When comparing this with CPython, we do lose the ability to include isolated surrogate code points in Unicode strings. There are even some unit tests for this case. But ultimately this seemed like an implementation detail like testing ref counting, one certainly not worth time spent supporting.

It's worth mentioning that one alternative is to create our own representation, much like JRuby. Ruby's strings are mutable, unlike Python's. This forced the issue for the JRuby developers, because Ruby, like Python, needs good string performance. So JRuby uses byte arrays for strings, although they do use UTF-16 encoded, interned java.lang.String's to uniquely represent symbols (:xyz). Given that symbols are not strings, this works well. Ruby doesn't say anything about the encoding of such strings (ouch!), but JRuby does assume they're UTF-8 encoded when crossing the boundary with Java.

Supporting widened Unicode means having support for this in regular expressions. The first step was to just widen the SRE engine used by Jython to represent characters with int instead of short. So we always unpack to int in this case; see strip above. This engine is a direct translation of the CPython equivalent: it's a mini-VM, much like the pickle VM, and regexes are compiled to SRE bytecode. In the future, we may consider using JRuby's implementation (Joni, a port of Oniguruma to Java), but the devil is in supporting some specifics to Python. As was seen in the CPython case, it was quite straightforward to just doing the widening.

At this point, the biggest outstanding issue is backporting the changes to SRE to support wide character classes (aka big character sets), a pickle problem, as well as various bug fixes. A total of four test cases are currently failing in test_re in the asm branch.

And then that's it, at least until we start doing performance profiling.

Realizing Jython 2.5

noreply@blogger.com (Jim Baker) — Sat, 21 Jun 2008 03:43:00 +0000

Jython 2.5 is really, finally, unbelievably coming together. This is the next release of Jython, after last summer's 2.2. In a nutshell, we have completed all new language features using an Antlr parser, except for absolute imports. All bytecode generation work, now using an ASM backend, is done. Of course, there are many outstanding bugs. And Python is not just a language; we need to support fully the fact that "batteries are included". But let's look at where we are. Through the prism of what's new in 2.3, 2.4, and 2.5, here's what working:

2.3: sets (PEP 218), generators (255), source code encoding (263), universal newline (278), enumerate (279), logging (282), Boolean (285), distutils (301), new import hooks (302), pickle enhancements (307), extended slices, datetimes, optparse. Still to go: csv, removing a dictionary in builtin that ensures that interned strings don't get GC'ed (pre-2.3 behavior!, it helps to read what's new). Also various string, Unicode, and regex changes are mostly done in a separate utf16 branch that I'm currently in the midst of merging against trunk.

2.4: unifying long integers (237), generator expressions (289), string.Template (292, but also needs new utf16 work), decorators (318), reverse iteration (322), subprocess module (324), multi-line imports (328), removal of OverflowWarning, min & max with keyword support, sorted. But we still need partial import with sys.modules, and I'm sure some more stuff I forgot. Decimal and -m support are working in student branches, we just need to incorporate.

2.5: conditional expressions (308), partial functional (309, but we're cheating with a pure-Python version), distutils metadata (314), unified try/except/finally (341), coroutines and other generator functionality (342), with-statement, including contextlib (343), any, all. But we haven't done the exceptions remapping to new-style classes, absolute and relative imports, or all of the context manager support, such as in file. ctypes was a proposed Google Summer of Code project, but apparently PyPy has some work that's 95% the way there; we will talk with them at EuroPython. We need to look into what is necessary to make ElementTree work. sqlite3 depends on ctypes. As I was writing this, I tried out wsgiref; it works and I just committed it to the asm branch. (At some point, we will repoint everything like this to CPythonLib, but for now we are mixing it up as we go. Bear with us!)

Even quit() and exit() now work; I don't know when these oh-so-major features were added. We even now support large string constants. And of course, who can forget our support for the GIL (global interpreter lock) in Jython, something that Tobias Ivarsson, my Google Summer of Code student who is now working on an advanced compiler, added to __future__ as an Easter egg:

>>> from __future__ import GIL
Traceback (most recent call last):
(no code object) at line 0
File "", line 0
SyntaxError: Never going to happen!

I would imagine that's definitive, we go against Java's native threads and compile to Java bytecode. It would be hard to have a GIL, even if we wanted one.

However, we are just turning the corner. The The Antlr parser in the asm branch currently does not support partial parses, and this breaks not only interactive sessions but doctests. Until this is solved - and Frank Wierzbicki is working like mad on this - we can't merge this branch onto trunk. But that should happen very soon.With few exceptions, we simply go against the standard Python unit tests. Straightforward, cunning, or devious, we have labored against these unit tests. And in others, we have used Python as our foil: we support the same 2.5 AST parse tree, and we know this by comparing our parses with CPython's for all of the standard library - including those unit tests.

There's a lot more going on. I can't say enough about the work done by Charlie Groves, Philip Jenvey, Alan Kennedy, Nicholas Riley, and others to make this happen. Leo Soto, my other GSoC student, is making amazing progress on supporting Django on Jython, while finding and fixing bugs in Jython itself. Supporting Django forces us to find those gaps in compatibility. Similar efforts are going on with Pylons, TurboGears 2 (Ariane Paola, GSoC), and Zope (Georgy Berdyshev, GSoC). I'm also working on greenlet/Stackless support and involved in a collaboration with Jeremy Siek and Joe Angell at the University of Colorado to add gradual typing (yes types! but only when you want to) to Jython. We have a T2000 contributed by Sun to let us see how much concurrency - in this case 32 hardware threads, 64 GB of memory - Jython can take advantage of. And so on.

Back to work!

Updates - 2008-06-24: we have support for new-style exceptions, the parser is now usable (but there are a couple of bugs left there), and Unicode support has been updated to UTF-16. See this posting, Flipping the 2.5 Bit for Jython.

Django on Jython: Minding the Gap

noreply@blogger.com (Jim Baker) — Fri, 04 Jan 2008 04:59:00 +0000

Summary

The most important thing to know about Django on Jython is that we are almost there, and with clean code. End-to-end functionality is demonstrated by the admin tool running in full CRUD, along with a substantial number of unit tests and syncdb. But this has been achieved by so far requiring only 6 lines of code in changes to Django trunk. (There will be more, however, see below.)

Running on Jython

To run Django on Jython, with a PostgreSQL backend, the following steps are necessary:

Use the Modern branch of Jython. This consolidated the bugs, workarounds, and patches of numerous people - plus a bunch more - in a stable, almost-ready-to-be-merged-into-trunk version of Jython. The most important aspect is that we have tried to make Jython conform more to CPython, using Django as our guide, although there are some gaps - especially if Django already had incorporated fixes. Our driving goal is to converge on these gaps over time. Please note that is intended to be stable, performant code.
Use the Django trunk (tested with rev 6992, later should be OK too).
Apply these two patches, django.dispatch.robustapply (diff) and django.views.debug (diff) due to Leo Soto. I would imagine these will be in Django trunk soon.
Copy these three files from CPythonLib to Lib: gettext.py, locale.py, optparse.py. Please note that these files are only partially working on Jython, that's why they haven't been promoted yet (gettext.py actually works, as verified by test_gettext.py, but depends on still failing locale.py). But they are very close, and they appear to be fine for Django. Certainly fine for this round of development!
Use the database backend zxjdbc_postgresql, which was contributed by Leo Soto. Frank Wierzbicki has an experimental backend for MySQL, this should be incorporated soon.

Status

Here's what works:

syncdb and the very cool Django admin run; many unit tests pass. You can run with internationalization enabled. You do need to run the dev server with --noreload for now. We need to document here how to run with modjy, which is Alan Kennedy's servlet container for WSGI apps.

In running the model unit tests, here are the things we seem to be missing, accounting for most of the approximately 75 failures:

Many doctests are fragile, because they depend on the dict traversal ordering; in Jython, this is different that CPython, and if we adopt ConcurrentHashMap, it's not even repeatable. This would seem to be a pervasive bug in Django.
We still have some encoding problems, again seen in doctests. An example where output is expected to be lower case hex, not upper case. I fixed the problem in PyUnicode, but there are more places.
Problem with the ManagerDescriptor handling, in django.db.models.manager.
No decorators yet! (But they are coming soon, and are now available experimentally for Jython in the newcompiler work I have been leading.)

There may be some other rough categories, we need to look at the failures more systematically. All that doctest noise is certainly annoying!

Next Steps

On the Django front, get more of the unit tests running!

Before we can push modern into trunk, the following needs to be done:

The test_extcall unit test currently fails. This appears to be a dependency on dict traversal being repeatable, a bad assumption. However, it's a mind bending test. The 2.3 version is particularly problematic because it's not modular at all. Google's GHOP has just produced an improved version for Python 2.6 - we will look at this as a starting point.
Tristan King provided a near complete subset of the functionality for time.strptime, as implemented in org.python.modules.time.Time. This needs to be enhanced. I just tested this, and all unit tests in the CPythonLib version of test_time now pass except for strptime -- specifically the conversion specifier '%c' -- so we can also move to that, and discard our Jython version, when this is completed. That should be soon!
Decide whether we should use ConcurentHashMap or not as the backing hash map for dict and __dict__. CHM introduces creation overhead, but it should prove to be far more scalable on multicore systems. The programming model is also far nicer with respect to Jython.

Updates - 2008-06-24: I should have put this up a while ago, but Django on Jython is becoming a reality. Most importantly, Leo Soto is working with me through the Google Summer of Code on this project. The modern branch was merged into a trunk earlier this year, and has since been retired. CHM in fact has the right semantics, something I may discuss at a future point. Django has redone the doctest dict literals that were causing problems, and Leo provided a general solution when used with XML/XHTML.

Pythoneers Monthly Meeting: This Wednesday, December 20, in Boulder, Colorado

noreply@blogger.com (Jim Baker) — Sat, 16 Dec 2006 16:17:00 +0000

Important Change: we will be meeting at bivio Software instead of Jill's to better accommodate this month's demos.

This coming Wednesday (December 20) we are having our monthly meeting for the Front Ranage Pythoneers. Come join a lively discussion of Python demos, features, tips & techniques, and directions, both for fun and professional development.

Here are the meeting specifics:

Date/time: Wednesday, December 20, 6-8 PM
Location: bivio Software, Inc., 28th and Iris. Above Hair Elite in Suite S. Google Maps link
Tom Churchill and Vinny will demo Churchill Navigation's earth-rendering engine (which looks like Google Earth, only apparently even better and faster ;) ). Vinny (their main Python guy) will explain how they built the glue logic (and why they decided against SWIG) and perhaps some of the implications of using Python as a scripting language in a real-time (60 fps) environment, and the techniques we employed to keep the graphics pipeline from stalling when making an expensive call into their engine from Python.
Brian Granger from Tech-X will help us think more deeply about concurrent Python programming, especially as seen in a new version of IPython he has been working on.
BoulderSprint. Eric Dobbs proposed we adopt Jython, and this looks like we have enough momentum to actually get some useful work done. We will talk about the upcoming sprint to be held on Saturday, January 6.

We will have food & drink available. Did I mention the free beer? Hope to see you there.

- Jim

Pythoneers Monthly Meeting - This Wednesday, Nov 15, in Boulder, Colorado

noreply@blogger.com (Jim Baker) — Sat, 11 Nov 2006 02:55:00 +0000

This coming Wednesday (November 15) we are having our monthly meeting for the Front Range Pythoneers. Please note, we have moved to Jill's at the St Julien Hotel. Come join a lively discussion of Python features, tips & techniques, and directions, both for fun and professional development.

Here are the meeting specifics:

Status: Always on
Date/time: Every 3rd Wednesday, 6-8 PM
Location: Jill's at the St Julien Hotel (9th and Walnut), the bar area. Jill's combines a beautiful room, great food and beverages, and happy-hour pricing. And for this meeting: your first draft beer (up to 20 people) is free!

Hope to see you there.

We've recently set up a wiki. There, you can help us expand the Guide to Front Range Pythoneering. Or contribute ideas to future sprints and jams.

Boulder Sprint: Adding Oracle support to Django

noreply@blogger.com (Jim Baker) — Mon, 06 Nov 2006 19:08:00 +0000

We had a great turnout on Saturday for the first Boulder Sprint held by the Front Range Pythoneers. Our goal was to provide production-level support of Oracle in Django 1.0. I'm glad to report that we made a strong start on this goal.

Django developer Jacob Kaplan-Moss flew out to Boulder from Lawrence, Kansas, providing us both leadership and guidance into the Django internals. (Next time I hope Jacob doesn't have to fly out the next day on a 6 AM flight!) From Array Biopharma, we had five developers: Ian Kelly, Matt Boersma, Matt Drew, Michelle Cyr, and Mitch Smith. Eric Dobbs, of Bivio, contributed both the space and his seasoned Python skills. And there was me (Jim Baker). Thanks to everyone for your hard work!

We worked on a number of key issues for supporting Oracle:

Perhaps most important, Jacob split out Oracle-specific functionality into the Oracle backend, allowing for more modularity. Django uses quite portable code, in conjunction with the Python DB2 API, but Oracle has its peculiarities. Being pragmatic, we had to work through that.
Mapping Django's TextFields to Oracle's CLOBs, not LONGs, which pretty much are deprecated. (Remember Django's origin, we certainly need support for text!) However, supporting CLOBs required some changes: no buffering in the Python layer, just iterate directly over the cursor; explicitly read in data from the LOB reference; prepare the OCI by giving cx_Oracle explicit type information (also necessary for timestamps with greater than one second precision).
Pagination queries. Django's ORM grew out of supporting PostgreSQL, which has OFFSET and LIMIT clauses, useful for the pagination queries often seen in stateless web apps. Oracle actually has quite good support for this type of queries but this fact is not well-known. And frankly it's a bit clumsy to use, requiring doubly nested subqueries. See Oracle guru Tom Kyte's article in Oracle magazine for more details. I made some progress on this front, but I still need to integrate it into the new django.db.backends.oracle.OracleQuerySet class added by Jacob.
Test schema support. Oracle uses the concept of "user schema" where other databases might use "database". There's a bit of trickiness in working appropriately with this, especially if there are tablespaces being set up for this test. Eric took the lead on this.
Mitch Smith wrote two gnarly Oracle-specific queries that have almost got
introspection and Django's "inspectdb" command working correctly.
Part of our goal was to get all existing tests to pass from runtests.py, and we're about 70% there.
Array Biopharma now has their test web app, donuts, running.

And this is just what I saw from my side of the conference room table! We do have a photostream for the Boulder Sprint. Check out Matt B. and Michelle contemplating at the Oracle. Or Matt B. and Ian asking rhetorically, "How can we screw up a 3-line function?" There is also a wiki.

For the moment, until we get the work integrated in the Django trunk, you can checkout the Boulder Sprint branch here with Subversion:

svn co http://code.djangoproject.com/svn/django/branches/boulder-oracle-sprint

Boulder Sprint this Saturday

noreply@blogger.com (Jim Baker) — Tue, 31 Oct 2006 17:30:00 +0000

On November 4, 9 AM-6 PM, the Front Range Pythoneers is holding a sprint to complete the support for Oracle in Django. Why might you want to attend? Whether you're interested in Django, portable object-relational mapping code, how to optimize an Oracle execution plan for generated SQL, or just doing some intensive coding in Python, this should be a great opportunity to learn and contribute.

See the wiki for more details: http://wiki.python.org/moin/BoulderSprint