Notebook list

(1) Thu Sep 01 2005 22:34 CET-1CEST:

I've implemented the frame-recording thing I posted about yesterday. It's now the function annotate.findcreators, and it seems to work *very* well. In fact, I think I prefer it to annotate.groupby: it doesn't have to guess, it seems to give better information and it might work well in a GUI (the information it gives is hierarchical).

And I can't help noticing that the SoC is coming to an end...so I thought I should post about what I've done.

* Everything is split (with the exception of annotate.findcreators at the moment) into Python-implementation-specific (CPython, PyPy, IronPython etc.) and implementation-dependent parts. There's a cpython directory which contains everything CPython-specific.

To get another implementation running, you should only need to write classes to scan instances of object and whatever other types you have that need special treatment, and write functions to get a set of roots and to get the name of a type.

* The scanner works. It's written in an implementation-independent way (I hope). If the implementation supports type.mro(), it can construct a wrapper of a type from the wrappers of its superclasses - i.e. given code to scan class Foo and class Bar, it can scan class Baz(Foo, Bar).

In practice this means that only scanning code for very strange objects (in the CPython implementation, C objects) needs to be defined, and the scanner will sort out the rest itself.

The scanner gets wrapper classes from the module sizes, which should be filled in by the implementation-specific part, and which can also be used to treat some objects and types specially. For example, the scanner puts an entry in here which causes no wrappers to be counted as taking space.

* The code is quite useful, with a fair amount of ways of analysing the scanner's output (including making graphs :-). In a short time I found some waste of memory in SCons and in about half an hour reduced its memory usage from 24MB to 22MB. (I haven't tried to reduce it any further yet - there's certainly room for improvement.)

* There are (hopefully useful) docstrings on all public functions and classes. I've written part of a tutorial, which has links to HappyDoc-generated docs made from the docstrings, and a document describing the internals of the scanner.

* The CPython-specific code can scan all built-in types which are at all possible to scan.

On the other hand:

* I haven't written any code to scan any types from extension modules. I will do when it becomes a problem - lots of programs don't use extensions much.

* There's a bit of a mess when making copies of wrappers. I think I can fix that by adding an extra level of indirection.

* Each wrapper contains a reference to the object it's wrapping, and quite a lot of code uses it. That's not very good for mutable types, since the object can have changed when code looks at it, and it also prevents pickling the sets of wrappers for future use.

* There's no nice GUI yet...once I've fixed the problems with wrappers, I think I'll have a go at writing one.

Also, I want to port it to other implementations. PyPy looks very tempting :-)

Wed Aug 31 2005 10:15 CET-1CEST Algorithms and whatnot:

I've been trying to think of the best way to use the frame recording code.

It could be useful to be able to see all the functions which created objects along with which objects they created. Then you could look back to see which functions called them, and which objects were created by the back function calling the other function.

Then you could just give out a head which represented Py_NewReference, or something like that, and created all the objects. The back functions of that would be the pieces of code which created objects themselves.

That seems to be a sensible way of representing it, I think. I'm not quite sure how to implement that...it looks like a suffix tree might be the best way.

Mon Aug 29 2005 14:46 CET-1CEST More fiddling with SCons, and some more docs:

I've looked at SCons some more. A huge amount of its memory is taken up by dicts. Instances of BuildInfo seem to take up a lot of space too, but they're really a big mess and it doesn't look too easy to fix. Never mind.

I've started writing a document about how the profiler works internally. It's also an excuse to learn LaTeX :-) At the moment it goes through how the scanner works, and has a couple of ideas for improving it.

Sat Aug 27 2005 15:35 CET-1CEST Trying on SCons:

The profiler found that there are lots of empty dictionaries in instances of SCons.Node.FS.File, so I replaced a few of them with wrappers which only make a real dictionary when needed.

Before the change (output is from scons --debug=memory -n):
Memory before SConscript files: 7516160
Memory after SConscript files: 17731584
Memory before building: 17731584
Memory after building: 23470080

After the change:
Memory before SConscript files: 7520256
Memory after SConscript files: 16392192
Memory before building: 16392192
Memory after building: 21549056

So that small change (writing a lazy dictionary class which took about half an hour, and 28 other lines, mostly changing {} to lazydict()) has reduced memory usage by 2MB out of 24. Which is not bad, but not very big either. I'll have another look later to see what else can be reduced.

Fri Aug 26 2005 22:35 CET-1CEST:

Well, I've got the recording-frames idea going, but no code to use it yet. It needed changes to _Py_NewReference and _Py_Dispose in the end. Also, I've made the wrappers use slots, so the profiler itself uses a *lot* less memory than before.

I've been trying to profile SCons. Looks hopeful: when compiling Unununium, there are several megabytes of empty dicts and lists. Maybe it would help to have a lightweight dict-wrapper class that only created a real dict when it was written to.