Intrepid Blog

msgpack is a fast and small binary format for json. The Python msgpack module is even a drop-in replacement for the json module.

PyPy is an implementation of Python in Python with a tracing JIT. If your Python script runs for more than 1 second, it will probably be quite faster on PyPy.

PyPy is almost a drop-in replacement for Python. Only extensions modules written in C can be a problem — for instance the msgpack module.

Thus I wrote a pure Python fallback for the msgpack module, which has been merged upstream. As a rough benchmark, I measured the pack and unpack time of a 30MB msgpack-object from the wild.

Note that:

• The execution times for the fallback with normal Python are all off the chart (15.4s, 15.1s, 10.1s, 10.0s).
• PyPy is faster on the second run: it had time to trace and optimize at runtime.
• For packing, PyPy (nightly) is almost as fast as the original C extension.
• For unpacking, PyPy (nightly) is 340% slower. Yet it more than 10 times faster than normal Python.

This code should be in the next official release of msgpack-python. If you want to use it already, check out the git repository.

The first version did not run this fast on PyPy. With PyPy’s jitviewer the compiled code and assumptions of PyPy can be examined. These are the tweaks I used in descending order of impact.

• Instead of StringIO, the fallback uses PyPy’s own StringBuilder. StringIO allows writing, reading, seeking and what more. StringBuilder only allows appending and thus it is easier for PyPy to optimize. This increased performance of packing by an order of magnitude.
• Using constant format strings for struct.unpack allows PyPy to optimize it. Thus struct.unpack("I", f.read(4)); f.read(n) instead of struct.unpack("I%ds"%n, f.read(4+n))
• Using stream.write(a); stream.write(b) instead of stream.write(a+b) increased performance.
• Adding an explicit fastpath
• PyPy usually specializes a function well for its most common path, however in some cases it needs help. In this case a function returns a concatenation of several strings. However: in the most common case it only does one: ret=''; ret+=something; return ret. PyPy does not recognize that ret is not needed at all in this case, so I added an if-statement before initializing it for the case where there is only one concatenation.

Clearly the unpacking code could be faster, if anyone with expertise on PyPy’s JIT would look at it, that would be great.

I compiled PyPy 2.0-beta1 for Debian Squeeze amd64.

I wrote a simple Python script to pack and unpack the data files of the indie game FTL, which is by the way quite enjoyable. You can find it here on github.

It may happen that Sparrow for Mac uses an excessive amount of disk space. For only the headers of my 9GB of e-mail, Sparrow used 39GB. I found a work-around.

The problem
Sparrow uses a Tokyo Cabinet, called data.tch, as storage. Mine was 39GB. What could eat up all that space? I only have 9GB of e-mail in total. With a simple script I looked into the cabinet and found it actually contained less than 1GB of data. The cabinet was wasting a lot of space by not being packed tightly.

The work-around
To repack the Tokyo Cabinet, run:

tchmgr optimize "~/Library/Containers/com.sparrowmailapp.sparrow/Data/Library/Application Support/Sparrow/(your-email@gmail.com).sparrowdb/data.db/data.tch"

Replace (your-email@gmail.com) with your e-mail address. You can install the tchmgr program using sudo port install tokyocabinet, if you are a MacPorts user.

This reduced the size of the cabinet to 1.1GB!

Cause?
It is odd that the cabinet became so loosely packed. It may happen if Sparrow adds and removes a lot of entries, but there seems no reason for Sparrow to do this. There might be an underlying bug. I reported this issue to the developers, but they have yet to respond.

You can fix this error by explicitly setting the default audio device. For instance, put this:

pcm.!default {
 type hw
 card 0
}

ctl.!default {
 type hw
 card 0
}


in $HOME/.asoundrc

$ shnsplit -o 'cust ext=ogg oggenc - -o %f' \
  -f CDImage.cue -t "%n.%p - %a - %t" \
  CDImage.ape
$ cuetag CDImage.cue *.ogg

If you’re on Ubuntu, you’ll need to install cuetools and shntool. To split ape, compile and install this port of mac.

De Radboud Universiteit censuurt onder het mom van een duidelijke scheiding tussen intern en extern nieuws sinds kort het niet-meer-onafhankelijke blad de Vox.

Gelukkig kunt u het nieuws alsnog lezen op RU Nieuws. Op deze site wordt al het interne nieuws gelekt.

For a project I need to store blobs of data (~10MB) and access them over TCP. I don’t need any features like removing, updating, authentication, statistics or replication: it’s simply not required or already handled by some other part of the project.

I only want to be able to store a blob and receive a key for it; retrieve a blob by its key and list all keys of the blobs stored.

I couldn’t find anything that fit it. Thus I created bertha. Lets delve right into it, shall we:

To run the server:

./berthad-vfs 0.0.0.0 1234 tmp data

There’s a python client

>>> from bertha import BerthaClient
>>> c = BerthaClient('serf', 1234)
>>> list(c.list())
[]
>>> key = c.put_str('Hello world')
>>> key
'64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c'
>>> c.get(key).read()
'Hello world'
>>> list(c.list())
['64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c']
>>> ctx = c.put()
>>> ctx.f.write("Do some")
>>> ctx.f.write("Streaming")
>>> ctx.finish()
'975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0'
>>> c.get('975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0').read()
'Do someStreaming'

Which also install a bertha commandline tool:

$ bertha list
975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0
64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c
$ bertha get 975001fb9bdc0f72a78ca6326c55af86348d4c84da7ba47b7ed785a03f6803b0
Do someStreaming
$ echo Hi | bertha put
c01a4cfa25cb895cdd0bb25181ba9c1622e93895a6de6f533a7299f70d6b0cfb
$ bertha get c01a4cfa25cb895cdd0bb25181ba9c1622e93895a6de6f533a7299f70d6b0cfb tmp
$ cat tmp
Hi

The berthad code is pretty small: at the moment under a thousand lines of C with lots of comments. GETs are pretty fast: berthad uses Linux’ splice syscall, which usually makes the network card read directly from the buffer the harddisk wrote to.

When I try to set the displaymode of my external monitor to 1024×768@60 the preferences application doesn’t listen properly and sets it to 1024×768@120, which my monitor does not support. This seems to be a bug in the developer preview.

To remedy this bug, I wrote a simple commandline tool to set displaymodes in Mac OS X. It’s on github.

On the developer preview of Lion, the “+” button for 802.1X profiles was removed.

You can use the iPhoneConfigurationUtility application (Google for it) to create a mobile configuration profile with the 802.1X settings, that also imports perfectly fine on Lion.

Simply install from Subversion.

I have some remarks on Steve Jobs’s “Thoughts on Flash”.

1. “Adobe claims that we are a closed system, and that Flash is open, but in fact the opposite is true.”. I agree with Steve: Flash is pretty closed. However, the iP(hone/od/ad) isn’t open either.
   1. H.264, which Jobs touts as a great modern replacement of flash, is patented. You have to pay whatever the MPEG LA fancies you to pay for use of the standard. This is not really different from the control of Adobe over Flash.
   2. iPhone OS is closed. You need to buy yourself into the iPhone Development Program. Again Apple can shut the program down whenever they like, which is not really different from the control of Adobe over Flash development.
2. Safari is an intermediate layer and creates sub-standard apps. Jobs claims that third-party intermediate layers result in sub-standard apps. Jobs argues that an intermediate layer will keep developers from platform enhancements, won’t result in targeted great apps and won’t put apps directly on the shoulder of the platform. If he is thinking about Safari, he is right: web-apps for the iPhone just don’t feel great, can’t use all the platform enhancements and don’t result in great targeted apps.

Dear readers,

I’m looking for an apartment in the neighborhood of Nijmegen. I’m not picky, but can’t afford more than €300,- in total per month.

Contact me per e-mail (bas@this domain).

Bas

Update I’m settled!

This week, various student organisations protest against major cuts in the funding of education and research.

A reader of dutch, can visit their site.

As pointed out by Jorg Kennis, Twitter‘s new lists feature make it hard to determine the amount of unique followers. I’ve written a simple script, using a slightly modified TweePy, to determine the amount of unique followers.

Example usage:

bas@w-nz ~/twitter-unique-followers $ python twitter-unique-followers.py JorgK -ubwesterb
Password:
rate_limit_status: remaining_hits: 112
Jorg Kennis
 followed directly by 182
 in lists
  RPtje/vriendenbekenden subscribed to by 0
  JorgK/TechNL subscribed to by 0
  sentfanwyaerda/nijmegen1 subscribed to by 9
  JorgK/Friends subscribed to by 0
  sjorsjes/Community subscribed to by 0
  robinspeijer/iPhone subscribed to by 1
  nielsschooneman/iPhone subscribed to by 0
  JeanPaulH/iPhoneclub subscribed to by 5
number of unique followers: 198

You can download it here. I could write a simple webpage with the same functionality, if anyone would mind.