This is a plot of the amount of documents created in a bulk update at the same time against the average amount of documents created per second it yields.

And for (two concurrent threads. Tested on a dual core machine)

The values of B are 1, 2, 4, 5, 8, 11, 16, 22, 32, 45, 64, 90, 128, 181, 256, 362, 512, 724 and 1024

As you can see, a higher value of causes the graph to shift to the right which means more for the same time. Bulk update really does make a difference. Or non-bulk-update really sucks. Also adding threads does help a bit, but not as much as expected.

There are some more interesting graphs to plot ( against ). More graphs tomorrow.

(For those interested, the raw data from which these graphs were plotted.)

Today I’ve been busy hacking together a little script to import all e-mails of a long e-mail thread into a couchdb database to write views to extract all kinds of statistics. I already imported these e-mails into a MySQL database a few months ago, but was quite disappointed by the (performance) limitations of SQL. The e-mail thread contains over 20,000 messages which weren’t a real problem for MySQL. When importing, however, couchdb was adding them at a rate of only a few dozen per second with a lot of (seek)noise of my HDD.

So I decided to do a simple benchmark. First of, a simple script (ser.py) that adds empty documents sequentially. It’s averaging 16 per second. It occurred to me that couchdb waits for a fsync before sending a response and that asynchronously the performance would be way better. A simple modification to the script later (par.py) it still averaged 16 creations per second.

I guess, for I haven’t yet figured out how to let straces tell me, that it’s the fsync after each object creation which causes the mess. couchdb itself doesn’t write or seek a lot, but my journaling filesystem (XFS) does on a fsync.

Can anyone test it on a different filesystem?

Update Around 17/sec with reiserfs.

Update I had some trouble with the bulk update feature. I switched from svn to the 0.7.2 release. I got about 600/sec, which dropped to a steady-ish 350/sec when using sequential bulkupdates of 100 docs. Two bulk updates in parallel yield about 950/sec initially, dropping to 550/sec after a while. Three parallel updates yield similar performance.

When I want to shutdown an instance of the server, I have to interrupt the select. I have yet to find a satisfying way of doing this. At the moment I create a pair of linked sockets with socketpair. I include one of them to the sockets on which to block until there is data to read in the select call. To interrupt, I simply write some data to the other socket which will cause data to be available on the socket which in turn will interrupt the select.

There must be a more elegant solution.

The installation of Gentoo went quite smooth. For those interested the xorg.conf, make.conf and kernel .config I use.

Internal mic/speaker and external jacks; video; DRI and AIGLX; USB; PCMCIA; wireless; fingerprint reader and ethernet all seem to work just fine. I haven’t tested the firewire, ssd slot, bluetooth and n-capabilities of the wireless yet.

In comparison with my T60 the volume and the backlight buttons aren’t hardware controlled. Gnome recognizes the volume buttons, but not the backlight ones. I’m still working on those.

Thanks to Bram for the tip.

After the check-in we went for a quick diner and straight of to the Delirium Café for the “Beer Event”. The café, even though it’s the largest one I’ve ever visited, was barely big enough for the masses of geeks. One could buy a ‘red dot’ from one of the organizers for 20,– euros. With it you can order any beer (one of the 25 on the tap) at the bar without paying until you spend the 20,– euros. This wasn’t checked though. I feel it hard to imagine, and my head felt even harder the next morning, that one would consume more. On that next morning the real magnitude of the conference became apparent. The first talk, at a large (really, large) college room at the Free University of Bruxelles, was attended by at least a thousand if it weren’t two-thousand people.

The talk about Perl 6 was the most interesting for me. I didn’t really like Perl <5 primarily because of having too many ways to do the same in exactly the same way but with a different syntax. I knew that Perl 6 was a total and backwards incompatible redesign of the language, build on top of a generic and good virtual machine called Parrot. Parrot, which I hadn’t given a proper look yet, turned out to be a lot greater than expected. You write support for a new language in Parrot by writing in a subset of Perl 6, which with it’s new Regular Expressions and specializations (tokens: regex without backtracking, etc), was looking very suited for it.
Except for all the new syntactic very very sweet sugar (on which I won’t (yet) elaborate) they added in Perl 6, the greatest one (which is actually more of a Parrot thing) is being able to extend Perl during runtime: writing new parser rules. One application is being able to define ‘!’ as a faculty operator. I’m itching to play with it.

Another very interesting talk was the one about Gallium3D, which is an effort to rework the 3D API on Linux and Xorg for 3D accelerator drivers. It primarily abstracts most of the operations of the video card to being a very specialized language processor and not entirely surprising makes heavy use of the llvm. The abstraction was effective enough to being able to take a driver originally written for Linux, rework it for Gallium3D and actually being able to be run on a Windows version of Gallium. That it runs on Windows really can’t bother that much people. That it also now runs on Cell’s SMP’s, can. The speaker joked that one finally can play 3D games on your PS3 [ on Linux ].

Most of the talks I attended were pretty interesting. Most of the speakers, though, weren’t that great speakers. The thing I missed most with FOSDEM was that there was no easy way to get in touch with people of a certain project. There were booths of a lot of projects, but there the people behind the booth were just standing there to show you stuff in the hallway that was too narrow. There were separate rooms for projects where there were talks given about the project, but there was no real time in between. A few hours nothing but people of the project meeting in a project room would have been great.

Miscellaneously, the OLPC is a great toy; the EEE is even uglier than it looks on the web; the MacBook Air is even thinner than you think and the FSFE people with whom we stayed at the hotel were great. I could tell a lot more, but quite frankly I’m very tired and sleep calls.