Maybe writing a dialect of Tickle that is very compressible would be nice. I guess using one byte markers to mark beginning and ends of data would be very compressible. Eg: [here starts a tuple] [an int] 5 [an int] 5 … [here ends the tuple], etc. Basically repr but then with binary data and no extra spaces.

The format you are describing is the one currently used by pickle. In protocol 1, a tuple of three 5 is serialized as:
(I5
I5
I5
t.
Or in the newer binary protocols:
(K\x05K\x05K\x05t.
(where K is the opcode for 1-byte integers)

If you are curious, I do have an article about how pickle work. It doesn’t cover the details of the newer protocols, but it should give you a good start to learn them afterward. Anyway, I am currently working on the next version of the pickle protocol for Python 3K. So if interested to help, just send me an email.

1024   5514   3351   3367
2048   10918  6910   6414
4096   20763  12792  11833
8192   39492  23893  22166
16384  77385  46063  44242
32768  152429 89529  86426

Or that pickle at least has got a very compressible tuple format.

Now I wonder how it depends on the size:

1    37    31    27
2    48    38    33
4    68    50    43
8    99    71    62
16   153   103   91
32   254   150   147
64   467   235   253
128  952   431   489
256  2005  834   974
512  3959  1739  1782

Again with the same test object but then with a different amount of elements. First column is the amount of elements. Second the size of pickle, third of pickle with fast attribute and the last of tickle.

Maybe writing a dialect of Tickle that is very compressible would be nice. I guess using one byte markers to mark beginning and ends of data would be very compressible. Eg: [here starts a tuple] [an int] 5 [an int] 5 … [here ends the tuple], etc. Basically repr but then with binary data and no extra spaces.

import pickle, tickle
from StringIO import StringIO
from gzip import GzipFile

def load(data, load_method):
  s = StringIO(data)
  gz = GzipFile(fileobj=s, mode="rb")
  return load_method(gz)

Again, using your example ‘obj’:

>>> len(dump(obj, pickle_dump))
717
>>> len(dump(obj, pickle_dump_fast))
342
>>> len(dump(obj, tickle_dump))
386

I am quite surprised to see how well pickle streams, generated using
the ‘fast’ attribute, compress. However for tickle streams, the gain
is minimal, which, I presume, means that your protocol is compact.

gzip doesn’t make a whole lot of a difference on small rpc packets, I guess. Maybe delta-compression would though (which is just a fancy word in this case for persisting the state of the compressor between messages). I’ll write two backends, one Tickle and one ‘GzPickle’ to test it.

To make writing templates more convenient I’ll write a pack lilke format string that is converted magically by (un)pickle to an appropriate tuple.

When writing in Python itself, it’s suboptimal anyway. Using classes really doesn’t make that much of a difference compared to using a table of functions in python anyway. I should write a c version of Tickle if it ends up to be useful.

>>> obj = []
>>> for i in xrange(100):
… obj.append((i, str(i)))
…
>>> len(pickle.dumps(obj))
2178
>>> len(pickle.dumps(obj, 2))
1098

Anyway, I saw that you used pickle.HIGHEST_PROTOCOL in your tickle module, so probably already knew this. Yet, you can improve the size even more by using the (undocumented) ‘fast’ attribute, which turns off memoization.

>>> s = StringIO()
>>> p = pickle.Pickler(s, 2)
>>> p.fast = True
>>> p.dump(obj)
>>> len(s.getvalue())
696

However by disabling memoization, pickle will enter in an infinite loop if it encounters an object that is cyclic—e.g., L = []; L.append(L). Now if you really care about size, nothing stop you from using the gzip module.

Overall, there is a few neat things about your serialization protocol. I like how you organized the code—i.e., each type has his own class, instead of one big serializer class. I am sure there is some drawbacks to this approach, but it is neat nevertheless.