On Python and Pickles

Currently foremost in my mind has been my annoyances with Python.

My current gripes have been with pickle.

Rather than taking a conventional approach and devising a fixed protocol/markup for describing the objects and their state, they invented a small stack based machine which the serialisation library writes bytecode to drive in order to restore the object state.

If this sounds like overengineering, that’s because it is. It’s also overengineering that’s introduced potential security problems which are difficult to protect against.

Worse than this, rather than throwing out this mess and starting again when it was obvious that it wasn’t meeting their requirements, they just continued to extend it, introducing more opcodes.

Nevermind that when faced up against simpler serialisation approaches, such as state marshalling via JSON, it’s inevitably slower, and significantly more dangerous.

And then people like the celery project guys go off and make pickle the default marshalling format for their tools rather than defaulting to JSON (which they also support).

Last week, I got asked to assist with interpreting pickle data so we could peek into job data that had been queued with Celery. From Ruby.  The result was about 4 hours of swearing and a bit of Ruby coding to produce unpickle. I’ve since tidied it up a bit, written some more documentation, and published it (with permission from my manager of course).

For anybody else who ever has to face off against this ordeal, there’s enough documentation inside the python source tree (see Lib/pickletools.py and Lib/pickle.py) that you can build the pickle stack machine without having to read too much of the original source.  It also helps if you are familiar with Postscript as the pickle machine’s dictionary, tuple and list constructors work very similarly to Postscript’s array and dictionary constructs (right down to the use of a stack mark during construction).

Posted in Software Development | Tagged , , , , | Leave a comment

LCA2012: Day -1: Arrival and Registration

So, I’m in Ballarat for LCA2012, courtesy of my employer.  I’ll be presenting on Tuesday along with David on some of the work I’ve done on a few tools for systems automation.

The flight was uneventful – although the landing was a bit rough…

The drive from the car rental place to Ballarat was fortunately uneventful, but plagued with a combination of idiots who insist on doing things the hard way (for crying out loud, don’t try to shoot past me on the left if I’m indicating a left merge already >_< ) and the fact the car’s speedo reads about 5-7kph under actual resulting in us doing about 92kph down the western highway when I thought we were doing 98kph (I have a 100kph limit still.  damned P-Plates).  The car is a newish Camry and is by far the newest thing I’ve actually driven any distance now – quite impressed in some ways, but disturbed by the lack of vision due to the body shape (pillars are huge >_< ) and the very light steering.

Fortunately we reached Ballarat and found the motel without troubles.

Registered at the University and picked up the schwag pack.  This year’s bag is a crumpler sack style thing, contianing: A catalyst branded screen cleaning cloth + case, Bulletproof Networks branded mousemat (looks like it might be faux leather or even real leather), an LCA branded keep cup, intel branded universal USB -> smartphone charger, DSD branded stubbieholder and careers spiel pamphlet, LCA t-shirt, LCA conference guide and a Freetronics LeoStick.

Met up with some people and went to Murphy’s – an Irish pub on the main drag – for dinner.  Had the Irish Pizza which was quite pleasant, and generally emptied my wallet on food. (Ouch).

All in all a good start to the week – tomorrow is miniconfs.

Posted in Uncategorized | Leave a comment

On opinions, beliefs and conflict…

Opinions.

Everybody seems to have one.  The problem is that everybody tends to have a different one.

From difference breeds the basis for conflict.

Scales of conflict can range from a gentle discussion about the differences in opinion to genocide and eugenics.  Fortunately people are rarely in a position to execute the most extreme of these, but it is all bred from the same source.

We like to think that modern humans are civilized and can discuss differences in opinion in a calm and polite manner.  We like to think that when we are discussing opinions that we can do so in a receptive matter.

The truth is that not all humans are as receptive or open minded as we like to think ourselves to be.

The more we believe an opinion, the more we are likely to defend it strongly.  The more we believe an opinion, the more likely we are to take offense when somebody else speaks to the contrary.  The more belief involved, the worse the following conflict will be.

Belief is orthogonal to rationality.  They may coincide on a particular topic because the belief is formed from rational thought, but just as frequently they will not.

We believe in freedom of expression.  We frequently forget that by expressing our beliefs we can be infringing upon the beliefs of another.

Should you choose to openly infringe on the beliefs of others, you should be prepared for the conflict that follows.  If your expression of your opinions leaves a larger majority feeling threatened, you may have quite the conflict to deal with.  Morality grants you no boons in this – if you cannot hold your ground in doing so, you shouldn’t step out so far.

This is not pointing blame.  This is not assigning culpability.  This is just the nature of opinions.

This is my opinion. It’s not right, wrong, better, worse, or any combination or permutation of the above.  It is simply mine.

I respect that you have an opinion.  Maybe it aligns, maybe it doesn’t.

If you truly respect freedom of expression, you will accept this, and you will accept my reasoning.  You don’t have to like it, you don’t have to agree with it.  You are perfectly entitled to disagree.

I am just as guilty of being inflamed when people challenge this belief – more so that through challenging it, they are merely reinforcing it. It leaves me frustrated as frequently I see the other party unable to put aside their personal stake or fail to disassociate from the initial argument, and in doing so, they fail to see this pattern emerging in the conflict that follows and instead try to insist that their beliefs are somehow more “right” or “correct” than this.

Philosophy aside, understand that this is also why I don’t approve of contentious minorities trying to ‘fight the good fight’ against the better part of incumbent belief.  I may agree with your underlying opinion, but to draw such attention down upon yourself when you’re ill prepared to survive it is fool-hearty.

If you were to ask me how I would change the world – if I genuinely felt that an incumbent belief required ‘intervention’ in order to correct some injustice I would most likely answer the following:

Recruit quietly.  Work silently to your goals through example.  Unless you have overwhelming support, do not challenge the old ways head-on, but work to slowly build a foundation and support for the new without challenging the old.  Eventually you’ll find the strength required to plow forward and supplant the old, or you will have reached a point where opinion can sway to your beliefs on it’s own.

Posted in Uncategorized | Leave a comment