Sunday, March 18, 2007

lab 73 - MIDI


lab 73 - MIDI


I've written a module to read MIDI format files. I needed this because I wanted more input for my software synthesizer. I was getting bored listening to the same old track, and I haven't yet come up with any computer generated music. This seemed like a quick and easy option to get a large amount of music to listen to.

The code reads in the whole MIDI file and stores it in memory, using an ADT for the Header that contains an array of Tracks and each Track has an array of Events.

I also wrote a midi2skini command that interleaves the multiple MIDI tracks into a single stream of skini messages for the synthesizer (see earlier labs). It sorts and orders the events converting tick delta to realtime.

I've been trying this out on some bach midi files. It's been working quite nicely with the organ like sounds produced by the inferno synth.

% echo 1 > /dev/jit
  % midi2skini bwv988-aria.mid | sequencer ...

You need JIT enabled when using the sequencer.



Thursday, March 15, 2007

lab 72 - wikipedia


lab 72 - wikipedia


I've been working on modifying dict to use the Wikipedia database. I mentioned this in lab 70. So here's what I've got so far. It's not beautiful; the wiki syntax parser needs a lot of work

The general idea is I want to use acme-sac as a Wikipedia browser. But there are other reasons too, such as gaining experience of using inferno to work on some large text databases.

Acme brings some nice things to a database like Wikipedia. Because of the nature of acme you don't have to rely on people making wiki links to find other articles. You can right-select almost any text to search the index. Right-selecting single words often opens a Wikipedia disambiguation page.

If you want to get this working try following the steps below.

You need to use the latest acme-sac copy from svn. It has some fixes to support big files, including Acme.exe, otherwise none of this will work.

Download the Wikipedia database. This site will explain about Wikipedia downloads. Go here for the dump files. For the English version look for a file called something like pages-articles.xml.bz2. This file is about 2.1 GB. Download it and extract it.

This was the first snag I hit. I didn't have NTFS on my laptop drive, only FAT32, so I wasn't able to extract it into a single file. I started extracting it to smaller files and looked at creating a virtual big file using a Styx server; but before I got anywhere with that idea I bought an external hard drive, reformatted to NTFS to handle big files, and just went with the single file approach.

Extract the file somewhere and rename it or bind it to /lib/dict/wikipedia. You then need to build the /lib/dict/wpindex file.

Generate the index inside inferno,

 % dict/mkindex -d wp > rawindex

I had to step outside inferno for the next bit and used 9pm archive for the plan9 sort and awk commands. Reformat and clean the index entries using /appl/cmd/dict/canonind.awk

 % awk -F' ' -f canonind.awk rawindex 

then sort and remove carriage-returns

 % sort -u -t' ' +0f -1 +0 -1 +1n -2 < junk |
    tr -d '\r' > /lib/dict/wpindex

Hopefully, you can now type in adict -d wp in acme-sac and in the new window type some text, right-select it and a result from wikipedia will be found.


svn revision 71

Thursday, March 08, 2007

lab 71 - pruning


lab 71 - pruning


I've tried to reduce acme-sac source tree to what I consider the core. I'm cutting out dead wood to encourage new growth. Except that what counts as dead wood is highly debatable. I removed files I tended not to use, but my well traveled paths through inferno are not necessarily going to match yours. So while acme-sac still stands alone, it depends on the larger world of inferno-os for diversity and range of applications.

The smaller mass of code is intended to have less inertia. Not only can a programmer more easily understand it all, but he can also make changes system wide and so turn it to new directions. For example, if a new system library were to be imagined that could be applied to the whole limbo code set, the size of the code should not present so much resistance that an individual would not attempt it.

This reduction effort started in lab 58. The source from that became acme-sac. The recent "right-sizing" removed a lot of code. It's now down to a size I can feel comfortable with. What is the total lines of code a single developer can manage?

Number of lines of C code in acme-sac is 131,930. Compared to over 750,000 lines in the inferno-os distribution. Number of lines of Limbo code in acme-sac is 282,970. About 466,872 lines in inferno-os.

Some of the big changes I made are removing the native os tree; it does not seem relevant to acme-sac at this time. I moved all the C source under /sys, and rewrote all the mkfiles. I removed all libraries acme-sac does not depend on: libtk, libprefab, libdynld, libfreetype, libkern, libnandfs. I also removed a significant amount from /appl/lib and /module.

For a very ambitious project in code reduction, read this (PDF), from Alay Kay and associates at Viewpoints Research.

Friday, March 02, 2007

software temples


software temples


I've been listening to the seminars about long term thinking from the Long Now Foundation. I highly recommend them.

The purpose of these seminars is to encourage long term thinking. They succeed at that. For the last few weeks as I've been listening to them I've been contemplating the future of software and the history of civilization. I can't say that I've ever been interested in history. But after attempting to look forward more than 100 years it only now occurs to me what a good idea it is to look backwards far enough to see the patterns and cycles in human history.

Some of the seminars are inspiring. Danny Hillis talks about his progress in constructing a clock that will run for 10,000 years. At first it sounds insane, but after listening to him I can't help but admire him and what he's doing. The real purpose of such a scheme is to trick yourself into a different kind of thinking. Similar to the idea of forcing change in your behavior by changing your social role, such as giving a lecture will force you to learn the subject matter, building a 10,000 year clock forces a discipline of long term planning that generates new ideas which otherwise wouldn't occur.

Inevitably my thinking is grounded in the world of computing. Whatever new lights cross the sky I relate it to what I already know.

When I started programming I thought 5 years was long term. I couldn't think of an application being used longer than that. Now I've been working 10 years I have to maintain software at least that age, some of it even written by me! But I'm still not thinking long enough. Where I work the software I write must last 30 to 50 years, the typical age of the products they support. If I don't at least try then I know future maintainers will feel the pain I feel today, the pain of supporting software from extinct vendors running on ailing platforms with years of accumulated bloat.

What does it take to support software 100 years, or a 1000?

Clay Shirky in his seminar "Making Digital Durable" says that over a long enough period of time digital preservation always comes down to social issues. To keep a data format long term you need to preserve the application that interprets it. To run the application you need the operating system. To run the OS you need the hardware. The layers of abstraction don't stop. The longer into the future you want to preserve something the more layers you must deal with by somehow emulating them. They become social issues because the only way you can deal with this is to lower risk; use cheap and flexible components that have wide, popular dispersion and high degeneracy, by which he means multiple degenerate formats that represent the same information.

Would it be possible to create a long term social organization that can preserve information over millennia?

Say you as an independent agent have thought about this problem and you decide you need to coordinate with a large group. Then say many others reach the same decision, and you begin to coordinate using a system that only emerges through your coordination. By subscribing to this system you may have lowered the risk of preserving your valuable information. Only time will tell. The whole coordinating group improves their chances too if the artifacts they are preserving have some positive benefit to their survival.

When you look at history for examples of long term social organizations only a few really jump out. Religion and monarchy are two stand out examples of institutions that have lasted over 1000 years. In some cases, like the English Monarchy and the Church of England, the institutions are combined. More recent social systems I'd consider fit the same pattern are the Free Market, Science, and most recently the Free Software movement.

Given that list of systems for social organization, and given you wanted a message preserved over 1000 years, based on past performance, which would you choose?

All of these systems are an emergent phenomenon of complex adaptive agents. Stephen Lansing describes one in particular in his seminar, "Perfect Order: A Thousand Years in Bali." He describes the Water Temples in Bali, a religion that coordinates a large group of rice farms to optimize their yield. This system has remained remarkably consistent for 1000 years.

Douglas Adams discuses the same religion in his essay, "Is there an Artificial God?". The Bali system was very efficient but in the 1970s the IMF tried to get them to modernize their processes using 5 year plans, technology packets and genetically modified rice. The system improved briefly, but then because the coordination was gone among farmers, the pests crept back into the fields and yields worsened. Eventually the Balinese farmers returned to their system of Water Temples to coordinate the timing of planting and flooding their fields that had worked so well to eliminate pests.

Douglas Adams says, "It's all very well to say that basing the rice harvest on something as irrational and meaningless as a religion is stupid—they should be able to work it out more logically than that, but they might just as well say to us, 'Your culture and society works on the basis of money and that's a fiction, so why don't you get rid of it and just co-operate with each other'—we know it's not going to work!" [1]

Are the Water God's in the Balinese sense real or artificial? Is it correct to even ask the question in that way? I think it might be wrong to judge it as truth or fiction and instead observe it as an emergent phenomenon that increases the long term survival of the group. To put it in evolutionary terms. There is no absolute fitness in nature for large scale long term social systems. (The Big Here Long Now)

'There is no such thing as the "fittest" kind of organism. We can only talk about how an organism propagates in a given niche, how its life strategies have become adapted to that niche. It is no more or less fit than another kind of organism that has adapted to some other niche.' - Ursula Goodenough, "The sacred depths of nature."

In another seminar Sam Harris in "The View from the End of the World" takes a long term look at religion. He reaches the conclusion that belief in Religion is enormously maladaptive and dangerous for human survival. But I think the weight of evidence is on the side of Religion. The world religions we have today have survived a very long time, and they have survived because their subscribers have continued to prosper. Their past performance is pretty good and the prediction that they will suddenly and catastrophically fail in some way doesn't seem reasonable. In the ecology of social systems in the world there is diversity, they compete, adapt and evolve. The books of religion, the ceremony, the beliefs no matter how strange bind the group.

What has this to do with software?

When complex adaptive agents made from software become more common, and they optimize for long term survival we will begin to see software take on behaviors that bear similarities to human social systems.

If people (complex adaptive agents) care about long term software they may adopt the trappings of religion to see the long term survival of their systems.

My long term bet is that we will see a software religion emerge in the next 100 years.


[1] Aaron Swartz saw a similarity in the terminology used to promote free markets and understand economics.