Monday, December 20, 2004

lab 24 - google api

NAME

lab 24 - call google api from inferno

DESCRIPTION

I looked around to see what was available. Inferno has no WSDL or SOAP support, and limited xml. However, most of the SOAP messages are boiler plate, so we just need a program to handle the http. The program svc/webget/webget is something I hadn't noticed before. It serves a file /chan/webget which we can write GET and POST messages to and read back the result.

With this in mind I tried writing shell scripts to read and write the SOAP messages to the google api url.

I needed to debug webget a little because the POST didn't work. The changed files are attached below.

The google api download came with sample SOAP messages. I inserted my key (removed from the attached files), and placed the string PHRASE where the search phrase was to appear. The shell script runs sed on the file replacing PHRASE with the actual search phrase then writes the POST message and SOAP message to /chan/webget

     <> /chan/webget {
      phrase=$"*
      x := "{sed 's/PHRASE/' ^ 
       $phrase ^'/' doGoogleSearch.xml}
      size=${len $x}
      echo POST∎ ^ 
       $size ^ 
       ∎0∎http://api.google.com/search/beta2 ^
       ∎text/xml∎no-cache >[1=0]
      echo -n $x >[1=0]
      getlines {f = ${unquote $line}
       if { ~ ${hd $f} OK} {
        f = ${tl $f}
        read ${hd $f}
        exit
       } {exit}
      }
     }

The first line read from /chan/webget is the result status. Either

     OK length reqid type url

or

     ERROR reqid message

I use the length returned from the OK message to read the whole rest of the message to standard output.

To build, copy the limbo *.[bm] files to /svc/webget/webget and mk install. The google.sh and spell.sh files should be changed to point to your .xml soap messages. Change the key in the .xml files to your key. Then for example,

     % svc/webget/webget&
     % spell britany spars
     britney spears
     % 
     % fn g {google $* |grep URL:}
     % g plan9
     URL: http://www.cs.bell-labs.com/plan9dist/
     URL: http://www.cs.bell-labs.com/wiki/plan9/plan_9_wiki/
     URL: http://www.plan9.org/
     URL: http://www.ecf.toronto.edu/plan9/
     URL: http://www.ecf.toronto.edu/plan9/plan9faq.html
     URL: http://plan9.aichi-u.ac.jp/
     URL: http://cm.bell-labs.com/
     URL: http://www.fywss.com/plan9/plan9faq.html
     URL: http://plan9.att.com/
     URL: http://www.plan9.com/

This also works through an http proxy if you configure the proxy in /services/webget/config.

I piped the SOAP output from through a command called rss which I originally wrote to format RSS XML. This just removes the markup and makes it a little easier to read.

CONCLUSION

How easy was that! Well it could be easier. Is all that SOAP boilerplate really needed? <Insert your rant against SOAP here>

FILES

lab24.tgz

Tuesday, December 14, 2004

lab 23 - tickfs

NAME

lab 23 - annotating a file system

DESCRIPTION

i have tried out some experiments with annotating the filesystem; really, several iterations of the same idea.

my latest attempt is tickfs, which i think comes close to doug mcilroy's idea of a metadata file system.

each file served by tickfs is a binary relation, the key is usually an absolute pathname, and the attribute is defined by the name of the relation. each relation is also bi-temporal. it's a historical record of changes to each key, value pair and it determines whether a key, value pair is true at a given time.

adding metadata is simple,

      % echo ${quote i . /doc/sh.ms 'the inferno shell' } > /n/tickfs/desc
      % echo ${quote i . /doc/sh.ms 'rogger peppe' } > /n/tickfs/author

the dot indicates the current time, but any time represented as seconds since the epoch can be given.

queries are pipelines,

      % 9ls sh.ms |taq /n/tickfs/author |taq -j /n/tickfs/desc

the query uses the timestamp of the files to return the currently effective values for the attributes 'author' and 'desc'.

it relies on a disciplined user to maintain a single logical namespace and to keep it consistent with the namespace of keys within tickfs. plan9 and inferno make it easy to alter the namespace; but the discipline of arranging a clear, consistent namespace is harder.

an example that includes fossil is to bind a directory from dump over the logical namespace. the currently effective values will likely vary over time, and so obtain different results from tickfs.

             % 9ls /doc |taq /n/tickfs/desc
             % bind /n/dump/2004/0101/doc /doc
             % 9ls /doc |taq /n/tickfs/desc

also, i have tried indexing file contents within tickfs, so combinations of keyword search and named descriptors can be used

      % search bug |taq /n/tickfs/author |grep caerwyn |taq /n/tickfs/desc

tickfs runs on inferno, but lookfs, which is an earlier iteration using libstyx, is very similar and should run on plan9.

FILES

The source for this lab is in the archive on my homepage.

lab 22 - news aggregator

NAME

lab 22 - news aggregator

DESCRIPTION

Rssget downloads an RSS xml file indexes it into tickfs. Then using tickquery, which should be copied into /dis/svc/httpd you can query the RSS aggregator using address http://localhost/magic/tickquery.

% mkdir rss
% cd rss
% mount -c {tickfs} /n/rss
% touch /n/rss/index /n/rss/desc /n/rss/chan /n/rss/title
% svc/httpd/httpd&

use localcron to call rssget periodically on your favourite news feeds.

tickquery looks if the first keyword begins with 'date:' the rest of the word is treated as a rng(1) and restricts the results to within the date range.

For example, to search for RSS entries since 1st of the current month mentioning google,

date:1, google

or show me all entries collected on the 21st of the current month

date:21

Here is a demo of this service. more interesting examples I have yet to work out. it gets more interesting given temporal indexes of mail, mailing lists, rss feeds, stock ticker, visited web pages, images and annotations of local files and links. then a context around a particular event can be constructed...

FILES

The source for this lab is in the archive on my homepage.

lab 21 - popi

NAME

lab 21 - popi

DESCRIPTION

Ported popi from the book Beyond Photography - The Digital Darkroom which is online and many of the examples from the book should work.

I took the C code, translated it to limbo, then tried to add the trig functions and polar coordinates. I referenced Rich Burridge's version of popi to get ideas while adding the extra features.

See the guide file here for many of the examples from chapter 3

The combination of the program and the online text makes a good example of an active essay. While reading the text the user can type in the formulas and edit them to explore further into the tool and it's possibilites. It enhances reading of the text and understanding of the tool.

CONCLUSION

This took longer to debug than I expected and I still have doubts about parts of it. The polar coordinates when used as a target do not work, rounding errors leave gaps in the matrix.

I was going to implement conway's life as a special function and a method for repeating a transformation so life could be run on an imported graphic. There are endless possibilities here.

I would like to implement a similar tool for sound wave editing. I could then plug the resulting waves into signalfs for looping and components of instruments.

FILES

,p> popi.b

lab 20 - libmux

NAME

lab 20 - libmux in limbo

DESCRIPTION

To learn about mux in detail I tried to implement libmux in limbo. It actually resembles the mux from libventi more than libmux. They are pretty similar.

I was going to do a detailed description of the mux in devmnt but it turns out this has already been done here by nemo.

When I get this working it can be applied to the venti lib for session rpc, to build a libfs for direct client 9p interaction, bypassing devmnt, and for application protocol muxing, say for sam.

This is a partial implementation. I didn't complete send, recv, gettag and settag functions, so this has not been tested. I'm just posting it here now so I don't lose track of it. I'll come back to it later, maybe writing more for the Venti library or libfs. This is more an excercise to understand the workings of libmux and venti's session rpc.

FILES

mux.b mux.m

Monday, December 06, 2004

lab 19 - wm

NAME
     lab 19 - understanding wmclient

DESCRIPTION
     In trying to understand wm(1) and its supporting
     libraries wmclient(2), tkclient(2), and titlebar,
     I modified them to resemble (if you squint) rio. I
     haven't adopted rio's menus or button controls.
     Here's a screenshot.

     I began with a copy of wmclient. I wanted to peel
     away the layers between the application and wm. I
     removed the dependency on titlebar and created a
     border that would manage the window. I removed
     whatever else I thought was unneccessary. I ended 
     up with myclient which talks directly to wm. 

     Window management is based on button clicks on the
     border. Button 1 moves , button 2 reshapes, and
     button 3 deletes the window.

     I liked the window management so I wanted to use
     that for all windows. To copy the border changes
     back to wmclient.b was easy.

     But most windows programs in inferno use the
     tkclient. For that I needed to modify the
     titlebar. I created four buttons which fill the
     four edges of the rectangle. Each button is bound
     to the three buttons which control the window
     management.

     I made the color of the borders different for
     tkclients and wmclients. There aren't many
     wmclients: acme and wm.

     The borders  change color when the window gains or
     loses keyboard focus, but it must accept kbd input
     for this to happen.

     wmclient creates a new window with a transparent
     image, so it's filled with whatever is beneath it.
     This leads to an obvious tool grab to capture and
     image of a part of the desktop. Reshape the window
     to grab the part of the picture then click button
     1 in the image and it writes it to the file named
     on the command line to grab.


CONCLUSION
     I like the results. I'll probably use this going
     forward and see what other mouse bindings I like.
     But this leads me to modify the tk for commands
     like sh etc., tweaking everything.

     For some reason wm/sh calls a Tk command which
     refers back to the titlebar assuming the .Wm_title
     exists. So I included a modified wm/sh.

     I'm tempted to make more mods to wm/sh to remove
     the menus and make it more like and acme window,
     that is, middle click sends, which is really the
     only menu option I ever use.

FILES
     grab.b myclient.b sh.b titlebar.b titlebar.m

     tkclient.b wmclient.b mkfile

Wednesday, December 01, 2004

lab 18 - mux

NAME
     lab 18 - mux

DESCRIPTION
     Much of my recent reading has led me back to
     various implementations of a multiplexer. This is
     an important and powerful abstraction and I want
     to understand it better. I know very little about
     it now. I'd like to implement one in limbo, for
     example to multiplex the link between sam and
     samterm as suggested in the protium paper. Hence
     my interest in sam in the earlier lab.

     Here is a some of the things I've been reading
     recently about multiplexers. See if you see a
     pattern. J. Hickey's Master thesis at MIT
     implements mux for plan9 which puts a 9p
     multiplexer into user space and provides
     asynchonous IO for user threads.

     Search 9fans for multiplexing where there is some
     interesting discussion on the use of multiplexers
     in plan9.

     For example, an idea by rog and russ's response

     And futher proposals by russ proposal for mux, 
     local 9p multiplexing and remote 9p multiplexing.

     See also an implementation of a generic rpc
     multiplexer, libmux, in plan9port

     The various window systems by Rob Pike, mux, 8½,
     and rio are multiplexers for their environment,
     the screen, mouse and keyboard.

     The spree game engine is a multiplexer at the
     application leve for the object hierarchy managed
     by the server.

     And in inferno /emu/port/devmnt.c is the
     multiplexer for 9p.

     In the original exokernel paper the authors argued
     the principal function of an operating system is
     to multiplex access to hardware, and should do
     nothing else. Multiplexers are vital to providing
     9p service and in protium the authors argue they
     are a vital piece of infrastructure for
     distributed applications. 

     In summary, this is something I need to learn more
     about. I'll proceed in these labs with studying
     the code to libmux and devmnt and try to implement
     the mux for the sam session.

                    

Monday, November 29, 2004

lab 17 - samterm

NAME

lab 17 - fixup inferno samterm

DESCRIPTION

There's an inferno version of samterm intended for connection to remote sam running on unix/nt/plan9. It looks like it hasn't been updated for the latest tkclient interface. Get it working.

Simple edits to add samtk.b:/^tkcmds/ and samtk.b:/^cmd/

One of the bigger changes I made is to spawn a new proc samtk.b:/^tkproc/ for each file to handle events from the toplevel chan. This was the quickest way I could think of doing it. It might be more correct to create arrays of channels like those already in samterm.m:/Context:/ but that seemed like a lot more work.

I use the idiom of returning the pid samtk.b:711 so I can kill the proc when the file is removed, otherwise the windows would be left hanging around. I added the pid to samterm.m:/pid/

Menus were placed wrongly when middle or right clicking in an edit window. Menu(9) uses post command and takes x y coords. Read bind(9) it should take %X %Y so placement is relative to system screen instead of edit window. samtk.b:51,50

Cut doesn't work. Or rather, paste doesn't work once I've done a cut.

CONCLUSION

Mostly works. I tried some simple edits, opening and closing files. Mount cmd(3) from different machines to get remote connections to sam. All files names are relative to the remote sam of course, not the inferno namespace.

FILES

mkfile sam.b samstub.b samstub.m samterm.m samtk.b samtk.m

Thursday, November 25, 2004

lab 16 - canto

NAME

lab 16 - set up a styx service for active essays

DESCRIPTION

I created a readonly unauthenticated styx service for readers of IPN, who have inferno installed, and want to read the lab notes in acme and execute the embedded scripts. From here on I'll use the term canto for active essays authored and "read" (rather to execute or chant) in the Inferno environment.

This entry in my notebook is my first attempt of the new style. It describes the files and formats of the service.

Mount the service and open acme on the file for this lab.

mkdir /n/canto
mount -A tcp!canto.hopto.org!6700 /n/canto
acme -c1 /n/canto/16/index.txt

Each canto is in a numbered directory containing the following files:

text
the raw contents of the notes in man(6) format
post.html
the text file converted to html using man2html for sending to the blog.
guide
the file containing executable code within acme (see below)
index.txt
the main contents, usually text formatted using man(1).

Instead of embedding possibly long scripts within the main text, they are in the guide file, and at the appropriate point in the text are referenced. Right click on the link and then middle click on the selected text.

guide:/^pi/+1

While reading a canto at least three frames will be open; the index.txt, the guide file containing the executable expressions, and the Errors file containing the output from the scripts. I prefer to run acme with one column which fills half the screen and leave the other half of the screen for windows I open such as shell windows, debugger, task manager, etc.

guide:/^man/+1

The canto directory also contains limbo source code, possibly a mkfile and the executable dis files. The canto will contain links to the source instead of directly embedding samples in the text. E.g.

man2html.b:1

This will open other windows in acme and jump to the point in the code being discussed in the text.

Monday, November 22, 2004

active essay

in the second post to this blog i talked about wanting this lab notebook to be more interactive, with something like mathematica notebook or squeak project in mind. after reading more about squeak, and thinking again of what i want, i'll try and describe how i'll proceed with this notebook.

Alan Kay and Mitchel Resnick have used the term "active essay" to describe a publication that includes computational objects. Resnick describes them as "new forms of narrative expression, in which manipulable computational objects are integrated with text, graphics, and video." Kay in particular stresses the point that the objects in the essay can be deconstructed to learn how to put them together again. The squeak environment uses many visual elements targetted at children for building simulations of ideas they are learning in school--a constructionist view of education for math and sciences.

Kay believes the most important use for modern inexpensive personal computers is that "they form a new kind of reading and writing medium that allows some of the most important powerful ideas to be discussed and played with and learned than any book."

See the essays on education at squeakland.

here are some guiding principles for my notebook (blatantly stolen from other sources.)

  1. express ideas with narrative and programs in ways that are not possible with traditional media.
  2. all ideas can be expressed symbolically. this is the unifying idea of Mathematica. it is so obvious it doesn't seem worth mentioning. but it will require some work to be able to express everything we need to do in an active essay. the mathematica notebook is an expression in the mathematica language.
  3. every object can be deconstructed and reconstructed. provide the source for everything so it can be picked apart, including pictures, sounds, programs. "authoring is always on" -Kay

so the format of the lab notebook will now be presented as if the file is open in acme. text can be selected and executed. i'll make use of references to code using the acme Look format. i'll make use of plumbing to open programs in the system.

acme is a boon for the active essay. already emails on 9fans essentially contain this active element, simple one line shell scripts that a user of acme mail can select with middle mouse button to execute.

inferno already meets most of the above criteria. although, it is not always easy to express some actions in shell, such as window placement and manipulation. as part of the lab i'll be coming across these issues and hopefully solving them to make the authoring simpler.

essential to the idea of the essay is its means of distribution. i'll likely make a styx service available so the lab notes can be mounted and explored. but the simplicity of the format means the essays can be mailed, or just uploaded to the blog, maybe as a tar file with the associated code and files. i'll continue to convert the notes to html and post them to this blog.

related work

IETMs (Interactive Electronic Technical Manuals). the standards for IETMs are used heavily by the military. they include DTDs for SGML or XML documents. there are different levels of implementation of these concepts from static documents, to hyperlinked, to documents that contain computational elements. the focus is on guiding a technician through maintenance procedures rather than on exploring and creating ideas.

WIKIs in general offer the "authoring is always on" principle but do not offer the computational elements. However, the Plan 9 wiki, when edited within acme, does offer the embedded scripts. And then when combined with a mounted file server such as the plan9 sources it becomes a distribution mechanism for active essays.

look at the annotated source by nemo for the plan9 kernel. it contains all the acme references so that it can be read and plumbed to jump to the correct line while reading the source.

Friday, November 19, 2004

squeak

been very busy of late (new baby!) and had no time for inferno programming after work. I hope to get back into some kind of routine within the next month.

TODO list

  • clean up dsp module and write more instruments
  • try out the alphabet typesets with dsp, tickfs, grids
  • write a general btree module for indexes
  • port plan9 sort to inferno
  • ...

Aside from inferno I've been playing with the squeak smalltalk environment. I've been very impressed with this. The morphic interface and etoys scripting is very cool. It got me thinking about how these and other ideas could be transferred to inferno, and then I got into a kind of thought paralysis--I got too confused.

another interesting paper I read yesterday was protium an infrastructure for partitioning applications. i wonder if some ideas for session persistence could be explored within inferno.

i downloaded panda3d a 3d game and simulation engine with python scripting. wouldn't it be cool to link this to inferno as a new interface? the squeak environment mentioned above has moved in a similar direction with croquet.

Monday, October 25, 2004

lab 15 - cluster file

NAME

lab 15 - cluster file

DESCRIPTION

I'm still curious about layering of filesystems. Ds(3) was an example I looked at before writing signalfs. Another example was cryptfs (lab 2). Ds currently resides in /os/port/devds.c. A while ago I moved it into /emu/port/devds.c but hadn't made any use of it. I have wanted to use it for a rudimentary cluster file system, so I ported ds to limbo to play with distributed files and further explore files that are built from layering of other file systems.

I translated the C code and used styxservers to manage the simple, one level namespace. Here's some testing. This is really as much as I've tested it so far.

% for (i in `{seq 0 9}) {zeros -v $i 1024 8 > /tmp/chunk ^$i}
% echo cat c1 /tmp/chunk2 /tmp/chunk3 >ds/ctl
% echo cat c0 /tmp/chunk0 /tmp/chunk1 >ds/ctl
% echo mirror m0 ds/c0 ds/c1 > ds/ctl
% cat ds/ctl
cat c1 /tmp/chunk2 /tmp/chunk3
cat c0 /tmp/chunk0 /tmp/chunk1
mirror m0 ds/c0 ds/c1
% ls -l ds
--rw-rw-rw- M 36 caerwyn caerwyn 16384 Dec 31  1969 ds/c0
--rw-rw-rw- M 36 caerwyn caerwyn 16384 Dec 31  1969 ds/c1
--rw-rw-rw- M 36 caerwyn caerwyn     0 Dec 31  1969 ds/ctl
--rw-rw-rw- M 36 caerwyn caerwyn 16384 Dec 31  1969 ds/m0
% cat ds/c0 > /tmp/t1
% cat /tmp/t1 > ds/m0
% cmp ds/c0 ds/c1

I read the googlefs paper again today. With that in mind a cluster file system could be pieced together using some inferno components.

A kfs(4), or any standard fs, represents the master namespace. All files contain only a ds configuration--the chunk IDs and partioning info. All the inferno nodes write to the master registry (4) the chunks they will serve. Ds grows files by reading chunk id's from a master process that uses the registry to allocate new chunks. A client must navigate the master namespace to the file containing the ds configuration and mount ds in it's namespace. Then it has a distributed file which communicates directly with the nodes storing the disks in the network.

CONCLUSION

Not much thought given to the deeper problems. How would multiple writers on different clients append to the same file? Here is the source for dsfs.b

Thursday, October 21, 2004

lab 14 - map reduce

NAME

lab 14 - map reduce functional grid programming

DESCRIPTION

I read about Google's MapReduce from Rob Pike's interview at slashdot. At the same time I've been studying Inferno's alphabet-grid(1) and am wondering if I can implement map reduce using alphabet.

Here's an imaginary example

          % - {rng , | remote |
           mapreduce "{tq -1rm /n/tick/tick.bt} "{tock } |
           /create /tmp/result
          }

Suppose that tick.bt is a log of time spent on tasks where each record is the timestamp, task and number of seconds spent on the task that instance. Rng produces 1 or more date ranges. Remote converts type /fd to an endpoint. Mapreduce will then split a date range, such as one year, into M smaller date ranges. For each subrange it calls rexec passing it the address of an available node, the subrange and map function as parameters.

The output from all the map functions is directed to R endpoints. The R parition function could be hash(key) mod R as suggested in the paper. Then mapreduce rexec's a reduce worker, which reads in all the data from the endpoint, sorts it, and for each key calls the reduce function with the key and list of values (or /fd) as parameter. In this example tock, the reduce function, sums all the time values for a task and outputs the total.

I've made the example specific to tickfs and the use of triads merely because I already have these tools and makes it easier for me to grasp. The google paper uses key, value pairs. I'm ignoring all the other factors they consider, such as fault tolerance, locality, and much else.

Here's another example. In the distribution on my homepage I include a command nsearch for searching a tickfs index. The command is given a list of keywords. Given the first keyword, which might be a date range, it builds an initial set of keys. It then partitions this set among a fixed number of threads. Each thread test the record coming in on a channel against the index and the search term given as parameter to the thread. The reduce function would be an identity function, simply passing through it's input. This is a map, filter, reduce pipeline. Alphabet seems to provide the tools to express this whole query and more on the command line, including distributing the processing among nodes.

The implementation needs somewhere to lookup the available list of nodes. Administering all the nodes would need some fancy fs that managed the status of all executing workers. I'd keep this to an absolute minimum for now.

CONLUSION

This all sounds very promising but I don't know how to implement it yet. Here are some more notes while I think this through.

The revelation for me is the importance of functional programming to distributed computing. It wasn't long ago (lab 1) that I discovered limbo shell supported functional programming. Alphabet takes this to the next level by defining types. Alphabet-grid provides the connection between processing modules on distributed nodes. Altogether it provides a framework for distributed computing I'm still coming to grips with. It is a different way of thinking about computing than I am used to.

REFERENCES

MapReduce Interview

Monday, October 18, 2004

lab 13 - flute

NAME

lab 13 - implement the flute instrument from STK.

DESCRIPTION

I implemented more of the STK library but his time as a straight forward translation to a limbo module. Much of the protected classes and filters are in dsp.b as ADTs. They all share a similar interface that includes functions mk for building the object and tick for processing the next sample.

The instruments are generally larger to implement but follow the same interface. They can be plugged into a signal module and then read and controlled from within signalfs.

I've included a few simple modules that can be used to start a new instrument. I also tried to implement the more complicated Flute. It's close, but still doesn't sound right. It all needs a lot more debugging.

To test the flute,

% signalfs -a /mnt/dsp
% echo add flute.dis flute > /mnt/dsp/ctl
% sequencer /mnt/dsp/flute < bach.ski > /dev/audio

Sequencer has some small improvements to handle more voices and open them as needed. The number of voices does not need to be limited but it gets bogged down with four.

CONCLUSION

Still more work to be done, but I'm almost at the point where I can start building a base set of hopefully interesting instruments.

FILES

The latest dsp.b dsp.m flute.b sequencer.b signal.m signalfs.b simple0.b simple1.b simple2.b

Wednesday, October 13, 2004

lab 13 - sound library

No code to post tonight because it's unfinished.

I'm converting all of STK to limbo, but not directly into signalfs. I'm creating a module that will contain all the sound sources, filters, and effects in the STK, with one ADT for each sound. This can then be used by signalfs to serve a file, which can be a combination of any of the ADTs, or by any other limbo app.

Rog has suggested an alternative application using the shell alphabet. I will try this once the library is written.

Rog pointed out how inefficient signalfs is in its current form. I agree; the performance is terrible, which makes it compleletly unusable for realtime sound support. This re-implementation will improve performance. But any hardcore DSP programmer is only likely to snicker at our attempt to implement DSP in limbo. At the end of the day I'm doing this to create a framework for ease of experimenting with DSP, not to create a sound system that will out perform all others. That is the tradeoff I make by writing this in limbo.

Another possible implementation is to create a C library module or device compiled into emu. This would perform well but be less malleable. I'd rather code in limbo.

Monday, October 11, 2004

lab 12 - oscilloscope

NAME

lab 12 - implement an oscilloscope for signals from signalfs.

DESCRIPTION

I implemented an oscilloscope called scope to view the signals produced by signalfs, or other PCM data such as an iaf file stripped of it's header.

% scope < bach.raw > /dev/null

It writes it's input to it's output, but it doesn't sound very good if directed to /dev/audio. It writes small blocks of samples with small delays between writes, making it sound very choppy.

Scope tries to draw 25 frames a second, getting a tick from a timer, and reads 1/25th of a second of samples from the input, then draws it on a Tk panel.

This is might be useful when recording input from a microphone

% scope < /dev/audio > out.raw

It takes as parameter the sample rate and number of channels, stereo or mono.

CONCLUSION

Not being able to listen and see the waveform at the same time makes it less useful than I hoped. How do I keep in sync the visual and audio stream?

I'd like to present a similar graph using the FFT, but I keep getting errors from the fft module. It correctly does the transform but isn't able to permute the numbers into normal order (I think, I do not know enough about this.) Commenting out the "optional code" in /appl/math/fft.b seemed to make it work, at least make it not exit.

FILES

scope.b

Thursday, October 07, 2004

lab 11 - microsoft

I spent the time this evening downloading the Microsoft C/C++ toolkit and the SDK. I have only built emu for linux and plan9 so far. So it's time to start building an XP version. I also expect to be modifying the audio driver to add some more advanced features, such as 8 channels.

While waiting for the downloads I found this site about audio networking interesting. A DTMF decoder would be a nice lab to do some night.

I tried compiling the inferno distribution for XP. I'm missing LIB.exe. I used link /lib instead. Got most of it compiled, including emu.

Wednesday, October 06, 2004

lab 10 - delay

NAME

lab 10 - delay line

DESCRIPTION

I am continuing to add signal modules to signalfs copying the implementations from stk. Today I'm working on the delay line, and whatever else I can implement in two hours.

The delay line does not fit the model of signals I have created so far. From the STK it looks like it is used more as a utilitly class than a standalone filter. Its used by the echo class which actually does the mix of the current input with the delayed input. I could of course do the same thing and have delay as functions with the dsp module. Trying to use the delay, or echo, brings up a number of issues.

How am I going to stack multiple filters ontop one another and still be able to ctrl each one independently? To access to each ctl file I'd need to know the conversation number. This might be tricky to find out if I have multiple instruments each being built from many modules.

I want to alter the effect during playback independently of the instrument being played. But I'm not sure how to fit it in with a simple instrument. Where in the stack should it go? And how will I control it if it's placed under the instrument?

This goes back to the problem of needing some kind of patch bay. Given a particular instrument we need to now all the effects tied in to it. Then we want to write to the ctl file of any of them, not via the instrument but directly, and alter the effect. We need to remove the exclusive access condition on the ctl, although we could place it on data instead.

If I didn't do this I'd need a naming convention within the ctl file that was at the end of the filter pipeline. But that is ridiculous because what else am I using a fs for.

Therefore, If I put the, say, echo filter in front of the instrument, I still send note events to the instrument, but read sound back from the echo data file.

Is the sequencer going to be able to manage all this? The skini language may have to include naming of instruments using filenames. That is, events are sent directed to specified ctl files (filters, instruments) but audio data is read from only one, the one at the end of the pipeline (is pipeline the right term here? filter chain, patch? sink?).

I need to specify the sequencer language and a means for retrieving all the conversation directories for a pipeline before going further.

FILES

delay.b delayl.b

Tuesday, October 05, 2004

lab 9 - tmap draw image

NAME

lab 9 - create tmap image directly instead of using Tk

DESCRIPTION

I modified tmap to use draw-image(2) because using tk(2) commands to create so many rectangles was slow and used a lot of main memory.

The changes were straight forward. The Tk cmd

TkCmd(t, sprint(".fc.c create rectangle %.1f %.1f %.1f %.1f "
 + " -fill #%s -outline black -width %.2f",
 r.x, r.y, r.x+r.w, r.y+r.h, dc[depth],  0.5));

becomes

rect := Rect((int r.x, int r.y), (int(r.x+r.w), int(r.y+r.h)));
img.draw(rect, t.display.color(dc[depth]), nil, (0,0));
img.border(rect, 1, t.display.black, (0,0));

where img is the global Image destination. I then update the image on the canvas

tk->putimage(t, "tmap", img, nil);
tk->cmd(t, ".fc.c coords tmap 0 0; update");

tmap is the name of the image previously created with image(9) and associated with canvas(9).

CONCLUSION

It works much faster and uses much less main memory. It still uses a lot of heap memory. Tk is nice but has it's limits. It works well for creating and removing the highlight rectangles. But it isn't appropriate for creating the main image, because I don't want to keep track of every rectangle image, and there can be so many entries. The largest I tried was about 300,000 entries. I'd like to try creating a large image with 1 million entries.

I'm still improving this because I intend to use it as some kind of radar for monitoring file system activity.

FILES

tmap.b treemap.b treemap.m The screenshot is of strip view of the Inferno distribution.

music spree

NAME

Music band spree client and engine

DESCRIPTION

I have been reading through spree code and wondering what engines I could write to learn more about this type of filesystem. Here is one idea that incorporates signalfs.

The aim is to create a distributed electronic music band. Each player is a client to spree and can create instrument objects and play notes on these instruments. The other players will have the instruments and sounds replicated locally by their spree client; and similarly instruments and notes they play are heard by the other musicians. Each spree client runs a separate signalfs but with a common set of signal modules.

The spree action commands will be SKINI with extra commands for creating and configuring the instruments. The spree engine manages the shared configuration of instruments. Note events are broadcast to all players. The spree client is therefore a type of sequencer, using timing and note events from the engine to trigger reading sound from the signalfs and writing to /dev/audio

Using OpenAL, or implementing it in signalfs, each instrument and player is placed in 3D space. The engine manages the positioning of objects, which now includes listeners, and each player hears the sound relative to their position in the virtual room. This requires a surround sound speaker system.

If this is implemented successfully it not be too difficult to generalize. Surely if we get the synchronization working for audio we could do the same for visual object simulations.

These ideas are influenced by my recent reading about Croquet. I'd like to see the implementation of the TeaTime protocol. The time synchronization seems the most challenging aspect of this idea.

Friday, October 01, 2004

corrections: lab 8

Here is a new tmap.b with the memory leak removed. I just needed to call Tk cmd delete all on the canvas to remove all the old rectangles.

And finally a screenshot of tmap using output from du -a / on the root of the Inferno distribution. You can see a file and it's path highlighted in red.

I've played about with the coloring a bit, but haven't found anything I really liked. The current scheme is for colors to represent depth. It goes from dark blue to light blue, through greens and yellows and ends on red for the deepest nodes.

Thursday, September 30, 2004

lab 8 - treemap

NAME

lab 8 - small update to treemap.

DESCRIPTION

Tmap is a small util that takes the output of du and presents a treemap using algorithms from UMD HCILab tmap

To make it a bit more useful I want to zoom into the treemap and zoom out again. Button one selects a rectangle highlights it and it's path and presents information about the path and file sizes. Button two zooms one level into the selected area, where the mouse clicked. Button three zooms out one level.

This was easy to add using Tk. I bind a command to the mouse events and send a message down a channel. On receiving the message I change the root of the tree, regenerate the layout and redraw the rectangles.

CONCLUSION

I find this a useful tool to find patterns in disk usage, such as large or duplicate files. Often the file hierachies are very deep, and so drilling down interactively is great.

I should re-implement this by drawing directly on an image instead of using Tk. I seem to be running out of memory using Tk because I am not keeping track of the Tk tags when creating new rectangles, and I'm creating so many of them. It may also be faster, but I don't know.

There is a bug in the squarified view where the directory is rendered as if it is a file, using the totalsize of all sub elements. I haven't found what causes this. The other views seem okay.

I think the treemap is a beautiful view of the filesystem. I'd like to put it to more use.

In a collaborative environment the treemap of a filesystem is a shared map of the system. The rectangles, which represent files, are color coded for the users who have them open. This could fit in as an engine to spree, or collab. We'd need to monitor all styx messages, similar to Styxmon.

The prog device shows all files open and process activity. Processes might also be presented as a treemap.

The treemap could then be made more interactive. Files are plumbed, to open and edit them. The files are annotated and new annotations are seen by all users of the treemap.

This looks like a good excuse to play with spree. I'm thinking along the lines of VisitorVille and Plan9's faces.

FILES

tmap.b treemap.b treemap.m

REFERENCES

UMD HCI Lab HCIL

Tuesday, September 28, 2004

lab 7 - sequencer

NAME

lab 7 - sequencer; create a simple sequencer that can play back a subset of SKINI messages.

SETUP

Inferno 4th Edition release 20040830.

DESCRIPTION

2004/0928 20:37 SKINI is the sequencing language from the STK. It is a readable form of MIDI, and was designed to be "extensable and hackable"; all of which make it ideally suited to this application and Inferno. Here is a brief example

// Measure number 1 =0
NoteOn    0.416667     2   72   64
NoteOff   0.208333     2   72   64
NoteOn    0            2   71   64
NoteOff   0.208333     2   71   64
NoteOn    0            2   72   64
NoteOff   0.416667     2   72   64
NoteOn    0            2   67   64
NoteOff   0.416667     2   67   64

There is one command per line. The line begins with command name followed by parameters separated by space or tabs. The second parameter is always the time delta. For the NoteOn command, the third argument is channel (or voice), fourth is midi pitch, and fifth if velocity (I guess! I'm ignoring it for now).

These SKINI messages are interpreted by sequencer, which sends messages to one or more instruments and reads back audio data.

I created a basic instrument module, which should be used as a template for later instruments, called simple. It uses the adsr and wave modules.

The wave module I have described previously. Adsr is a standard Attack, Decay, Sustain, Release envelope. I copied the implementation from the STK.

Simple and sequencer use a new interface to assist in using other signal sources.

Sig: adt {
 ctl: ref Sys->FD;
 data: ref Sys->FD;
  
 open: fn(s: string): ref Sig;
 read: fn(s: self ref Sig, nsamples: int): array of real;
 readbytes: fn(s: self ref Sig, nbyte: int): array of byte;
};

This will evolve. It is currently part of the dsp module.

The source signals are opened by init. I made a small modification to signalfs.b moving the init call from signalfs.b:/serve to signalfs.b:/requestproc to avoid deadlock.

I created /mnt/dsp for mounting signalfs and /mnt/dsp/raw because I am hardcoding the names of some signal sources and raw sound files. The raw files from the STK should be bound or copied to /mnt/dsp/raw. Therefore start as

% bind /n/j/stk-4.1.3/rawwaves /mnt/dsp/raw
% signalfs -a /mnt/dsp

Setup the modules and /dev/audio

% bind -a '#A' /dev
% echo rate 22050 > /dev/audioctl
% echo chans 1 > /dev/audioctl
% echo add wave.dis wave > /mnt/dsp/ctl
% echo add adsr.dis adsr > /mnt/dsp/ctl
% echo add simple.dis simple > /mnt/dsp/ctl

Run a SKINI file.

% sequencer /mnt/dsp/simple < bachfugue.ski > /dev/audio

The sequencer assumes 4 voices currently. It's very basic; just for testing while creating new instruments. It will most surely be rewritten.

CONCLUSION

Here are the latest. adsr.b bachfugue.ski dsp.b dsp.m sequencer.b signal.m signalfs.b simple.b wave.b

The SKINI language has more commands and features than implemented here.

Again, it is slow. I should buffer writes to the /dev/audio, maybe a few seconds worth, so the audio sounds smooth. Otherwise, I need to write to a file first then stream the file to /dev/audio. However, It's enough for testing while creating some of the more complex instruments from the STK.

The sequencer could be a module of signalfs like any other. Reads of the data return the audio data. The ctl file is a synthetic file which could be edited within any editor. But this is a slightly different interface than other signals. An alternative is to use the skini file as a source file just like the raw files for the wave module. The sequencer module then represents patterns, which can be combined, looped, and sequenced just like any other signal.

DAY DREAMING

I can have a grid generating the sounds. What is possible with unlimited cpu power? Using a grid we should be able to create, in realtime, a very complex syntesized soundscape. Could we use the plan9 grid for this? Run emu on each node, serve a signalfs, bind them all into a local namespace, then generate the patterns. 8 channel, 24 bit, unlimited voice and polyphony.

REFERENCES

STK is now a link on the side bar.

Wednesday, September 22, 2004

lab 6 - signal filter

NAME

lab 6 - signal modules; implement filter (again) for signalfs and test the recursiveness of the file server.

SETUP

Inferno 4th Edition 20040830. See also labs 4 an 5.

DESCRIPTION

2004/0922 21:24 Changed filter.b to use the clone interface. The ctl file accepts a source command which takes a filename or another signal source which may be served by the same signalfs

% echo source /usr/caerwyn/lab/6/mnt/wave > mnt/filter/2/ctl

The filter module then opens ctl reads the number of the conversation directory and then opens data. The filter is then responsible for setup of the signal source. It should probably pass through commands written to ctl that it does not understand, so the wave can be controlled through the filter ctl.

I made a small change to signalfs.b to not fork the namespace so the mount of the server affects the current process and the filter module is able to access the wave

176c178
<  pidc <-= sys->pctl(0, nil);
---
>  pidc <-= sys->pctl(Sys->FORKNS, nil);

CONCLUSION

The recursion (if that's the right term) works. Filter reads data from wave served by the same file server; or it could be a different file server running remotely.

This was a quick lab session to resolve my doubts about signalfs. Need to really start implementing all the filters. Especially those that combine multiple sources.

FILES

dsp.b dsp.m envelope.b filter.b noise.b signal.m signalfs.b wave.b

Tuesday, September 21, 2004

lab 5 - signalfs v2

NAME

lab 5 - signalfs version 2; generalize and enhance the signalfs still further. Create a server that can load all the signal models, and each module has a connection directory per conversation.

SETUP

Inferno 4th Edition 20040830.

DESCRIPTION

The directory hierarchy presented by signalfs is now similar to ip(3).

/ctl
/module/clone
/module/n
/module/n/ctl
/module/n/data

One more requirement is that the server must be recursive, a signal module must be able to open other files served by the same signalfs. (The ds (3) device does this.)

To illustrate, say we implement a module to return a Fibonacci number. I use the clone mechanism so each client has it's own connection to the module. To read the fifth Fibonacci number

          {
           d=/mnt/modfs/`{read 10}
           echo 5 >[1=0]
           read 10 < $d/data 
          }<> /mnt/modfs/clone

The module itself uses the ctl message to determine whether it should open a connection to clone and read the Fibonacci number for the number - 1. If the ctl number is 0 it returns 1, ending the recursion.

The module is a single function. The parameters are controlled by the ctl file, and the result is read from data.

The fileserver framework manages the clone and naming and loading of modules. I used wmexport.b as the starting point for building a framework.

2004/0918 21:28 The ip(3) device comes close to what I want. A directory for each proto (module); a clone below that, and a numbered directory for each instance. The numbered directory has ctl and data. The numbered directories aren't removed but keep state variable tracking whether in use.

What if we get a read on the data file from a process other than the opening process? We want to deny this.

2004/0919 16:51 Writing and debugging.

2004/0919 20:08 Is this something like spree, where the loaded module is an engine?

2004/0919 22:41 Given the layout of signalfs a filter is given the directory for the source module. The filter opens the clone file to get a connection. It then has exclusive access to that module. The filter exposes the same interface, so I could create multiple connections to that filter. But what if I want to alter the sinewave that the filter is reading from? Do I want shared write access to the sine ctl file? I'd need to know the connection the filter was reading from. No. The filter is in complete control of it's source.

2004/0921 22:01 Writeup and debugging.

Signalfs now knows nothing of the format of the signal. The signal is responsible for converting arrays of real to bytes. The signal interface has changed

          Signal: module {
           configstr: string;

           init: fn(args: list of string);
           config: fn(s: string): string;
           read: fn(n: int): array of byte;
          };

Config returns an error string or nil if successful. Here's an example setup

          % mkdir mnt
          % signalfs mnt
          % echo add /usr/caerwyn/lab/5/wave.dis wave > mnt/ctl
          % lc mnt
          ctl   wave/
          % <> mnt/wave/clone {
           d=mnt/wave/`{read 10}
           echo file /usr/caerwyn/lab/3/sinewave.raw >[1=0]
           read 8 < $d/data | pcm
          }
          19788
          16364
          12685
          8807

CONCLUSION

Here is the current version. signalfs.b, signal.m and the sinewave module again wave.b

Once this is debugged, I've reached the point where I can write all the signal modules. I still have no real plan for a sequencer. It may end up being shell script. I haven't tested the recursiveness yet.

I could have implemented signals using file2chan (2) except I am supporting the more complicated interface of clone, ctl and data. I hope it will be worth it having all the modules organized together under one server.

Tickfs might be better adapted to this interface. The ctl message for lookup are written to the ctl file, and I can control permission on the data file.

At some point I should provide a brief taxonomy of fileservers. E.g., as represented by ip(3), ds(3), file2chan, kfs (traditional), env(3), etc. Traditional file access from underlying storage (kfs, tarfs); Conversational object access (draw); Shared object access to virtual storage (env); Device interface (eia, audio); Single file but connection specific output: (using fid).

REFERENCES

http://cbbrowne.com/info/fs.html

Thursday, September 16, 2004

lab 4 - signalfs

NAME

lab 4 - signalfs; generalize wavefs to a family of file servers that support a variety of sound sources and filters.

SETUP

Inferno 4th edition release 20040830. Using wavefs and the Synthesis Toolkit in C++.

DESCRIPTION

I started with the wavefs source and abstracted out an interface for modules that the fileserver can call. The signal processor implements this interface. The compiled dis module that implements a specific dsp is passed as parameter to the signalfs which exports the standard files

/signal
/signalctl

The interface is quite simple

Signal: module {
 configstr: string;

 init: fn(args: list of string);
 config: fn(s: string);
 tickFrame: fn(): array of real;
};

Messages written to /signalctl are passed to config, and reads return the contents of Signal.configstr. Reads from /signal, which is readonly, are sent data generated by tickFrame. I rewrote the wave loop module using this interface. wave.b The server is now mounted as

% mount {signalfs wave.dis sinewave.raw} /mnt/dsp/wave
% lc /mnt/dsp/wave
signal    signalctl

I implemented a few other DSPs. noise.b envelope.b filter.b The original source for all of these is from the STK. Each signal has it's own config options. Here's an example using the filter

% mount {signalfs filter.dis} dsp/filter
% echo acoef 1.0 0.1 0.5 > dsp/filter/signalctl
% echo bcoef 0.5 0.5 > dsp/filter/signalctl
% echo source dsp/wave/signal
% cat dsp/filter/signalctl
rate 22050
chans 1
gain 1
source dsp/wave/signal
acoef 1 .1 .5 
bcoef .5 .5 
% read 40000 < dsp/filter/signal > /dev/audio

CONCLUSION

It is too slow. We need another iteration to read vectors of ticks instead of single ticks.

I like the layout. It is simple to add new filters and sources. A filter can be applied to another signal file by just writing to the control file.

We'll need a mixerfs or voicefs to combine many voices and something to do the sequencing. It's nice generating sounds from the shell.

REFERENCES

Cook, Perry and Scavone, Gary P. Synthesis Toolkit in C++ http://www-ccrma.stanford.edu/software/stk/

Wednesday, September 15, 2004

lab 3 - wavefs

NAME

lab 3 - create a filesystem, wavefs, that serves a looped waveform. This is our staring point for playing with DSP.

SETUP

Inferno 4th edition release 20040830. Using styxservers(2), audio(3), audio(6). and Synthesis Toolkit in C++ stk. Emu is running hosted on Windows XP. (I haven't gotten the sound figured out yet using esd.)

DESCRIPTION

I've been looking for tools to experiment with DSP. The synthesis toolkit has a good all round collection of DSP routines and the code is easy enough to read. I wanted this "sound workbench" well integrated with inferno. Where possible make the resources files or file2chans.

The first step was to make a sound source; an oscillator. For this I created a file server that serves two files

/ctl
/data

I started with the sample code from styxservers-nametree(2). The server is given a raw audio file to loop. I took the example raw files included in the STK distribution. These are 16bit mono, big-endian two's complement.

The data file served is an unlimited stream of data in 16bit mono, little-endian, two's complement suitable for streaming to /dev/audio

% mkdir /mnt/dsp
% mount {wavefs sinewave.raw} /mnt/dsp
% echo rate 22050 > /dev/audioctl
% echo chans 1 > /dev/audioctl
% read 40000 < /mnt/dsp/wave > /dev/audio

Reading the ctl file describes attributes of the sound file, similar to /dev/audioctl Change attributes, such as pitch, by writing to ctl. For example to create a sinewave rising in pitch

{for (i in `{seq 300 10 700}) {
 echo pitch $i > /n/dsp/wavectl; 
 read 4000 < /n/dsp/wave }
}> /dev/audio

seq is a port of the plan9 command.

I made some helper programs to look at the wave form.

% read < 100 /mnt/dsp/wave |pcm |gr

The ffts doesn't seem to work, or I misunderstand how to use it.

CONCLUSION

These are the final versions: pcm.b gr.b wavefs.b ffts.b seq.b And here is the sinewave from the STK package. sinwave.raw Given the above I have a model to build a family of fileservers. Each serves a ctl and data file. Reading from data always reads the sound samples in a format suitable for /dev/audio and writing to ctl controls the effect, filter, or oscillator.

The filesystems can be layered. So a filter is started by giving it as parameter the wave file

% mount {wavefs sinewave.raw} /mnt/dsp
% mount {lowpfs /mnt/dsp/wave} /mnt/dsp

And so on. It'll be interesting to see how layering filesystems on a larger scale works out. A patch bay of filesystems may be required.

This is also an excuse to learn about DSP.

Saturday, September 11, 2004

lab 2 - cryptfs

NAME

lab 2 - use file2chan to create a cryptfile that encrypts/decrypts all read and writes to an underlying file. This cryptfile can then be used by kfs to create a crypt file system.

SETUP

Inferno 4th edition release 20040830. Using kfs(4), file2chan(2), keyring-crypt(2).

DESCRIPTION

Keyring-crypt contains ECB algorithms for block encryption and random access to files. I'll use the Ideaecb for this session.

I setup a simple file2chan prog that layers atop another file and passes through the read/writes. cryptfile0.b Tested this.

% > t1
% cryptfile0 /chan/crypt t1
% echo this is a test > /chan/crypt
% cat /chan/crypt
this is a test
% ls -l /chan/crypt
--rw-rw---- s 0 caerwyn caerwyn 0 May 27 14:41 /chan/crypt

The size of the file is always 0. I checked the /appl/cmd/disk/kfs.b for calls it makes to the file. All reads and writes are blocks; blocksize can be given as a parameter. It calls fstat in one place to get the file size. I changed it to take the size of the file as a parameter.

% diff /n/fossildump/2004/0910/usr/inferno/appl/cmd/disk/kfs.b kfs.b 
370a371
> wrenlen := 0;
400a402
>   's' => wrenlen = int arg->earg();
2863c2865,2868
<  return int (d.length / big RBUFSIZE);
---
>  if(wrenlen != 0)
>   return int (big wrenlen / big RBUFSIZE);
>  else
>   return int (d.length / big RBUFSIZE);

I wrote the next iteration of cryptfile to perform block writes which must be a multiple of 8 for Ideaecb. cryptfile1.b

I set BUFSIZE to 8 and tested. I tested without the encryption until I got the block read/writes correct. Here is an example with encryption on.

% zeros 1024 1 > t1
% cryptfile1 /chan/crypt t1
% echo this is a test > /chan/crypt
% read 10 < /chan/crypt
this is a % 
read 10 < t1 |xd
0000000 76b3129a eab1c334 c0df0000
000000a

I tested cryptfile1 with kfs. I changed the BUFSIZE to 512 and used the same size for kfs.

% zeros 1024 2048 > kfs.file
% ls -l kfs.file
--rw-rw---- U 4 caerwyn caerwyn 2097152 Sep 11 12:37 /n/local/kfs.file
% cryptfile1 /chan/crypt  kfs.file '0123456789abcdef'
% mount -c {disk/kfs -b 512 -r -P -s 2097152 /chan/crypt} /n/kfs
kfs: reaming the file system using 512 byte blocks
kfs: initializing minimal user table
% echo this is a test > /n/kfs/t1
% ls -l /n/kfs/t1
--rw-rw-rw- M 14 none adm 15 Sep 11 12:40 /n/kfs/t1
% cat /n/kfs/t1
this is a test

CONCLUSION

Layering of filesystems is fascinating. There is very clear separation between programs, a well defined interface, and reuse of code to accomplish something new. In this example we have the kfs on top of cryptfile on top of the host file system. And of course other file systems can be run on top of the files within kfs, such as tarfs.

I haven't looked into the security of this solution. ECB itself is not as secure as CBC, but only ECB is usable for random access to blocks. There is likely to be known plaintext within the kfs.file such as the magic word at the start. Also, decrypted data is stored in memory and might be swapped to disk by the host system.

A small enhancements would be to prompt twice for password, or passphrase, without echo.

Thursday, September 02, 2004

lab 1 - postdate

NAME

lab 1 - implement ideas from postdate in inferno sh. write shell functions that are polymorphic along the valid timeline.

SETUP

Inferno 4th edition release 20040830. Using sh(1), and tickfs(1) and associated commands, rng(1), taq (1).

DESCRIPTION

Sh has command blocks which can be passed as params to commands, and executed. I want to exploit this to see if I can implement much of the flavor of Postdate in the shell. I already have tickfs in Inferno which supports bi-temporal binary relations. So the value for a tickfs entry could be the sh command block.

Create a new relation with entry of command block

% mount -c {tickfs} /n/tick
% touch /n/tick/sh.bt
% echo ${quote i . add {fc $1 $2 +}} > /n/tick/sh.bt
% rng , |taq -1rm /n/tick/sh.bt
1094173750 add '{fc $1 $2 +}'

I want shell function to return the command block from tickfs. It has to be a substitution function to keep it as a command block, because echo causes it to be a string.

% subfn pd {
 f = $1
 (a b c) = `{echo `{date -n} $f . |taq /n/tick/sh.bt}
 result = ${unquote $"c}
}
% echo ${pd add}
{fc $1 $2 +}

I can now call the function add; the code comes from a lookup in tickfs

% ${pd add} 1 1
2

I'll create a function I want to vary along the timeline. I can also define variables using the same method. I'll create a substition function that uses rng to give the date in epoch format. And then use that in ${pd} to select the effective fuction or variable

% subfn rng {
 r = $1
 (a b c) = `{rng $r}
 result =  $b
}

% date ${rng 20040901}
Wed Sep 01 00:00:00 EDT 2004

% subfn pd{
 (f r) = $*
 (a b c) =  `{echo ${rng $r} $f .  | taq /n/tick/sh.bt}
 result = ${unquote $"c}
}

Pdfn defines new functions in tickfs. It takes args rng name {cmd}

% fn pdfn {
 (r args) = $*
 echo ${quote i ${rng $r} $args} > /n/tick/sh.bt
}
% pdfn 20040101 rate {fc $1 2 x}
% pdfn 20040601 rate {fc $1 4 x}
% pdfn . rate {fc $1 8 x}
% pdfn 20040101 a 10

Now call these functions at different times

% ${pd rate 0401} 1
2
% ${pd rate 0701} 1
4
% ${pd rate 1201} ${pd a 1201}
80

In Postdate I had a dictionary stack. In Inferno the /n/tdb/sh.bt, or other .bt, file is the dictionary. I can push and pop easily from list in sh.

% pdstk = (n/tick/sh.bt)
% fn pdpush { pdstk = ($1 $pdstk)}
% fn pdpop {pdstk = ${tl $pdstk}}

I have to redefine pdfn and pd to use ${hd $pdstk} instead of hardcoded /n/tick/sh.bt.

The usual mode of processing tickfs result sets is in a pipeline. If I define a temporal package as a tickfs file with a main sh command block, the pdpkg command will call main on one triad at a time from stdin. It doesn't need to convert from YYYYMMDD format to epoch because triads always come in epoch format. We'll get around that by just defining another pd function, say epd, that takes seconds since the epoch.

% subfn epd{
 (f r) = $*
 (a b c) =  `{echo $r $f .  | taq ${hd $pdstk}}
 result = ${unquote $"c}
}

% fn pdpkg {
 pdpush $1
 getlines {
  (t k v) = ${unquote $line}
  ${epd main $t} $k $v
 }
 pdpop
}

% pdfn 20040201 main {fc $2 10 x}
% touch /n/tick/tick.bt
% echo i . a 1 > /n/tick/tick.bt
% echo i . b 2 > /n/tick/tick.bt
% echo i . c 3 > /n/tick/tick.bt
% rng , |taq -1rm /n/tick/tick.bt |pdpkg /n/tick/sh.bt
10
20
30

CONCLUSION

I have created shell functions and variables that can vary along the valid time line. I created packages, blocks of temporal shell code, that can be applied to tickfs result set. It is more featureful that Postdate, since we have the whole shell and inferno at our disposal. It is slow. I'm not concerned with the perfomance now. I want to find out if there's some interesting functions that I can implement that can vary along the timeline.

Postdate also has the effective valid time stack. We could implement the stack in the same way as pdstk but really the effective time is in the callstack since it is passed as a param to every call of pd.

Tuesday, August 31, 2004

commentary: an open laboratory

what is an open laboratory?

it is an online community of computer scientists (programmers, hackers, hobbyists) experimenting with computers and software. the lab is not affilated with any corporation or university or government. the lab should provide to its members the same benefits of working in an actual laboratory; diversity of thought, collaboration, peer review ...

why a lab?

the distinction we'd like to make is that the lab is used for learning, discovery, and innovation. we are not developing products for market, or rewriting existing applications for open source distribution. the lab environment would encourage collaboration by making the notebook the medium for sharing work products before they reach the stage suitable for presentation in a report. coming up with new ideas is messy. but peer review, discussion and mentoring are valuable at this early stage.

how would an open lab operate?

we'd begin by agreeing on a common programming environment (inferno, squeak, mathematica, ...), and a format for the lab notebook. the lab notebook needs to be reviewed and signed by a colleague in the lab, possibly a mentor. every lab session is documented in the notebook. it includes all details needed to repeat what was done and analysis and conclusions drawn from the lab session.

at some point a report is written for the lab's journal which describes the path of research taken. the report's details being derived from the notebook. through peer review, discussion, and further experimentation good ideas rise to the top and reputations are built.

why the common programming environment?

convenience. foster a lab culture. a good candidate for a programming environment is one where all the source is available; the system is small and well written so that any part of it can be modified for experimentation; and the complete system is easily understood.

commentary: getting started

even getting started is harder than i thought.

i'm having trouble deciding what tools to use to present the notebook. i already started it as a blog here thinking i'd just write html as i go and post it at the end of each session.

but thinking a little more about it, i want more interaction, and i want it to integrate well within inferno.

i could write it in troff man macros, which would work well in inferno. but inferno doesn't support full troff and preprocessors eqn, tbl, and pic. i considered LaTeX; but this seems more appropriate for a final printable version of a report.

looking around on the web; the Mathematica Notebook is closer to what I had in mind. a interactive technical notebook. code can be run, graphics and sound presented and manipulated.

another similar concept is the Squeak environment. Workspaces can be saved and exported. These can present models, simulations, experiments with objects.

so what do we have in inferno. plumbing can drive the interactivity for the notebook. the notebook could be presented through wm/man and other files, objects, can be plumbed from there. the notebook file system could be exported for viewing.

we'll have to experiment as we go :)

Saturday, August 28, 2004

Inferno lab

this is my inferno lab notebook.

each entry, when it describes a programming session in inferno, should include at least the following.

  • purpose; what i hope to get done in the session
  • apparatus and settings; so someone else could reproduce what i've done
  • diary; record of actions and results
  • analysis and conclusions

i use inferno for fun. the sessions i describe take place at home, in the evenings, usually monday to friday, 9:00pm to 11:30pm. This isn't always enough time to get things done, so some of the tasks may seem quite trivial.

i want to move through a wide range of subjects in computer science and elsewhere, wherever my curiosity takes me. i don't want to limit myself by false subject boundaries. i hope to experiment a little, learn a lot, and, just maybe, discover something new. inferno presents a good environment to do this work, for a variety of reasons i'll explore while i'm using it.

welcome. and if you like the idea join the lab.