Thursday, September 30, 2004

lab 8 - treemap


lab 8 - small update to treemap.


Tmap is a small util that takes the output of du and presents a treemap using algorithms from UMD HCILab tmap

To make it a bit more useful I want to zoom into the treemap and zoom out again. Button one selects a rectangle highlights it and it's path and presents information about the path and file sizes. Button two zooms one level into the selected area, where the mouse clicked. Button three zooms out one level.

This was easy to add using Tk. I bind a command to the mouse events and send a message down a channel. On receiving the message I change the root of the tree, regenerate the layout and redraw the rectangles.


I find this a useful tool to find patterns in disk usage, such as large or duplicate files. Often the file hierachies are very deep, and so drilling down interactively is great.

I should re-implement this by drawing directly on an image instead of using Tk. I seem to be running out of memory using Tk because I am not keeping track of the Tk tags when creating new rectangles, and I'm creating so many of them. It may also be faster, but I don't know.

There is a bug in the squarified view where the directory is rendered as if it is a file, using the totalsize of all sub elements. I haven't found what causes this. The other views seem okay.

I think the treemap is a beautiful view of the filesystem. I'd like to put it to more use.

In a collaborative environment the treemap of a filesystem is a shared map of the system. The rectangles, which represent files, are color coded for the users who have them open. This could fit in as an engine to spree, or collab. We'd need to monitor all styx messages, similar to Styxmon.

The prog device shows all files open and process activity. Processes might also be presented as a treemap.

The treemap could then be made more interactive. Files are plumbed, to open and edit them. The files are annotated and new annotations are seen by all users of the treemap.

This looks like a good excuse to play with spree. I'm thinking along the lines of VisitorVille and Plan9's faces.


tmap.b treemap.b treemap.m



Tuesday, September 28, 2004

lab 7 - sequencer


lab 7 - sequencer; create a simple sequencer that can play back a subset of SKINI messages.


Inferno 4th Edition release 20040830.


2004/0928 20:37 SKINI is the sequencing language from the STK. It is a readable form of MIDI, and was designed to be "extensable and hackable"; all of which make it ideally suited to this application and Inferno. Here is a brief example

// Measure number 1 =0
NoteOn    0.416667     2   72   64
NoteOff   0.208333     2   72   64
NoteOn    0            2   71   64
NoteOff   0.208333     2   71   64
NoteOn    0            2   72   64
NoteOff   0.416667     2   72   64
NoteOn    0            2   67   64
NoteOff   0.416667     2   67   64

There is one command per line. The line begins with command name followed by parameters separated by space or tabs. The second parameter is always the time delta. For the NoteOn command, the third argument is channel (or voice), fourth is midi pitch, and fifth if velocity (I guess! I'm ignoring it for now).

These SKINI messages are interpreted by sequencer, which sends messages to one or more instruments and reads back audio data.

I created a basic instrument module, which should be used as a template for later instruments, called simple. It uses the adsr and wave modules.

The wave module I have described previously. Adsr is a standard Attack, Decay, Sustain, Release envelope. I copied the implementation from the STK.

Simple and sequencer use a new interface to assist in using other signal sources.

Sig: adt {
 ctl: ref Sys->FD;
 data: ref Sys->FD;
 open: fn(s: string): ref Sig;
 read: fn(s: self ref Sig, nsamples: int): array of real;
 readbytes: fn(s: self ref Sig, nbyte: int): array of byte;

This will evolve. It is currently part of the dsp module.

The source signals are opened by init. I made a small modification to signalfs.b moving the init call from signalfs.b:/serve to signalfs.b:/requestproc to avoid deadlock.

I created /mnt/dsp for mounting signalfs and /mnt/dsp/raw because I am hardcoding the names of some signal sources and raw sound files. The raw files from the STK should be bound or copied to /mnt/dsp/raw. Therefore start as

% bind /n/j/stk-4.1.3/rawwaves /mnt/dsp/raw
% signalfs -a /mnt/dsp

Setup the modules and /dev/audio

% bind -a '#A' /dev
% echo rate 22050 > /dev/audioctl
% echo chans 1 > /dev/audioctl
% echo add wave.dis wave > /mnt/dsp/ctl
% echo add adsr.dis adsr > /mnt/dsp/ctl
% echo add simple.dis simple > /mnt/dsp/ctl

Run a SKINI file.

% sequencer /mnt/dsp/simple < > /dev/audio

The sequencer assumes 4 voices currently. It's very basic; just for testing while creating new instruments. It will most surely be rewritten.


Here are the latest. adsr.b dsp.b dsp.m sequencer.b signal.m signalfs.b simple.b wave.b

The SKINI language has more commands and features than implemented here.

Again, it is slow. I should buffer writes to the /dev/audio, maybe a few seconds worth, so the audio sounds smooth. Otherwise, I need to write to a file first then stream the file to /dev/audio. However, It's enough for testing while creating some of the more complex instruments from the STK.

The sequencer could be a module of signalfs like any other. Reads of the data return the audio data. The ctl file is a synthetic file which could be edited within any editor. But this is a slightly different interface than other signals. An alternative is to use the skini file as a source file just like the raw files for the wave module. The sequencer module then represents patterns, which can be combined, looped, and sequenced just like any other signal.


I can have a grid generating the sounds. What is possible with unlimited cpu power? Using a grid we should be able to create, in realtime, a very complex syntesized soundscape. Could we use the plan9 grid for this? Run emu on each node, serve a signalfs, bind them all into a local namespace, then generate the patterns. 8 channel, 24 bit, unlimited voice and polyphony.


STK is now a link on the side bar.

Wednesday, September 22, 2004

lab 6 - signal filter


lab 6 - signal modules; implement filter (again) for signalfs and test the recursiveness of the file server.


Inferno 4th Edition 20040830. See also labs 4 an 5.


2004/0922 21:24 Changed filter.b to use the clone interface. The ctl file accepts a source command which takes a filename or another signal source which may be served by the same signalfs

% echo source /usr/caerwyn/lab/6/mnt/wave > mnt/filter/2/ctl

The filter module then opens ctl reads the number of the conversation directory and then opens data. The filter is then responsible for setup of the signal source. It should probably pass through commands written to ctl that it does not understand, so the wave can be controlled through the filter ctl.

I made a small change to signalfs.b to not fork the namespace so the mount of the server affects the current process and the filter module is able to access the wave

<  pidc <-= sys->pctl(0, nil);
>  pidc <-= sys->pctl(Sys->FORKNS, nil);


The recursion (if that's the right term) works. Filter reads data from wave served by the same file server; or it could be a different file server running remotely.

This was a quick lab session to resolve my doubts about signalfs. Need to really start implementing all the filters. Especially those that combine multiple sources.


dsp.b dsp.m envelope.b filter.b noise.b signal.m signalfs.b wave.b

Tuesday, September 21, 2004

lab 5 - signalfs v2


lab 5 - signalfs version 2; generalize and enhance the signalfs still further. Create a server that can load all the signal models, and each module has a connection directory per conversation.


Inferno 4th Edition 20040830.


The directory hierarchy presented by signalfs is now similar to ip(3).


One more requirement is that the server must be recursive, a signal module must be able to open other files served by the same signalfs. (The ds (3) device does this.)

To illustrate, say we implement a module to return a Fibonacci number. I use the clone mechanism so each client has it's own connection to the module. To read the fifth Fibonacci number

           d=/mnt/modfs/`{read 10}
           echo 5 >[1=0]
           read 10 < $d/data 
          }<> /mnt/modfs/clone

The module itself uses the ctl message to determine whether it should open a connection to clone and read the Fibonacci number for the number - 1. If the ctl number is 0 it returns 1, ending the recursion.

The module is a single function. The parameters are controlled by the ctl file, and the result is read from data.

The fileserver framework manages the clone and naming and loading of modules. I used wmexport.b as the starting point for building a framework.

2004/0918 21:28 The ip(3) device comes close to what I want. A directory for each proto (module); a clone below that, and a numbered directory for each instance. The numbered directory has ctl and data. The numbered directories aren't removed but keep state variable tracking whether in use.

What if we get a read on the data file from a process other than the opening process? We want to deny this.

2004/0919 16:51 Writing and debugging.

2004/0919 20:08 Is this something like spree, where the loaded module is an engine?

2004/0919 22:41 Given the layout of signalfs a filter is given the directory for the source module. The filter opens the clone file to get a connection. It then has exclusive access to that module. The filter exposes the same interface, so I could create multiple connections to that filter. But what if I want to alter the sinewave that the filter is reading from? Do I want shared write access to the sine ctl file? I'd need to know the connection the filter was reading from. No. The filter is in complete control of it's source.

2004/0921 22:01 Writeup and debugging.

Signalfs now knows nothing of the format of the signal. The signal is responsible for converting arrays of real to bytes. The signal interface has changed

          Signal: module {
           configstr: string;

           init: fn(args: list of string);
           config: fn(s: string): string;
           read: fn(n: int): array of byte;

Config returns an error string or nil if successful. Here's an example setup

          % mkdir mnt
          % signalfs mnt
          % echo add /usr/caerwyn/lab/5/wave.dis wave > mnt/ctl
          % lc mnt
          ctl   wave/
          % <> mnt/wave/clone {
           d=mnt/wave/`{read 10}
           echo file /usr/caerwyn/lab/3/sinewave.raw >[1=0]
           read 8 < $d/data | pcm


Here is the current version. signalfs.b, signal.m and the sinewave module again wave.b

Once this is debugged, I've reached the point where I can write all the signal modules. I still have no real plan for a sequencer. It may end up being shell script. I haven't tested the recursiveness yet.

I could have implemented signals using file2chan (2) except I am supporting the more complicated interface of clone, ctl and data. I hope it will be worth it having all the modules organized together under one server.

Tickfs might be better adapted to this interface. The ctl message for lookup are written to the ctl file, and I can control permission on the data file.

At some point I should provide a brief taxonomy of fileservers. E.g., as represented by ip(3), ds(3), file2chan, kfs (traditional), env(3), etc. Traditional file access from underlying storage (kfs, tarfs); Conversational object access (draw); Shared object access to virtual storage (env); Device interface (eia, audio); Single file but connection specific output: (using fid).


Thursday, September 16, 2004

lab 4 - signalfs


lab 4 - signalfs; generalize wavefs to a family of file servers that support a variety of sound sources and filters.


Inferno 4th edition release 20040830. Using wavefs and the Synthesis Toolkit in C++.


I started with the wavefs source and abstracted out an interface for modules that the fileserver can call. The signal processor implements this interface. The compiled dis module that implements a specific dsp is passed as parameter to the signalfs which exports the standard files


The interface is quite simple

Signal: module {
 configstr: string;

 init: fn(args: list of string);
 config: fn(s: string);
 tickFrame: fn(): array of real;

Messages written to /signalctl are passed to config, and reads return the contents of Signal.configstr. Reads from /signal, which is readonly, are sent data generated by tickFrame. I rewrote the wave loop module using this interface. wave.b The server is now mounted as

% mount {signalfs wave.dis sinewave.raw} /mnt/dsp/wave
% lc /mnt/dsp/wave
signal    signalctl

I implemented a few other DSPs. noise.b envelope.b filter.b The original source for all of these is from the STK. Each signal has it's own config options. Here's an example using the filter

% mount {signalfs filter.dis} dsp/filter
% echo acoef 1.0 0.1 0.5 > dsp/filter/signalctl
% echo bcoef 0.5 0.5 > dsp/filter/signalctl
% echo source dsp/wave/signal
% cat dsp/filter/signalctl
rate 22050
chans 1
gain 1
source dsp/wave/signal
acoef 1 .1 .5 
bcoef .5 .5 
% read 40000 < dsp/filter/signal > /dev/audio


It is too slow. We need another iteration to read vectors of ticks instead of single ticks.

I like the layout. It is simple to add new filters and sources. A filter can be applied to another signal file by just writing to the control file.

We'll need a mixerfs or voicefs to combine many voices and something to do the sequencing. It's nice generating sounds from the shell.


Cook, Perry and Scavone, Gary P. Synthesis Toolkit in C++

Wednesday, September 15, 2004

lab 3 - wavefs


lab 3 - create a filesystem, wavefs, that serves a looped waveform. This is our staring point for playing with DSP.


Inferno 4th edition release 20040830. Using styxservers(2), audio(3), audio(6). and Synthesis Toolkit in C++ stk. Emu is running hosted on Windows XP. (I haven't gotten the sound figured out yet using esd.)


I've been looking for tools to experiment with DSP. The synthesis toolkit has a good all round collection of DSP routines and the code is easy enough to read. I wanted this "sound workbench" well integrated with inferno. Where possible make the resources files or file2chans.

The first step was to make a sound source; an oscillator. For this I created a file server that serves two files


I started with the sample code from styxservers-nametree(2). The server is given a raw audio file to loop. I took the example raw files included in the STK distribution. These are 16bit mono, big-endian two's complement.

The data file served is an unlimited stream of data in 16bit mono, little-endian, two's complement suitable for streaming to /dev/audio

% mkdir /mnt/dsp
% mount {wavefs sinewave.raw} /mnt/dsp
% echo rate 22050 > /dev/audioctl
% echo chans 1 > /dev/audioctl
% read 40000 < /mnt/dsp/wave > /dev/audio

Reading the ctl file describes attributes of the sound file, similar to /dev/audioctl Change attributes, such as pitch, by writing to ctl. For example to create a sinewave rising in pitch

{for (i in `{seq 300 10 700}) {
 echo pitch $i > /n/dsp/wavectl; 
 read 4000 < /n/dsp/wave }
}> /dev/audio

seq is a port of the plan9 command.

I made some helper programs to look at the wave form.

% read < 100 /mnt/dsp/wave |pcm |gr

The ffts doesn't seem to work, or I misunderstand how to use it.


These are the final versions: pcm.b gr.b wavefs.b ffts.b seq.b And here is the sinewave from the STK package. sinwave.raw Given the above I have a model to build a family of fileservers. Each serves a ctl and data file. Reading from data always reads the sound samples in a format suitable for /dev/audio and writing to ctl controls the effect, filter, or oscillator.

The filesystems can be layered. So a filter is started by giving it as parameter the wave file

% mount {wavefs sinewave.raw} /mnt/dsp
% mount {lowpfs /mnt/dsp/wave} /mnt/dsp

And so on. It'll be interesting to see how layering filesystems on a larger scale works out. A patch bay of filesystems may be required.

This is also an excuse to learn about DSP.

Saturday, September 11, 2004

lab 2 - cryptfs


lab 2 - use file2chan to create a cryptfile that encrypts/decrypts all read and writes to an underlying file. This cryptfile can then be used by kfs to create a crypt file system.


Inferno 4th edition release 20040830. Using kfs(4), file2chan(2), keyring-crypt(2).


Keyring-crypt contains ECB algorithms for block encryption and random access to files. I'll use the Ideaecb for this session.

I setup a simple file2chan prog that layers atop another file and passes through the read/writes. cryptfile0.b Tested this.

% > t1
% cryptfile0 /chan/crypt t1
% echo this is a test > /chan/crypt
% cat /chan/crypt
this is a test
% ls -l /chan/crypt
--rw-rw---- s 0 caerwyn caerwyn 0 May 27 14:41 /chan/crypt

The size of the file is always 0. I checked the /appl/cmd/disk/kfs.b for calls it makes to the file. All reads and writes are blocks; blocksize can be given as a parameter. It calls fstat in one place to get the file size. I changed it to take the size of the file as a parameter.

% diff /n/fossildump/2004/0910/usr/inferno/appl/cmd/disk/kfs.b kfs.b 
> wrenlen := 0;
>   's' => wrenlen = int arg->earg();
<  return int (d.length / big RBUFSIZE);
>  if(wrenlen != 0)
>   return int (big wrenlen / big RBUFSIZE);
>  else
>   return int (d.length / big RBUFSIZE);

I wrote the next iteration of cryptfile to perform block writes which must be a multiple of 8 for Ideaecb. cryptfile1.b

I set BUFSIZE to 8 and tested. I tested without the encryption until I got the block read/writes correct. Here is an example with encryption on.

% zeros 1024 1 > t1
% cryptfile1 /chan/crypt t1
% echo this is a test > /chan/crypt
% read 10 < /chan/crypt
this is a % 
read 10 < t1 |xd
0000000 76b3129a eab1c334 c0df0000

I tested cryptfile1 with kfs. I changed the BUFSIZE to 512 and used the same size for kfs.

% zeros 1024 2048 > kfs.file
% ls -l kfs.file
--rw-rw---- U 4 caerwyn caerwyn 2097152 Sep 11 12:37 /n/local/kfs.file
% cryptfile1 /chan/crypt  kfs.file '0123456789abcdef'
% mount -c {disk/kfs -b 512 -r -P -s 2097152 /chan/crypt} /n/kfs
kfs: reaming the file system using 512 byte blocks
kfs: initializing minimal user table
% echo this is a test > /n/kfs/t1
% ls -l /n/kfs/t1
--rw-rw-rw- M 14 none adm 15 Sep 11 12:40 /n/kfs/t1
% cat /n/kfs/t1
this is a test


Layering of filesystems is fascinating. There is very clear separation between programs, a well defined interface, and reuse of code to accomplish something new. In this example we have the kfs on top of cryptfile on top of the host file system. And of course other file systems can be run on top of the files within kfs, such as tarfs.

I haven't looked into the security of this solution. ECB itself is not as secure as CBC, but only ECB is usable for random access to blocks. There is likely to be known plaintext within the kfs.file such as the magic word at the start. Also, decrypted data is stored in memory and might be swapped to disk by the host system.

A small enhancements would be to prompt twice for password, or passphrase, without echo.

Thursday, September 02, 2004

lab 1 - postdate


lab 1 - implement ideas from postdate in inferno sh. write shell functions that are polymorphic along the valid timeline.


Inferno 4th edition release 20040830. Using sh(1), and tickfs(1) and associated commands, rng(1), taq (1).


Sh has command blocks which can be passed as params to commands, and executed. I want to exploit this to see if I can implement much of the flavor of Postdate in the shell. I already have tickfs in Inferno which supports bi-temporal binary relations. So the value for a tickfs entry could be the sh command block.

Create a new relation with entry of command block

% mount -c {tickfs} /n/tick
% touch /n/tick/
% echo ${quote i . add {fc $1 $2 +}} > /n/tick/
% rng , |taq -1rm /n/tick/
1094173750 add '{fc $1 $2 +}'

I want shell function to return the command block from tickfs. It has to be a substitution function to keep it as a command block, because echo causes it to be a string.

% subfn pd {
 f = $1
 (a b c) = `{echo `{date -n} $f . |taq /n/tick/}
 result = ${unquote $"c}
% echo ${pd add}
{fc $1 $2 +}

I can now call the function add; the code comes from a lookup in tickfs

% ${pd add} 1 1

I'll create a function I want to vary along the timeline. I can also define variables using the same method. I'll create a substition function that uses rng to give the date in epoch format. And then use that in ${pd} to select the effective fuction or variable

% subfn rng {
 r = $1
 (a b c) = `{rng $r}
 result =  $b

% date ${rng 20040901}
Wed Sep 01 00:00:00 EDT 2004

% subfn pd{
 (f r) = $*
 (a b c) =  `{echo ${rng $r} $f .  | taq /n/tick/}
 result = ${unquote $"c}

Pdfn defines new functions in tickfs. It takes args rng name {cmd}

% fn pdfn {
 (r args) = $*
 echo ${quote i ${rng $r} $args} > /n/tick/
% pdfn 20040101 rate {fc $1 2 x}
% pdfn 20040601 rate {fc $1 4 x}
% pdfn . rate {fc $1 8 x}
% pdfn 20040101 a 10

Now call these functions at different times

% ${pd rate 0401} 1
% ${pd rate 0701} 1
% ${pd rate 1201} ${pd a 1201}

In Postdate I had a dictionary stack. In Inferno the /n/tdb/, or other .bt, file is the dictionary. I can push and pop easily from list in sh.

% pdstk = (n/tick/
% fn pdpush { pdstk = ($1 $pdstk)}
% fn pdpop {pdstk = ${tl $pdstk}}

I have to redefine pdfn and pd to use ${hd $pdstk} instead of hardcoded /n/tick/

The usual mode of processing tickfs result sets is in a pipeline. If I define a temporal package as a tickfs file with a main sh command block, the pdpkg command will call main on one triad at a time from stdin. It doesn't need to convert from YYYYMMDD format to epoch because triads always come in epoch format. We'll get around that by just defining another pd function, say epd, that takes seconds since the epoch.

% subfn epd{
 (f r) = $*
 (a b c) =  `{echo $r $f .  | taq ${hd $pdstk}}
 result = ${unquote $"c}

% fn pdpkg {
 pdpush $1
 getlines {
  (t k v) = ${unquote $line}
  ${epd main $t} $k $v

% pdfn 20040201 main {fc $2 10 x}
% touch /n/tick/
% echo i . a 1 > /n/tick/
% echo i . b 2 > /n/tick/
% echo i . c 3 > /n/tick/
% rng , |taq -1rm /n/tick/ |pdpkg /n/tick/


I have created shell functions and variables that can vary along the valid time line. I created packages, blocks of temporal shell code, that can be applied to tickfs result set. It is more featureful that Postdate, since we have the whole shell and inferno at our disposal. It is slow. I'm not concerned with the perfomance now. I want to find out if there's some interesting functions that I can implement that can vary along the timeline.

Postdate also has the effective valid time stack. We could implement the stack in the same way as pdstk but really the effective time is in the callstack since it is passed as a param to every call of pd.