Thursday, April 26, 2007

lab 77 - the unexpected markov


lab 77 - the unexpected markov


This is another unexpected use (again) of the markov program from the `The Practice of Programming', section 3.9 [1]. I wrote an implementation of markov under Limbo, and had fun feeding it with any texts (books, articles, interviews, novels ...).

But recently I've been also playing with caerwyn's synth [2] which included in acme-sac, and thought why not feeding markov with music?

Find out the answer by yourselves, I'll just provide some small hints in the accompanying guide file.


[2] synth from acme-sac under appl/synth


bachm.mp3 (the original bach file bach.mp3)

Sunday, April 15, 2007

lab 76 - using svn with gcode projects


This is not a typical lab, instead are some suggestions to work with svn repos (the ones provided by gcode); like inferno-lab or the rest of Inferno related projects at gcode.


To work with svn the easiest way is to install subversion under the host os and write a wrapper to have the svn command available under Inferno, like:

cat > $home/dis/sh/svn << EOF
load std

wdir=`{hp `{pwd}}
if {~ $emuhost Nt}{
 os -d $wdir svn $* | tr -d '
 os -d $wdir svn $* 
The svn script relies in hp, to convert the Inferno paths to host paths, so here is hp:
cat > $home/dis/sh/hp << EOF
# convert a inferno path to valid host path for os(1)
load std

if {no $*}{
 echo 'usage: hp  # to get host path'

if {~ $emuhost Nt}{
 fn slashpath () { sed 's|\\|\\\\|g'; }
 # put two where there's one
 emuroot=`{echo $emuroot | slashpath | slashpath}
 for p in $* {
  cleanname -d `{pwd} $p | sed -n '
  /^\/n\// {s|/n/(.)/|\1:\\\\|p; q;}
 # host letters subst
 # hls="{ns | sed -n '/#U.+/ s|^bind -. ''#U(.*)'' (.*)|s\|\2\|\1/\|p|p'}

 for p in $* {
  cleanname -d `{pwd} $p | sed -n '
  /^\/n\// {s|/n/local/|/|p; '^$hls^'; q;}
After giving them executable permission
% chmod u+x $home/dis/sh/svn
% chmod u+x $home/dis/sh/hp
and binding with bind(1) $home/dis/sh to /dis,
% bind -b $home/dis/sh /dis
this line can be added to your profile, to have them available when sh starts. After this commands under $home/dis/sh can be executed as any other sh(1) shell command available under /dis. To make an example now one can checkout inferno-lab running:
% svn checkout inferno-lab
And the same for the rest of svn commands, to check the status, diffs and commits. Good luck


svn, the original svn from acme-sac.
hp, the initial hp.

Thursday, April 12, 2007

lab 75 - scope & experiments


lab75 - scope & experiments


Since i did lab 67 i've been trying to improve/fix the t3 port and experiment with it. So this post has small report of the T3 port status, and some experiments under Inferno; note that they're not dependent of the handheld.


Some of the thing i've fixed are:

t3 touchscreen: Perform corrections to make it work as it should, this was important since it has direct impact on using Acme, and the rest of the system.

blank the lcd: Added code to turn off the lcd while playing music, so battery lasts longer. To do so i added an blank command to the devapm.c written by Alexander Syschev, and wrote a blight script that manages the lcd backlight,to control this from acme.

Since this changes apply to the t3 port, they can be found under lab 67 of the inferno-lab

While i haven't been able to fix the following segfaults, i've been able to obtain dumps and open them with gdb. So i've been able to find where the crashes happen, but still don't understand why they happen, so i'll have to read more.

* sys->print('%g', ...): with floating formats strings: "%g", "%f",when emu crashes it says

 "malloc failed at <addr>"
and if i disassemble emu, i can find that the <addr> corresponds to dopoolalloc(), and the core dump says that the crash happends at /libmath/dtoa.c:/rv_alloc\(i\)/ .

* emu -c1 /dis/sh.dis: emu starts loading Emuinit /appl/cmd/emuinit.b, it's able to load a few modules on which /dis/sh.dis depends (disdep /dis/sh.dis) but it crashes while loading Filepat, this seems harder since i'm still learning about arm architecture, and i know even less about jit.


Some of them were begun on lab 67 but i've improved them a bit and it's worth to dedicate them a new post.

scope.b: Parallel to the work of caerwyn with the DSP toolkit, i thought that it would be interesting to have a flexible oscilloscope, to view signals. So i started with the scope.b of lab 12, and started to add:
make it look nicer (add grid, and signal info)
make it more flexible, adding some options and widgets to control the oscilloscope. Also in this direction: allow resizal of the oscilloscope window.
and more
frequency spectrum of the signal (using fft).

And the result has been:

From inferno photos
If you want to play with it there's a guide file that i wrote while coding and can give some ideas about it's usage.

voip: this basically using stream to stream the recording of a microphone to the headphones of another computer, in short:
styxlisten tcp!firulillo!styx export '#A';
mount tcp!firulillo!styx /n/remote;
stream /dev/audio /n/remote/audio;
after some attemps i decided to pack it under a script that could be used both for publish '#A' (server) and to mount '#A' (client), the problem is that if you try, the sound appears delayed and some echo.

Reading stream(1) i noticed that stream ussies writes of bufsize (by default Sys->ATOMICIO: 8192), and to do it has to wait until it has bytes bufsize, so here was the reason for the delay and echo. Playing with bufsize, i set it by default to 1024 bytes, this works on a local net, probably it'll need adjustment for other situations.

player: i started this as a music player for the handheld, but i've ended using it under the PC, it's simply a script that lists the music files at cwd and plays them with a music player available in the host os: mplayer. A nice thing is that one can export a dir containing a collection of music files with:
 styxlisten -A 'tcp!*!styx' export /n/local/music
mount them from another machine under /n/remote and play them running:
 mount -A 'tcp!$server!styx' /n/remote
cd /n/remote && player

kbd2skini.b: This is just an idea in the direction of the DSP toolkit, currently the DSP toolkit is able to generate music, from standard music formats mainly contained in files (midi,skini).
I would like to play with the input devices available under inferno to use them as instruments. So i started to gather for similar things and found pc2midi (and some useful tutorials).

I've choosen the kdb as starting point, but it has it's limitations as it's discussed on the previous reference, since you don't have a pressure indication, instead the state of each key is either pressed / or not pressed, moreover there's no way to notice that there're multiple keys pressed at a time. We'll see what can be done with this, but could this be more than a joke?



Saturday, April 07, 2007

lab 74 - effective inferno


lab 74 - effective inferno


I read the Effective Java book recently. Every language needs a book that describes how to use it effectively. I wish there was a book for Effective Inferno.

Although common techniques may work in Inferno I know for sure there are some uncommon ones that may work better.

I'll try and describe at least one recipe that could be a chapter in that book.

To people who've asked me what distinguishes Inferno, I've answered that it's the concurrent language Limbo, or that it's a portable OS that runs on a VM, or that it uses the Styx protocol to create a distributed system, or that it's about software tools that work together with the Inferno shell.

These are the ingredients; on their own they are not unique to inferno. Limbo looks very like C and to a newbie it's not obvious what is special about it. There a lots of little niceties that make Limbo pleasant to use, e.g., array slicing and tuples, but these are details; they do not greatly impact an application architecture.

A lot of the Unix-like commands under /appl/cmd/ are straight ports from Plan 9. This is old school. And the fact that they're implemented in Limbo does not greatly differentiate them from their C counterparts. The special flavor of Limbo does not come out when used this way.

The OS interface also looks much like Unix or Plan 9, except for the VM piece, so what is special about that? The advantages of platform independent come out when doing distributed applications. So there must be techniques that need to be learned to do that effectively.

In Inferno it's a new world where each ingredient might be familiar to you in another context but we can cook things a little differently here.

Extension Points

For any system we need to know how we might extend it and how we combine the pieces that are already there. Inferno provides a number of interfaces at different levels of granularity where the programmer can extend the system. Lets quickly review the most common ones, available in most systems.

1. We can write libraries and extend the systems with new functions; such things as parsing file formats or protocols. We can't live without those libraries. Most of the ones in /appl/lib are of this form.

2. We can create new commands and add to our software tool set. Commands can be developed that do one thing well, and grow the system organically.

3. We can use the shell to combine commands using pipelines or to customize command interfaces or extend the system further with commands implemented in the shell.

These are the contributions from our Unix legacy. And they are great! Plan 9 followed and gave us everything-is-a-file, the 9p protocol, and private namespaces. Inferno, being a direct descendant of Plan 9, inherited all that.

4. We can build filesystems to provide new services. We can also architect a distributed system by basing applications around the Styx protocol.

We don't have to use Limbo to extend inferno in this way, we can use any language, and there are already several implementations of the styx library. The styx protocol is itself an incredibly powerful and flexible way of extending the system. It permits re-use of existing tools that operate on files.

5. We can extend the system as clients to file services. Two big examples from Inferno and Plan 9 are Acme and the Plumber. Because of the loose coupling between client and server we can extend the system while it is running. We can write new Acme plugins while we are working within Acme.

This is a great pattern for developing services that you want extended. Don't just be a client of the system extention points but create your own extension points.

You may think this is where I'd stop. This would be enough right? What a great system. Plan 9 is a lovely system for all the above reasons. But we haven't got unique to inferno yet. And there is something more lovely. Bear with me, this will take a few paragraphs.

6. Limbo supports dynamically loadable modules. We can define an extension point as a module interface that the client is expected to implement. Then we can load the client, one of possibly many implementations, as needed.

This is distinct from a module interface used by libraries where in practice there is really the one and final implementation. [1], [2].

What I mean here is that there a many clients that might do distinctly different things but are able to use a shared interface with other clients.

The simplest example is the Command interface with the shell controlling loading of modules.

Command: module {
 init: fn(ctxt: ref Draw->Context, argv: list of string);

This is the extension point. You implement a client that supports this interface and then the shell will call it. Implicit in the interface is the fact that you run as a separate process and you have file descriptors for stdin, stdout, and stderr.

Of course, we all know this interface because it's similar to ones used by shells on most operating systems.

Now you say, I've already covered this in point 2. But this is different. Inferno allows us to define new interfaces, that can be specific to our problem.

For example, the Inferno shell defines a more complicated interface for shell loadable modules that extend the shell and use shells syntax to for structured arguments.

But here's the kicker. Inferno's shell implements the Lispish pattern of using a standard syntax to represent data and code. Instead of parens it uses braces but other than that we have lists of nested lists.

 {f x {g y z } }

7. We can extend the system by re-using shell syntax to structure the input to our own applications

The shell parses a block, does syntax checking but doesn't eval it. It leaves it up to the command to evaluate, either builtin or external. So we can implement the shell pattern, but not use the shell builtin infrastructure. We define our own evaluator and pass it a shell block. We define, in a sense, our own shells specific to a problem domain.

The other big piece to this is that the module interfaces can include typed channels and that each run in a separate process so that the modules can communicate to each other, working together and forming the equivalent of pipelines, but with types.

I'll give a few examples to really push this point. This is not a trivial feature and it's easy to overlook.

The prime example in inferno is the fs(1) command.

Fs uses shell syntax but doesn't use shell to evaluate it. Fs is a tree walker that allows arbitrary mixing of components to operate on a tree. It's a remarkable command because it combines sh syntax reuse, with the module extension point for clients that use channels to communicate between processes. Its a lovely thing. The extension point looks like this.

Fsmodule: module {
 types: fn(): string;
 init: fn();
 run: fn(ctxt: ref Draw->Context, 
  r: ref Fslib->Report,
  opts: list of Fslib->Option, 
  args: list of ref Fslib->Value): ref Fslib->Value;

Here's an example of how I use fs to build the acme-sac distribution. I filter out .svn files, filter out .sbl and .dis files below /appl and object files below /sys, then copy the whole tree using a proto file to define the components to copy.

   fs write /n/d/acme  {filter  
   {and {not {match .svn}} 
   {not {match -ar '(/appl/.*\.(dis|sbl))|(/sys.*\.*(obj|a|pdb))$'}}} 
   {proto /lib/proto/full} } 

It's very lispish in that we've built a domain language, in this case for tree walking, which we can add to, a command at a time, re-using existing commands in possibly novel ways, and using shell syntax to represent the whole expression.

By combining loadable modules, sh syntax, a uniform interface, channels, processes and files, we have a unique programming environment. This is a powerful pattern that should be repeated.

This kind of programming is now quite unlike any other. It establishes a pattern for structuring applications. It leverages some of the quality ingredients inside inferno.

Another example is the sound synthesizer I've been working on. I'm consciously imitating fs here. I use a simple module extension point with processes communicating on channels, a sh syntax that combines the modules.

Clients implement an interface such as this,

 Source: type array of ref Inst;
 Sample: type chan of (array of real, chan of array of real);
 Control: type chan of (int, array of real);
IInstrument: module {
 init: fn(ctxt: Instrument->Context);
 synth: fn(s: Instrument->Source, 
  c: Instrument->Sample,
  ctl: Instrument->Control);

I defined a module called Expr that operates very similarly to sexprs(2) to handle the shell syntax. Then i can combine the modules in all sorts of ways on the shell command line:

   synth/sequencer {master  
   {fm  {waveloop sinewave} {adsr 0.01 0.21 0.3 0.08} } 
   {delay 0.085 0.4} {delay 0.185 0.2} 
   {delay 0.485 0.1} {delay 0.685 0.08}  
   {proxy {onezero} {lfo 0.18 1.0 0.0} }  } 

We're still not done. We haven't used all the ingredients. Lets throw in Styx.

Lets combine all the above, plus the fact that dis is completely platform independent so we can compile once and run anywhere that inferno emulator runs.

The example that would illustrate this is VN's grid. But I'm guessing since I haven't used it. You can also see all these ingredients combined in sh-alphabet's grid. Sh-alphabet is quintessential Inferno.

From these ideas I'm trying to implement a mapreduce framework. I will use sh syntax to form the expression dynamically, use module extensions to write new map and reduce functions, use Styx to create the mapreduce infrastructure, use dis to distribute code and have channels and files for communicating between processes.

The module extension points are as follows,

Mapper : module {
 map: fn(key, value: string, emit: chan of (string, string));

Reducer : module {
 reduce: fn(key: string, input: chan of string, emit: chan of string);

The mapreduce command is itself a file system that is exported to remote hosts so that clients read and write to it, reading instructions on what to do next, writing back status, so the master can keep track of what's going on. Command usage might be as follows,

mapreduce {reduce {map {reduce {map path mapfn} reducefn} mapfn}  reducefn}

Or using pipeline notation, similar to how sh-alphabet uses it,

mapreduce {map path mapfn | reduce reducefn | map mapfn | reduce reducefn}

Now we're talking. This is Inferno's sweet spot.

This recipe should be pushed HARD in inferno.

I'll stop here, even though I still haven't played the polymorphism card. Sh-alphabet does, but it's a complicated example to swallow.

I recommend programmers start by using fs(1). Then look at how your own programs can be re-worked to factor out modules using a module extension point that uses channels, and then use sh syntax for combining those modules.

The next step may then be to distribute those modules by building into it a styx service.


[1] Actually, there is a slight variation in the library idea in that a library can have multiple implementations, a kind of polymorphism. The Imagefile interface is of this form. This is a slightly more sophisticated library where we load a particular library depending on the kind of operation. But a user of the system is not really expected to extend the system at these points, even though they might. (Other examples are Filter and Encoding)

[2] Another variation on the command polymorphism is that we can bind alternative implementations over our namespace. But in practice this seems to be done rarely. Binding /acme/dis/cd.dis over /dis/cd.dis is one example. Maybe this could be exploited further.