lab 72 - wikipedia

By

NAME

lab 72 - wikipedia

NOTES

I've been working on modifying dict to use the Wikipedia database. I mentioned this in lab 70. So here's what I've got so far. It's not beautiful; the wiki syntax parser needs a lot of work

The general idea is I want to use acme-sac as a Wikipedia browser. But there are other reasons too, such as gaining experience of using inferno to work on some large text databases.

Acme brings some nice things to a database like Wikipedia. Because of the nature of acme you don't have to rely on people making wiki links to find other articles. You can right-select almost any text to search the index. Right-selecting single words often opens a Wikipedia disambiguation page.

If you want to get this working try following the steps below.

You need to use the latest acme-sac copy from svn. It has some fixes to support big files, including Acme.exe, otherwise none of this will work.

Download the Wikipedia database. This site will explain about Wikipedia downloads. Go here for the dump files. For the English version look for a file called something like pages-articles.xml.bz2. This file is about 2.1 GB. Download it and extract it.

This was the first snag I hit. I didn't have NTFS on my laptop drive, only FAT32, so I wasn't able to extract it into a single file. I started extracting it to smaller files and looked at creating a virtual big file using a Styx server; but before I got anywhere with that idea I bought an external hard drive, reformatted to NTFS to handle big files, and just went with the single file approach.

Extract the file somewhere and rename it or bind it to /lib/dict/wikipedia. You then need to build the /lib/dict/wpindex file.

Generate the index inside inferno, 

 % dict/mkindex -d wp > rawindex

I had to step outside inferno for the next bit and used 9pm archive for the plan9 sort and awk commands. Reformat and clean the index entries using /appl/cmd/dict/canonind.awk 

 % awk -F' ' -f canonind.awk rawindex

then sort and remove carriage-returns 

 % sort -u -t' ' +0f -1 +0 -1 +1n -2 < junk |
    tr -d '\r' > /lib/dict/wpindex

Hopefully, you can now type in adict -d wp in acme-sac and in the new window type some text, right-select it and a result from wikipedia will be found.

FILES

svn revision 71

Comments

Post a Comment

Popular posts from this blog

lab 110 - inferno archive edition

By
I've been occupied recently with archiving my digital media. I've been copying home videos on DV tapes to hard-disk, ripping audio CD's to WAV files, gathering photo collections, and trying to copy documents from Iomega disks, floppies, and my dusty old Acorn RiscPC. The plan is to have a copy of this data to give to each of my children. My Dad recently scanned and sent me all his photographs of me and my siblings growing up; he also included pictures of himself and my Mother when they met in Africa. With technology today each generation can build a digital library of family history to hand on to the next generation. In the past a family album may have been passed on to only one person. The accumulation of digital data still presents problems. It requires discipline to store files that are open and not locked into devices or proprietary formats. With digital preservation in mind I've tried to use file formats recommended for long term archiving. WAV files for audio, D...
Read more

lab 111 - wavloop

By
Image
WAV file format contains audio sample data and optionally meta-data that describe the offsets of sample loops and cue points. The loop offsets are used by sampler software to generate a continuous sound, and the cue points mark the point in the sample data where the sound fades away after the note has been released. A WAV file "smpl" chunk will identify the byte sample offset of the start and end of the loop in the sound data. Using wavplay.b as a starting point I tried to loop a sampled sound. My sample data comes from Virtual Organ software GrandOrgue and the sample sets created for it. In this case I'm using the Burea Funeral Chapel sample set. My first test was simply to treat the sample as-is and loop the sound using the given offsets. This did not give good results with a notable noise as the data from the end of the sample joined with the beginning. I realized nearing the end of writing this post that the mistake I made was treating the offsets as counts of...
Read more

lab 107 - midiplay

By
NAME lab 107 - midiplay NOTES Midiplay plays back MIDI files. It uses the synthesizer I described in lab 62 and the MIDI module from lab 73. The command takes only one argument, the path to the midi file. I've included one in the lab to demonstrate. Bind the audio device before using it. % bind -a '#A' /dev % midiplay invent15.mid The synthesizer has 16 note polyphony. It uses three oscillators, one at the pitch, one at half pitch, one at double pitch. There is also a filter, two delays and a vibrato. The sample rate is 8000hz and there is one mono channel (MIDI channel events are ignored). It performs well enough to work on my laptop without JIT turned on. All the synthesizer parameters can only be tweaked from inside the code at the moment. FILES lab - 107
Read more