Tuesday, June 23, 2009

Emacs: P3 Separate But Equal.

In the first post, we went over some basic theory. In the second post we calibrated Emacs's concept of terminal color with the reality of our terminal program. We also created a file called "color_test.el" which is useful for showing the common "faces" used in programming.

I this part we'll explain what a "face" is and show to get terminal faces and display faces to play nice.


A "face", in Emacs parlance, is all the characteristics of a piece of text. This includes it's font, size, whether it's bold or italics and it's color.

In the olden days, faces were created by hand. They're not too bad once you get a hold of them and they're surprisingly flexible.

Emacs faces can do all sorts of snazzy things like auto-detect whether they're on a terminal, change their color if the background color changes and invert themselves if they're on a black and white screen. Here's an example taken from the Emacs Elisp manual, section "Elisp/Display/Faces/Defining Faces".

(defface region
`((((type tty) (class color))
(:background "blue" :foreground "white"))
(((type tty) (class mono))
(:inverse-video t))
(((class color) (background dark))
(:background "blue"))
(((class color) (background light))
(:background "lightblue"))
(t (:background "gray")))
"Basic face for highlighting the region."
:group 'basic-faces)

Alas, you young punks don't wanna do it by hand. You'd rather use Emacs's built in customizer. Fair enough.

The Customizer.

I'm not going into a lot of detail here as there are other web resources dedicated to customizing Emacs.

For the sake of this article we need to know:

  • "M-x list-faces-display" shows you all the faces Emacs knows about.
  • Pressing when your cursor is on the face name will let you edit it.
  • In programming mode, all the fonts we care about begin with the wildly intuitive name "font-lock-".
  • When you're customizing in terminal mode don't forget about "Weight Bold" and "Weight Light". In most terminal emulators they give you extra colors to play with.

Multiple Customs.

If you're using a new version of Emacs, you can go into the customizer, click on the "state" button and select "Show All Display Specs". Then click on "Display" and choose "Check List". This lets you select the modes that you want the changes for. If this gets the job done, then you're done. I've had trouble with edits in TTY mode stomping on my edits in display mode, so I like to keep the variables separate.

After much (and I do mean multiple days) experimenting, I've chosen a more robust solution that is easier to maintain and is a lot harder to stomp on.

Whenever you customize a face and save it, Emacs replaces the function "custom-set-faces" with a new version that has your changes in it. The change is written into your custom file. This file could be the end of your ~/.emacs file or the file named in the "custom-file" variable. I'll use the generic "custom file" because I don't care where it actually is.

The way I handle multiple customizations is to customize them via the customizer. Then load the custom file back into Emacs and rename the custom-set-faces function so it only fires when you're in either terminal or display mode, but not both.

It's very easy to do and it's mostly cut and paste.

Load your Emacs custom file. Before any "custom-set-faces" commands, add these 2 functions:

(defun my-custom-set-faces-display (&rest faces)
"Load these faces if Emacs is in windows mode."
(when window-system
(apply 'custom-set-faces faces)))
(defun my-custom-set-faces-terminal (&rest faces)
"Load these faces if Emacs is in terminal mode."
(when (not window-system)
(apply 'custom-set-faces faces)))

Now look at your current "custom-set-faces" command. Is it set up for display mode? Then rename it to "custom-set-faces-display". If it's for the terminal then rename it "custom-set-faces-terminal". Now add an empty function call for the "other" function. If you set custom-set-faces-display, then add "(custom-set-faces-terminal)". If you set custom-set-faces-terminal, then add "(custom-set-faces-display)".

Mine looks like this:

((t :foreground "LightlyDepressed" :weight bold)))
'(font-lock-comment-face ((t :foreground "cyan"))))


Now fire up Emacs in terminal mode (emacs -nw) and edit a face. I'll make the font-lock-comment-face "Naval" colored for this example. Then save the change. Take a look at your custom file and you should see the color change in the function "custom-set-faces". Here's my example:

((t :foreground "LightlyDepressed" :weight bold)))
`(font-lock-comment-face ((t :foreground "cyan"))))
;; custom-set-faces was added by Custom -- don't edit or cut/paste it!
;; Your init file should contain only one such instance.
'(font-lock-comment-face ((t (:foreground "naval"))))
((t :foreground "LightlyDepressed" :weight bold))))

Delete the old custom-set-faces-terminal. Rename custom-set-faces to custom-set-faces-terminal, save your work and you're done.

If you wish to edit your display faces, just fire Emacs up in display mode and run through the same process.

There's no limit to the number of face sets you can add. You can have a different face set for every day of the week if you want. Just create a "my-custom-set-faces-" for any discriminator you want and rename custom-set-faces to match it.

Hopefully my absurd 10 day journey into Emacs's faces has been rendered down into something useful for you. Let me know if you found this helpful.

Monday, June 22, 2009

Emacs: P2 Color Me LightlyDepressed.

Now that we have some color theory under our belt, let's calibrate Emacs's concept of color with the reality of the terminal's.

First we have to get the real colors being displayed. I'm using gnome-terminal which has a built in color picker. If you're using a terminal that doesn't have it's own color picker, fire up "emacs -nw", do "M-X list-colors-display", then use something like "xmag" or gimp to get the color values.

My Color List.

From a gnome-terminal, select "Edit/Current Profile" from the menu. From the "Default" screen, click on the "Colors" tab. At the bottom of the screen you should see 2 rows of 8 colors. The first row is the 8 colors that make up the terminal's pallet. Left most is entry 0, right most is 7. The second row are the colors you get when you print the first row using "bold". Gnome-terminal has a 3rd row of colors that are the first row in "dark" mode, but you can't edit them.

If Emacs was smarter about terminal colors you could tell it about all 3 rows of colors and it could use "bold" and "dark" version to increase the chance of it's making a good color choice. Alas, were stuck with our one row of 8 colors.

Click on each color in order, and write down their Red, Green and Blue (RGB) values. For example, the 4th color in my pallet is kind of brown, with yellow below it. It's RGB value is 170/85/0, so pallet entry 3 is 170/85/0.

This is my list:

0 0 0 0
1 170 0 0
2 0 170 0
3 170 85 0
4 0 0 170
5 170 0 170
6 0 170 170
7 170 170 170

Once you have all 8 values, ask Emacs (in a terminal) for help on the variable "color-name-rgb-alist" (C-hv color-name-rgb-alist). The help should list all the color names that Emacs knows and their RGB values.

Scan the list for colors that match the gnome-terminal colors. If you find a *perfect* match, put the color's name besides it's color in your list. Only use the name if it's a perfect match. 0/0/0 was the only match for me. I labeled color 0 "Black".

For the rest of the colors, give them descriptive names that are not in color-name-rgb-list. The last thing we need is 1 name for 2 colors.

Here's my final list.

Black 0 0 0 0
Brick 1 170 0 0
Greeny 2 0 170 0
Brownish 3 170 85 0
Naval 4 0 0 170
DarkishMagenta 5 170 0 170
NeonPee 6 0 170 170
LightlyDepressed 7 170 170 170

Now we have to get the colors into Emacs. It turns out that that's pretty easy.


First, make a backup of your ~/.emacs, just to be safe. As an aside, because this series isn't nearly long enough, consider using RCS to backup any config files that you hand edit. Under Emacs RCS is trivial to set up and use. It's saved my monkey boy butt more times than I care to remember.

To set up RCS for your ~/.emacs, make a directory called ~/RCS. Then load your ~/.emacs file into Emacs. Hit C-xvv. That's it. You're done. Your ~/.emacs is now write protected and checked into ~/RCS. To check out your file so you can edit it, load it into Emacs and hit C-xvv.

Back to work.

Edit your ~/.emacs, and add the following code. If your ~/.emacs has a custom-set-variables or custom-set-faces function, place this code before either. Obviously you should use your own colors and names for the my-tty-color-define-8 commands.

;; Code for handling term based Emacs.
(defun my-tty-color-define-8 (name index rgb8)
"Set the tty pallet using 8 bit rgb values."
(tty-color-define name index
(mapcar (lambda (x) (+ x (* x 256))) rgb8)))

(if (and (not window-system) (= 8 (length (tty-color-alist))))
(my-tty-color-define-8 "Black" 0 '(0 0 0))
(my-tty-color-define-8 "Brick" 1 '(170 0 0))
(my-tty-color-define-8 "Greeny" 2 '(0 170 0))
(my-tty-color-define-8 "Brownish" 3 '(170 85 0))
(my-tty-color-define-8 "Naval" 4 '(0 0 170))
(my-tty-color-define-8 "DarkishMagenta" 5 '(170 0 170))
(my-tty-color-define-8 "NeonPee" 6 '(0 170 170))
(my-tty-color-define-8 "LightlyDepressed" 7 '(170 170 170))))

Save your ~/.emacs file and exit. Restart with "emacs -nw" Type "M-x list-colors-display". You should see your color names listed with the colors.

This might not seem like much of an achievement, but you've actually taken a pretty big step.

To check our your results, create a file called color_test.el in Emacs (-nw). It should put you into "Emacs Lisp" mode automatically. Now type in this program:

;; Comments are in 'comment-face'.
;; defun and defvar are in keyword-face.
(defun function-name-face (&optional is-in-type-face)
"string-face `constant-face' string-face"
(error "warning-face"))
(defvar variable-name-face)

;; To see the "doc-face" go into "perl-mode".
This should be in doc-face.

The program itself doesn't work. It's not even syntactically valid. All it exists for is you show all various "faces" that Emacs uses when coloring code.

To see doc-face, use "M-x perl-mode".

How do you like them colors? If you're happy happy, then you can skip post 3 of this series. If, however, you're like me and think that Red is a horrible color for comment text, then await with baited breath the last installment of the Emacs color saga.

Sunday, June 21, 2009

Emacs: P1: What Color Is My Painbow?

Last week I had a classic "Monkey Boy" moment. I decided to adjust the colors in my text editor. 10 days later I'm finishing a 3 post blog on it.

I worry myself some days.

This first post is going to be mostly theory work. Post 2 and 3 are more hands on.

Laying the Ground Work.

I use an editor called Emacs for most of my programming. It's an old editor, but its one of the most powerful editors out there. It also lets you edit files in display (windows) mode and from the shell (command.com for you Windows folks).

Now days most of the editing is done in display mode. No real surprise there. However, there are times when working from the shell makes more sense.

I routinely log into distant machines across slow connections. I could pop up a virtual session and wait for the window in the virtual session and then wait for the editor in the window in the virtual session and then wait for the file in the editor in the window in the virtual session, or I can use text mode, where are complete screen refresh is around 2000 bytes.

I hate waiting. Its a no brainer.

The down side of terminal mode is that you can only use characters to draw and you have a limited number of colors. Both of these could be overcome with modern technology, but it ain't going to happen so we have to get used to it.

Why do you have limited colors? Well, the underling technology differences between a terminal from 20 years ago and a modern graphic display is pretty significant.


Colors are made by mixing various amounts of Red, Green and Blue (RGB) together. If you crank up the RGB, you get bright colors, dial it down and you get dark. Wikipedia has a nice write up on color depth so I won't go into it here. The only thing you need to know is that by adjusting the RGB values you can change colors.

On modern display you have absolute control of every dot on the screen. Each one has it's own RGB setting which is independent of it's neighbor.

Old school color terminals were more like "paint by numbers" projects. You were given a pallet of colors (usually 8) that were hard wired into slots. If you set the color to pallet slot 0 and then printed, you got black text. If you printed in color 4 you might get blue. Unfortunately for us, these are the terminals that most terminal emulators emulate.

We have two problems when we want use Emacs in both terminal and display mode: First is that Emacs's support of terminal colors is functional, but not much more. The second is that the friendly Emacs customizers don't like it when you're a switch hitter. In fact they gets down right medieval on your monkey butt. Well, this ain't monkey butt, this is monkey boy butt. Accept no substitutions.

Terminal Colors

As I said before, most terminal emulators model the old style, 8 color pallets. There are ways for a program to ask the emulator for the number of colors available, but there isn't any way to get the actual RGB of each color.

What does Emacs do? It guesses! If you don't tell it otherwise Emacs assumes that you have an 8 color pallet with the following colors:

Slot Name Red Green Blue
---- ------- --- ----- ----
0 black 0 0 0
1 red 255 0 0
2 green 0 255 0
3 yellow 255 255 0
4 blue 0 0 255
5 magenta 255 0 255
6 cyan 0 255 255
7 white 255 255 255

The numbers after the colors are how much Red Green and Blue that each color is supposed to have. 255 is the largest number you can express in 8 bits (1 byte) of data. There are places internally where Emacs uses 16 bit (2 byte) RGB values which go from 0 to 65535. I got bit by this more than a few times so I'll try to point them out, or gloss over them when I can.

Emacs cares about the RGB values because you (the user) set colors by name not slot values. If you set the color of something to "CadetBlue1" 152/245/255, then run in terminal mode, Emacs needs to figure out which of the eight colors CadetBlue1 is closest to. It uses the RGB values to figure it out.

Oh, by the way, the name "CadetBlue1" comes from a variable called "color-name-rgb-alist". To see it's contents, fire up Emacs in display mode and type "M-x list-colors-display". You'll see the colors and their names.

Let's do some hands on. From a terminal, type "emacs -nw". It should start an Emacs session in the terminal. In Emacs type "M-x list-colors-display". You'll get a listing of the 8 colors that Emacs knows about. Note: On some systems you get more than eight. Lucky you. The theory is still the same.

If you're like me, you notice one thing first off. These colors look nothing like their names! The Red might be brick colored. Yellow may look brown. And my white has tattle tale gray! What happened?

Easy. Emacs has no idea what colors your terminal's pallet is set to and it's guess stinks. How do we handle the miss-match?

One option is to change our terminal to Emacs's pallet. Then we can vomit and claw our eyes out. Basic colors tend to be rather harsh on the psyche.

The second option is to tell Emacs what our terminal is really packing. That's the subject of the next post.

Monday, June 8, 2009

Going Deaf at Duffs

Whenever anyone from the company has to come up to Buffalo on work, we have to feed them.

This being Buffalo, and we being cliches, we inevitably drag them off to Duffs.

The thinking goes like this: Person A comes to Buffalo. Person A must want to try our local cuisine. Our local cuisine consists of Chicken Wings and Beef on Weck. Monday is Chicken Wings, Tuesday Weck. Today is Monday. Real men like hot Chicken Wings. Duffs' wing are really hot! We go to Duffs.

Never mind the fact that there is more to Buffalo than Chicken Wings. Ignore that fact that Person A may be here for is 10th time this year. Obfuscate the fact that hot wings and good wings independent variables. Me Buffalo, Me Wings, Me Hot, Me Duffs. *Sigh*

OK. I don't have anything against Duffs. Alright, it does remind me of a low rent vomitorium and the video games run on diesel, but besides that it's a nice enough place. I'd just like to see a little more depth in our chow.

But that's not why I'm writing today's blog.

The last time we were in Duffs, a person came up to me and handed myself and a couple of other people at the table "deaf cards".

For those of you who don't know, a deaf card is a card that allegedly deaf people hand out at airports as a way to beg for money.

The card usually follows a certain formula: First the introduction "I am a deaf person.", then the pitch "10 bucks would make me feel better about being deaf.", then a blessing "May god bless you for giving me 10 dollars." and then a graphic. Usually the graphic is something like a cartoon angle or a peace sign. I got a smiley face.

The first thing that bothered me is, I'm no where near an airport. We got rules! You street beg on streets, airport beg in airports and PBS beg on the radio. It's all part of the begging ecosystem. What's next, Hare Krishna telethons?

Also, I don't know that the person's deaf. I have no problem giving help to the needy. I'm well aware that with a disconcertingly small number of bad breaks, I could be out on the street. This guy is Alpha/Omega. Either I do good by helping someone out, or I'm encouraging a rodent to make a dishonest living by pretending to be deaf.

Then I notice that everyone at my table is whispering. Whispering? Why would you whisper around a deaf person? We're surrounded by the hearing? If he's stone deaf, like he's professing, then he can't hear us. If he can hear us, then he belongs in the slammer.

And if we're afraid of people hearing us, then why aren't we whispering around the people at the tables all around us? We've been blabbing for an hour. They've heard every word!

Then I had the big epiphany. Why would I give someone $10 for being deaf? This isn't like the 1820s where deaf people starved on street corners. In modern societies there are very few jobs that aren't accessible to the deaf. We have the technology and, I'd like to think, are more enlightened about deafness. Rare is the person who thinks it's a punishment from god.

I'm not saying there still isn't ignorance, I'm just saying it isn't the albatross it once was.

Thinking about it, where I work, every jobs in the building save phone receptionist and security guard could be handled by someone stone deaf. And even that, the security guard who monitors the cameras would have no problems.

Knee jerk simpletons may pretend that I'm picking on deaf people here. They're seeing what they want to see. I freely concede that someone who has a sense, and looses it, suffers. If they have an ability, they use an ability, they loose an ability, they have to adjust their life around the loss. I just don't see where deafness would make an otherwise healthy person into a beggar.

In some cases the loss is tragic. Beethoven never heard is final symphonies. In other cases, not so much. A Sumo wrestling rarely depends on the sense of smell. Neither would require you handing out cards.

On the other hand, what about people who are born deaf? They'll never hear music, but I'll never see magnetism. Am I missing out? No idea. And maybe the born deaf experience a clarity of though and tranquility of mind that I'll never know in my noise filled head. Again, no idea. Either way it ain't worth $10.

Because I pointed out that the deaf shouldn't be pitied, everyone at work now thinks that I'm a kitten burning poltroon of the worst stripe. Far from feeling like a wag, I think that I'm being more enlightened than most.

I also noticed that none of them gave the beggar any money. Ya' hear what I'm saying?

Wednesday, June 3, 2009

Aw C'mon, be a Minion!

Come my minions! Gather 'round me and do my bidding!!!

Nuts! That never works.

I'm starting a minion list. It's not a list of minions, it's a list of projects for my minions.

I'm going Internet because minions are sparse on the ground around these parts. I believe it's because my readership is made up of selfish swine who won't surrender their sense of personal identity for the betterment of me. But I press on! I'm brave that way.

My minion list is a list of projects, usually computer projects, that strike me as a good idea, but I don't have time to do them. It would be nice if a minion would do them (hint, hint).

I've talked about the minion list in the abstract many times, but now I've decided to codify it. That way, when someone else commercializes one of my ideas, say "Feels on Wheels" (I'll explain later), I can sue them for plenty cash.

Wanna Taste?

My first entry for the minion list is "Voice Messaging". Voice Messaging is like Instant Messaging, augmented with sound.

The concept is simple. A person wants to send you a message. Instead of typing, they have the option to record a message. The message is sent to you, along with any typing they wish to do. On your side your VM program converts the sound file to text and displays it like a normal IM. If the speech conversion garbles it too badly, you just highlight the part you can't read and it plays that part of the sound file.

This would rid the world of emoticons. If you want to know how someone is feeling, play the sound file.

I Object!

* Speech recognition sucks.

You're right as far as you take it. However, in this case we're not producing a finished document. It doesn't have to be perfect, just readable. The recognition only has to be gud enuf 2 undrsAnd. If you're unsure what was said, play the sound file.

* What's to stop someone from saying one thing, but typing another?

The conversion is done on the receiver's end. The sender can send text along with the sound, but the final arbiter of the text is the receiver.

* What's to stop someone from screaming obscenities or pulling other "funny" jokes.

Before playing a sound file, the program would normalize the sound levels. It could even warn you if the original has loud points in it.

* Couldn't someone just use the phone?

Yea, but VM is closer to IM than a phone call. The sender can send a VM, but you can ignore it until you're ready, just like IM.

* What about people who don't stop yakking?

Put a size limit on how big you'll receive. Make it settable per person. Even if someone does get carried away, it's still better than voice mail.

Almost any sound player lets you replay selected parts of a file so you can deal with the witlings who leave 5 minute messages and wait till the end to mumble their contact information.

* What file format should we use?

This is getting a bit low level, but something open format is an obvious requirement. It turns out that there is an open format called Speex that gets the job done.

A 14 second PCM, 16 bit, mono 48000 Hz .wav file is 1.4 meg. Convert it to an .mp3 shrinks it to 123K. Speex mauls it down to 66k. All with no real loss of sound quality for what we're doing.

* What about security?

As long as an open audio format is used, and you trust your audio player, then you're all set. If you use an audio player that does odious stuff like pop up web browsers (oy!) then you're asking for trouble. Playback should be based solely on the contents of the file, not it's extension.

* What technology should the be built on?

I'd bet that most IM protocols allow file transfer. It would simply be a case of the client programs handling audio file differently.

I Submit Like the Schweinhund I Am!

You have convinced me oh minion master. I supplicate at your feet. What is the first step?

For a start, someone could write a standalone application that reads Speex files and transcribes the contents. Some college out there has to have information on phonetic translation. Lets see what the state of the art is.

After that, add playback of selected sections. Once we have that settled then the rest would be cake, and how many masters let their minions have cake?

Tuesday, June 2, 2009

Sound Bite Me!

One of the true pleasures of programming is finding some niggling little problem and solving it with a simple bit of code. If you can solve it in a couple of hours, even better.

When I blog I tend to speak what I'm typing. If it doesn't sound right when I say it, it probably wont sound right when you read it.

One of the problems I have when I blog is that I talk faster than I type. I talk faster than I think. I start typing, and then I get a flash of inspiration. I run the idea through my head, and then, half a virtual page later, I realize that I haven't written anything down.

Then I have to try to recreate the idea from memory, but the flash is gone. By the time I've rebuilt it, or accept that I've forgotten it, my original thought is out playing in the yard and won't come back.

It frustrates the hell out of me.

I started playing around with ideas to making blogging easier.

At first I tried to check out the state of computer speech recognition. I figured I'd just blab on in my blog, and then I'd go through and clean it up by hand.

Computer speech recognition is still slow, expensive and it still sucks. Trying to run a editing session without using a keyboard is slower than typing with 2 fingers. I also wanted to blog via Linux, so firing up a Windows product ain't getting the job done.

Next it tried to do some sort of integration of speech and text together.

I envisioned loading a sound file into a sound editor, where I could chop it up and move it around. While editing the sound, I'd join text to it. When I moved a blob of sound, text that went with it would move to. Eventually I'd piece a blog out of all my rattlings on.

I still think that this is an interesting solution, but man, it would be some work! I also I think I'd end up with a crappy sound editor linked to a crappy text editor. No dice.

I also had a minor epiphany. I'm not going to keep these sound files around forever. I'm just brain dumping to a file for a few minutes until I've finished typing my original though. After that I can replay the recording and transcribe anything I think is useful.

I already know how to record from a mic on my Linux box. Adding that to my new insight I wrote a shell script that turns on the sound recorder, dumps the contents of the microphone into a file and, when I stop recording, plays it back (that way I can tell if I forgot to turn on the mic or something).

It worked like a charm! Every time I needed to make a note, I just fired up the script, decided on a name for the sound file and away I went. It was a little awkward, but a big step forward.

I called the script "sound_bite".

After than I needed to come up with a way to play back my sound bites, so I started adding flags to play the last sound bite or the first sound bite or list the sound bites and let me pick. Then, another epiphany! They're just frigging .wav files! Maybe I could just double click on them in my file manager. Oooo. Me one smart monkey!

Actually, once I got the file manager into the game, it cleaned up a lot of code. I didn't have to tell the recorder where to put the sound files, I would always put them into the same directory and give them a time stamp for a name. If I wanted to organize them better, I'd use the file manager to rename them or move them elsewhere.

The only thing left was making it easier to use. Typing in the command every time is a minor pain. I needed a quicker way to access it. That was easy too.

I hooked up sound_bite to a shortcut which put all the .wave files into the directory "sound_bites". I made another shortcut to bring up the file manager in "sound_bites" directory. I'm sure I could come up with a few dozen little tweaks, I know enough to stop typing when I'm done.

I'm now an official audio driven blogging fool!

Here is the entire source code for sound_bite. You may have to play with it a bit because the blogger code likes to play with it.


#!/bin/sh -

# Sound_bite: Written by Dale Wiles 6/2/09.

# Exit if an error occurs.
set -o errexit

if [ $# -eq 0 ]; then
  cat <<EOM
Usage: $0 directory

Move to DIRECTORY and start recording a wave file from the microphone.
The name of the wave file is yymmdd_hhmmss.wav.
  cd "$sound_dir" || exit 1

  # Make the output name based on the time, down to the second.
  # That way I can't overwrite existing files.
  # Alright, in theory during daylights savings time it could
  # overwrite.  I've added code for that almost impossable situation.
  while :; do
    out=`date +%y%m%d_%H%M%S`.wav
    if [ ! -e $out ]; then
    echo "Waiting...."
    sleep 1

  echo "Recording $sound_dir/$out"
  sox -t alsa default -v 7 $out
  echo "Playing $sound_dir/$out"
  aplay $out