Showing posts with label Programming. Show all posts
Showing posts with label Programming. Show all posts

Monday, August 1, 2011

Hey Unix, How About a Date?

The Problem.

I needed a way to warn users that their passwords were about to expire. To give them a sporting chance, I wanted to give them 2 weeks warning.

I could get the current date with no problem. Getting the password expire date was a bit of a pain, but I could figure it out. The hard part was figuring out how close to the 2 week warning date they were.

In a nut shell, doing math with dates is a royal pain. You have abstract concepts like “July 4th” permanently glued to a large spinning rock which is whizzing around the Sun.

Also, certain concepts that we're used to, such as 1 + 1 = 2, don't always hold up when doing date math. January 31st + 2 months = March 31st. No surprise. How about January + 1 month. That would be February 28th? Or would it be March 3rd (31 days after January 31st)? Add a month to February 28th and you get March 28th. March 31st doesn't equal March
28th. Oh boy!

The Simple (Linux) Solution
.

The most common way to deal with dates is to convert them to a number, do some math, and then convert the number back to a date. Unix (and Linux) system time is based on this concept. Unix dates are implemented as the number of seconds since the January 1st, 1970. In Unix parlance 1/1/1970 is called “the epoch”.

If you have a version of Unix that uses the GNU version of the date command (almost all versions of Linux do), then date math becomes trivial. The date command can convert to and from the epoch with relative ease.

To convert “August 1st 2011” to seconds from the epoch use:

date -u -d "8/1/2011" +%s

You should get 1312156800.

To convert it back use:

date -u -d "1970-01-01 1312156800 seconds" '+%m/%d/%Y'

You should get “08/01/2011”.

The problem here is 2 fold:

One, you're limited to dates between 1970 and 2038 for 32 bit computers. If you have a 24 bit computer then you're good until somewhere around the year 292,277,026,296 so it's not really a restriction.

The second problem is that most older OS's aren't running GNU date. They have their own propriety versions of date that won't let you work with arbitrary dates.

What I needed was a date converter that would work on many versions of *old* Unix. Things like Solaris 5 and HP-UX 10. These are nasty little beasts that barely have Bourne shell. I also wasn't allowed to add more advanced scripting languages to the system so Perl and Python solutions were both out.

I poked around on the Internet tubes and found a dearth of solutions. Most of them used other languages. Some gave example code that didn't handle leap years properly. Others were built around precomputed tables.

Time to step up to the plate.

My Solution.

Below is my solution. Its date format is the number of days after January 1st, 1582. That's the beginning of the Gregorian calendar and very few of my users are that old.

Internally it's mostly AWK scripts glued together by Bourne shell. It can handle dates up to 1/1/9794 and can probably go higher. I've tested it and think it's pretty bullet proof.

#!/bin/sh -

# Convert a date to/from the number of days after 1/1/1582 using only
# basic Unix commands. By "basic" I mean commands available on an
# HP-UX 10 box.
#
# 1/1/1582 is the start of the Gregorian calendar.

# Note: The Gregorian rules for leap years is:
#
# If the year is a factor of 400
# It's a leap year.
# Else If the year is a factor of 100
# It's not a leap year.
# Else If the year is a factor of 4
# It's a leap year
# Else
# It's not a leap year.

# This mostly uses awk because awk is much faster than using raw
# Bourne shell.

# To get a date from the Unix "seconds from the epoch" time use
# int($utime / (24 * 60 * 60)) + date_as_days(1970 1 1)
#
# date_as_days(1970 1 1) = 141714 by the way.

#
# Return the number of days in the previous months.
#
# For example the second entry is the number of days in January. The
# third entry is the combined number of days in January and
# February.
#
# The only parameter is the 4 digit year.
#
days_prev_month()
{
  echo $1 | awk '{
    year = $1

    # Pick the number of days depending of whether its a leap year.
    if ((year % 4 == 0) && (year % 100 != 0 || year % 400 == 0)) {
      print "0 31 60 91 121 152 182 213 244 274 305 335 366"
    } else {
      print "0 31 59 90 120 151 181 212 243 273 304 334 365"
    }
  }'
}

#
# Convert a date in to the number of days after 1/1/1582.
#
# The parameters are YYYY MM DD.
#
date_as_days()
{
  dad_month=$2; dad_day=$3

  # Get the number of days consumed by the years and the number of
  # days remaining in the current year.
  set - `echo $1 | awk '{
    year = $1

    # The modern calendar started in 1582.
    year_days = int((year - 1581) * 365.25) - 365

    cents = int((year - 1501) / 100)
    year_days -= cents

    cents_400 = int((cents + 3) / 4)
    year_days += cents_400

    print year, year_days
  }'`
  dad_year=$1; dad_year_days=$2

  # Now add the month and day contributions.
  days_prev_month $dad_year | awk "
    BEGIN { day=$dad_day; month=$dad_month; year_days=$dad_year_days }"'
    { whole_month_days=$month
      print year_days + whole_month_days + day - 1 }'
}

#
# Take the number of days since 1/1/1582 and convert it to
# year, month, day
#
days_as_date()
{
  df_days=$1

  # This awk script computes the year contributions to the date
  # and removes the effects of those years from df_days.
  set - `
  echo $df_days | awk '{
    df_days = $1;

    # The first 400 year leap year in the Gregorian calendar is
    # 1600 so we normalize our calculations from the first block
    # of 400 years that ends on 1600. That year is 1201.
    # There are 139157 days between 1/1/1201 and 1/1/1582
    #
    # Note: We use 1201, not 1200 because we want the leap year to be the
    # *last* year of the 400, 100 or 4 year block.
    n_days = df_days + 139157

    # There is one leap year every 4 years.
    days_per_quad_year = (365 * 4) + 1
    # Years that end in 00 arent leap years.
    days_per_cent = (days_per_quad_year * 25) - 1
    # Unless its divisible evenly by 400.
    days_per_quad_cent = (days_per_cent * 4) + 1

    # Calculate the contributions of each year block.
    quad_cents = int(n_days / days_per_quad_cent)
    n_days -= quad_cents * days_per_quad_cent

    cents = int(n_days / days_per_cent)
    if (cents == 4) { cents = 3 }
    n_days -= cents * days_per_cent

    quad_years = int(n_days / days_per_quad_year)
    n_days -= quad_years * days_per_quad_year

    years = int(n_days / 365)
    if (years == 4) { years = 3 }
    n_days -= years * 365

    df_year = 1201 + (400 * quad_cents) + (100 * cents) \
      + (4 * quad_years) + years

    print n_days, df_year
  }'`
  df_n_days=$1; df_year=$2

  # Get the day and month from the given year.
  set - `days_prev_month $df_year | awk "
    BEGIN{n_days=$df_n_days}"'
    { df_month = 1
      while (n_days >= $df_month) {
        df_whole_month_days = $df_month
        df_month++
      }
      df_day = 1 + n_days - df_whole_month_days
      print (df_month - 1), df_day }'`

  df_month=$1; df_day=$2

  echo $df_year $df_month $df_day
}

# Some test code. Feed it a number and get back a date.
# Feed it a m/d/yyyy date and get back a number.
#
# Note the complete lack of error checks.
if echo "$1" | egrep '/' >/dev/null 2>&1; then
  set - `echo $1 | tr '/' ' '`
  date_as_days $3 $1 $2
else
  days_as_date $1 | awk '{ printf "%02d/%02d/%02d\n", $2, $3, $1 }'
fi

Sunday, June 19, 2011

Python, GTK, GL and Incantation.

In computer terms, an “incantation” is a problem that can easily be solved if you know the proper mystical invocation. An incantation is especially annoying if it's something that has to have been done many times before, but no one as bothered to tell we mere mortals how they did it.

Linux suffers incantation at a near mystical level. People who write library code seem to take a sadistic pleasure in not telling people how to use said library.

I ran in to this in spades when I tried to use Python to hook up the OpenGL graphics library to the GTK widget system. I didn't have too much trouble finding examples of hooking GTK to Python, the PyGtk libraries are full of examples. I didn't have too much trouble finding OpenGL examples in Python. Search for “PyOpenGL Demo” and you'll probably find what you need.

My problem came from trying to hook Python, Gtk and OpenGL together as one big happy family. I couldn't find one complete example on the net. I could find some pieces, but no complete example.

To make a long story short, I finally found enough pieces, and read enough source code to have a pretty good idea of how to get the job done. In true Linux fashion it turns out to be pretty easy. The documented code is at my website. Hopefully there is now one less incantation in the world.

Monday, November 8, 2010

Hideous Bloat and Active Links.

By popular demand I've expanded my Emacs wiki from one function to an unwieldy two. The second function will scamper through a buffer and list all the active links found.

I use the same regexp that I did in the previous article so it links on CamelCase. To change the link style you just change the regexp. If you end up writing too many support functions then you may want to use that "variable" thing that all the cool kids are talking about.

The Code



(defun list-active-links (link-re &optional is-active-link-p)
"Scan a buffer looking for links and list all the active links.

LINK-RE is a regular expression which matches the link text.
IS-ACTIVE-LINK-P is an optional function which takes the link text and
returns true if the link is active. If not provided, `file-exists-p' is
used."
(let ((buffer-name "*ActiveLinks*")
(found-links ()))

(if (not is-active-link-p)
(setq is-active-link-p 'file-exists-p))

;; Gather up all the potential links and whether they're active.
(save-excursion
(goto-char (point-min))
(while (re-search-forward link-re nil t)
(let ((link-text (match-string-no-properties 0)))
;; If the link hasn't been checked, then save its value
;; and whether or not it has an existing destination.
(if (not (assoc link-text found-links))
(setq found-links
(cons (cons link-text
(funcall is-active-link-p link-text))
found-links))))))

;; Now list out the active links.
(if (get-buffer buffer-name)
(kill-buffer buffer-name))
(switch-to-buffer-other-window buffer-name)
(let ((active-links
;; Remove any links that don't have an associated file.
(delq nil (mapcar (lambda (x) (and (cdr x) (car x)))
found-links))))
(insert "#\n# The following links have destinations.\n#\n")
(save-excursion
(insert (mapconcat 'identity (sort active-links 'string<) "\n")))
(not-modified)
(message "%d matches found." (length active-links)))))

(defun check-active-links ()
(interactive)
(let ((link-re "\\<[A-Z][a-z]+\\([A-Z][a-z]+\\)+\\>"))
(list-active-links link-re)
;; Note: At this point we're in the "*ActiveLinks* buffer.
(set (make-local-variable 'link-to-re) link-re)
(local-set-key
(kbd "C-c C-o")
(lambda ()
(interactive)
(link-to link-to-re "\\<")))))

Sunday, November 7, 2010

Emacs, the Wiki and the Idiot.

Sir C.A.R. (Tony) Hoare once wrote:
Inside every large problem, is a small problem—struggling to get out.
Someone else, stealing from H. L. Mencken, changed it to:
Inside every large problem, is a small problem—struggling to get out. The solution to that problem is simple, elegant and wrong.
I tend to agree with the Thief. The simple reality is that many problems don't have elegant solutions, try as we may to find them. I ran into this when I tried to come up with a simple personal wiki. Enjoy my exploits.

Dale Writes a Wiki


I think the wiki is one of the great software concepts of the last decade or so. That's not hyperbole, I really think that making it easy to link documents together, in a casual and non-invasive way, is a brilliant and under-appreciated concept.

Most wikis are web based. This makes sense if you want the world to have access to the pages. I was looking more for a personal solution.

There are personal wikis that are stand alone applications, but most of them use a special format for your data. This strikes me as, well, stupid. Why would I lock up my data in a special format when a directory of text files should easily get the job done?

I do a lot of my text editing in Emacs, so I started looking around for an wiki based in Emacs. There are a few, but most of them are either unsupported, too elaborate or too invasive. I wanted a way to link text from any kind of file, not just wiki files. I wanted to be able to link comments in source code to documentation files to cake recipes if that's what works best. I also wanted it to leave the rest of my Emacs environment alone. I didn't want to enter Wiki-Mode just to put links in a Perl program. I wanted to stay in Perl mode.

The joy of being a programmer is, if you can't find it, you can always write it. That's what I started out to do.

At first I decided to play with [[bracket style links]]. Emacs Org Mode uses them and I like much about Org Mode, so I tried to emulate its style.

The attempts where not too successful. I pulled out some of Org Mode's handler code and started playing with it. It's pretty invasive and doesn't work well with old versions of Emacs. I use an old version of Emacs at work, and I wanted something small and easy to understand.

My next idea was to write my own bracket style linking library. At it's heart it would grab the text between [[]], convert it into a file name and then open the resulting file name. 194 lines of Elisp later and I had a working minor mode. It could open files, web pages, internal links and even run commands. I was very happy with it. It was reasonably small and reasonably easy to understand and reasonably noninvasive.

Because I'm a professional, I started to document my results. The problem is, I'm a little bit insane. Sometimes, when I'm working on a project, a little idiot voice calls from the fog of experience and tells me what I'm doing is wrong. The Idiot never tells me what's right, it just picks at the back of my head until I accept, eventually, that my perfectly working code is "wrong". Arg!

The more I documented, the louder the Idiot got. I got so frustrated that I put the code aside. I'm a big believer in documentation by Idiot. You take your code and ignore it for a few weeks or even months. Then you come back and try to read it. Every time to look at a bit of code and say "What idiot wrote this?" you either re-factor or document. Maybe the Idiot could figure out what I couldn't.

The Idiot was insidious. It would go away for a week or so and then come out of the shadows to jeer hints. It asked questions that I should have asked. Do I need brackets? Do I need to open web pages and run commands from links? Are my needs the needs of others? What's more important, abstract power or agility? What is the DAO of the problem? Why won't you see it!?

This went for over a month. Part of me was trying to find the right solution. Part of me wanted me to finish what I had and get on with my life.

Then, on a Sunday night, as a long work day loomed ahead, I was laying in my bed with my beloved wife and Charlie the metal eating dog. From nowhere, it came to me. All I really want is a way to take the word under the cursor and open a file with the same name. Then it came to me. What I really want is a way to grab an arbitrary blob of text under the cursor and then open up a file based on the text. Then it came to me. What I really want is a way to grab an arbitrary blob of text under the cursor and do something with it. The Idiot had spoken!

It's a cliche, but I really wanted to leap out of bed and start hacking away. Alas, the days of 12 hour hackfests followed by 10 hour workdays are a thing of the past. I had to go to sleep or I would die at my desk. I had to go to work or I would be thrown into the street. I had to get this program written or I would crack up.

After work I lit in to my task. One of the greatest pleasures in programming is simplifying. The more I wrote, the smaller the program became. Irrelevance fell like rain. I was beginning to understand. By the end, the entire program is 15 lines long. 24 lines if you include the code to hook it up to the key of your choice.

It was simple, elegant and worked like a charm. The Thief was wrong. The Idiot stopped jeering.

Hooking Up the One Function Wiki

The simple solution is to take the text at the end of this article and append it to your .emacs.el file.

To test it, fire up Emacs, and type "This is CamelCase text.". Move your cursor to the link text "CamelCase" and type C-cC-o. It should open up the file "CamelCase".

If you want to use other link styles besides CamelCase or want to open files in different way, read the documentation. Doing things like opening files in a specific directory or opening "CamelCase.txt" are trivial to implement. You just need to do a little Elisp programming and away you go!

The Text to Append.

(defun link-to(link-re link-start-re &optional handle-link)
"Grab the \"link\" under the cursor and open a file based on that link.

LINK-RE is a regular expression which matches the link text.
LINK-START-RE is a regular expression which matches the beginning, or text
just before the link.
HANDLE-LINK is an optional function which takes the link text and opens the
link. If not provided, `find-file' is called with the link text.

This function is usually called via local-set-key with mode specific
regular expressions.

This example will grab a CamelCase link and open a file with the same
name.

(global-set-key (kbd \"C-c C-o\")
(lambda ()
(interactive)
(link-to
\"<[A-Z][a-z]+\\\\([A-Z][a-z]+\\\\)+>\"
\"<\"))) Note: The \"<>\"s above should have \"\\\" in front of them, but emacs
thinks I want to print a key map when I try to include them in the help
text.

This example will grab an alphabetic string and do a google query on it.

(global-set-key (kbd \"C-c C-o\")
(lambda ()
(interactive)
(link-to
\"[a-z]+\" \"[^a-z]\")
(lambda (x)
(browse-url
(concat \"http://www.google.com/search?q=\" x)))))"
(let ((here (point)) link-text
(case-fold-search nil))
(save-excursion
(or (re-search-backward link-start-re nil t)
(goto-char (point-min)))
(unless (re-search-forward link-re nil t)
(error "No tag found to end of file"))
(setq link-text (match-string-no-properties 0))
(if (or (< (point) here)
(> (- (point) (length link-text)) here))
(error "No tag found under cursor")))
(if handle-link
(funcall handle-link link-text)
(find-file link-text))))

(global-set-key
(kbd "C-c C-o")
(lambda ()
""
(interactive)
(link-to
"\\<[A-Z][a-z]+\\([A-Z][a-z]+\\)+\\>" "\\<"
;; This regexp pair matches file names.
;;"[a-zA-Z0-9/~_][a-zA-Z0-9/._]*[a-zA-Z0-9]" "[^a-zA-Z0-9/.~_]"
;; This will open text files in your ~/Wiki directory.
;;(lambda (x) (find-file (concat "~/Wiki/" x ".txt")))
)))

Monday, September 7, 2009

Spanking Dawkins's Weasel

One of the problems with learning a new programming language is coming up with interesting "beginner" programs. Most beginning programs do something exciting like printing out "Howdy, Howdy, Howdy, Joe". Heart stopping stuff ain't it?

To ease this a bit, whenever I come up with an interesting idea for a beginner program, I'll write it up and you can give it a shot.

My hands down favorite simple program is something called Dawkins's Weasel.

Richard Dawkins is a evolutionary biologist from Oxford. He wanted to come up with a program to show the difference between a random mutation and a random mutation with selection. The result is a simple, straight forward program called "Dawkins's Weasel".

Note: Whenever I say "letter" below, I mean letters and/or spaces. Saying letter and/or space all over the place makes for a tedious read.

First we start with a target sentence, which we'll call "Target". This is what all good little sentences want to be when they grow up. Dawkins used "methinks it is like a weasel" and so will we.

Next we create a parent sentence called "Parent". This is a gibberish sentence of random letters that is the same length as Target. "y ksqmjwepqtgtyylmfexstvktpa" will work, as will "aaaaaaaaaaaaaaaaaaaaaaaaaaaa".

Our goal is going to be to evolve Parent into Target.

Parent will produce a litter of children. The fittest of the litter will then become the next parent. This continues until we produce a Parent that equals Target. It's survival of the fittest and all that jazz.

Unfortunately the sex life of a sentence is pretty dull. The way a sentence has a baby sentence is by copying itself and then changing a single letter at random. Not exactly the stuff of pornos. We'll call the baby sentence "Child".

As for who's the fittest? We'll use something simple. We'll count the number of letters in Child that match Target. For example, the sentence "aaaaaaaaaaaaaaaaaaaaaaaaaaa" would score a 2. "methinks it is like a measel" would score 27. If there is more than one pick of the litter, choose one. It doesn't matter how.

That's pretty much it. The only real variable you get to play with is the size of the litter. If you start with a litter of 50 then you tend to finish in less that 100 generations. Once you get the code working, starting dialing playing with the litter size to see how it effects the generation count.

I'll try to keep my eye out for other interesting algorithms worthy of posting. If you know of any that are fun, the add a comment.

Tuesday, July 7, 2009

A Window In The Ghetto.

One of the more baffling elements of Linux distributions is their constant relegation of scripting languages to the command line ghetto.

On one side you have the all the wowwie zowie windowy programs, most of which are written, for no good technical reason, in C. On the other hand you have thousands of useful Perl/Python/Java programs that rarely get used because they have command line interfaces.

If we really want to harness the power of Gnome, then we need to make it easier to write Gnome programs in the most popular languages that Gnome supports. These languages are scripting languages.

Scripting exclusively for the command line almost made sense 10 years ago. Old versions of scripting languages didn't really interface well with the Graphic User Interface (GUI) and Text User Interface (TUI) libraries. You often had to have completely separate interpreters for GUI and non GUI interfaces. Anyone remember perlx?

Now days that's bunk. Perl, Python and Java all have officially supported interfaces to GTK, and, in case you don't know, they're *MUCH* easier to use than the C interface. I'm talking an order of magnitude!

As for the TUI, how many tasks do server administrators do that would be made much easier with a simple curses interface?

Python includes a Curses interface by default (it's why I learned Python). Perl isn't too far behind with a very stable Curses package on CPAN. This could be added to any distribution in a matter of minutes. I'm not sure about Java, but if it doesn't exist it wouldn't take long to make one.

The point being, that we have to start pushing distributions to include text and graphic interface libraries with the languages they support, and give them the same status as the language. If the language is included by default, so are the GUI and TUI libraries. Don't let your hot Molly Ringwald fantasies blind you. The '80s are over. We need better interfaces.

Once the user base can depend on the libraries being there, we're going to open up Gnome to a whole to set of ideas. These new programmers greatly out number the current set of Gnome C programmers. They'll be able to fix user level problems that we hardcore programmers don't even know exist.

If you're convinced at this point, then start pestering Ubuntu. I think they're the closest to having all the pieces. Then we can go after Debian and then we'll gang up on Red Hat.

If you're not convinced, then let me over make my point with sort of a preemptive FAQ.

* We didn't do it last time, why do it now?


OK, you lead with an Ace.

First of all, modern bindings to the major scripting languages are dependable and stable. This wasn't true before.

Also, we're in a much more graphical time. Most users consider dropping to the command line a failure of the interface.

We need a bridge between the two.

* What if the bindings disappear?

The odds of the supported bindings disappearing any time soon is negligible. As for other bindings, if more people start using them, the more dependable they become. Does anyone really expect "vim" or "sendmail" to disappear any time soon?

* What's wrong with C?

Nothing, in it's place. If I were to suggest dropping all the scripting languages and doing everything in C, I'd be laughed off the Internet. Somehow we're supposed to believe that the argument is less absurd when you pop up a window.

* Distribution X has the bindings in the "Extras" section. What's wrong with that?


In the business world getting managers to allow "extras" on a system is a hard sell. Damn few middle managers get fired for failing to innovate.

Besides, if it's in it's own section then there must be something wrong with it. Right? Can I get back to you on that?

* You can get all those bindings at site X.com.

If the "Extra" argument is a hard sell, then the 3rd party site is a no-sale. They're right on this one.

A company pays for Red Hat license instead of using the free CentOs because it gives them someone to yell at if things go boom. If you depend on 3rd party software then you get sent to finger pointing hell.

I currently have a problem with my CentOS box at home. I'm using CentOS repositories and a few other reliable sites. Two sites depend on different versions of the same library. I can't upgrade until I find and remove the conflicting programs.

Run that past a manager and you'll be a Microsoft shop by the end of the day.

Besides, if the software is trustworthy and useful then it would be include with the distribution. Feel free to repeat this until your head explodes. It's what they do in corporateland.

* If you want to use the OS, learn the commands!

If you want to make me learn a bunch of esoteric commands, I'm using another OS!

Let's use the "chage" command as an example. It's a pretty simple command which I use about once a year. Every time I use it I have to look up the command flags, because I only use it once a year.

Now wrap it in a curses interface. I no longer have to use the man page. "chage-curses" pops up the user's info, lets me change it. It even uses a calendar to help me change the dates. It then munches up the interface changes into a command line call which make the actual changes.

Take it one step farther. I am a hot shot l33t Hax0r. I shouldn't be doing this peasant crap at all. I create "chage-gtk" in 30 lines of Perl, and now my helper monkey does all the chaging while I hack the cosmos.

* Surly you don't mean every scripting language?


Yep I do. For server admins the command line will live on forever. It's the easiest way to make bulk or automated changes.

Your average user isn't a server admin. We need to stop dressing them up like one and making them dance for us.

Every general purpose language that has a stable GUI and/or TUI and is already included in a distribution needs to have those bindings included. We need to show non-Linux users, and other programmers that we're serious bout getting out of the 1980s. That means that any general purpose language that can help us out needs to be embraced.

* What about languages that don't have stable GUI/TUI bindings?

Most of the popular languages have had stable bindings for years. Perl, Python and Java all have officially supported GTK/Gnome bindings.

If a language only has a stable TUI, but not GUI binding, then just include the TUI. If it has no stable bindings at all, then it remains command line only until it gets it's act together.

* Some languages have GUI/TUI bindings that are a bitch to make into packages.


Let the developer of both the bindings and the language know that that's the reason the binding isn't being included. Most will gladly fix their code or provide the distributions with packages.

* Jeez, our distribution is getting awfully big.

One: The graphic libraries shouldn't really be that big, they're just interfaces to existing libraries.

Two: This move is important! If going from a command line interface to a graphic interface is only being held back by a pack of Luddites, then Linux doesn't deserve to play with the big boys.

Three: This is also an advantage that we seem to be afraid to exploit. Does any other OS come with, by default, an easy way to create useful window based programs just using a text editor? We shouldn't be hiding this feature, we should be shoving it down peoples throats!

* Why should we include fooscript when real programmers use barscript?

You're an idiot.

This isn't about the size of your digital penis, it's about removing completely artificial barriers from the users. Gnome doesn't really serve the clever user well. This problem is trivial to fix. We need to do so in the most inclusive way.

If Perl gets the job done for you, great! If you like Python, no problem. Someone at the Guile compound must have got laid because it's perked right up in the last few months. Guess what? It has both a curses and Gnome interface. Welcome to the club Guile! (Assuming your bindings don't suck.)

* Real programmer program in C.

Actually they don't. Most casual programmers start by looking at a program that almost does what they want, and then pick at it. If the program is a script, it's much easier to play with.

For the inquisitive user, in real terms, modern computers are less powerful then they were in the days of Dos and Unix. Back then a curious user who had an itch, could scratch it with a batch file or shell script. This new command was on par with anything else on the system. If it was generally useful they could email it to anyone else and they could use the new command too. Now days a clever user has to either waste monkey cycles writing in C or they have to have other users download the script-gui library package.

Pithy Summary


This is a stupid pointless wall, and it needs to come down!

Tuesday, June 23, 2009

Emacs: P3 Separate But Equal.

In the first post, we went over some basic theory. In the second post we calibrated Emacs's concept of terminal color with the reality of our terminal program. We also created a file called "color_test.el" which is useful for showing the common "faces" used in programming.

I this part we'll explain what a "face" is and show to get terminal faces and display faces to play nice.

Faces

A "face", in Emacs parlance, is all the characteristics of a piece of text. This includes it's font, size, whether it's bold or italics and it's color.

In the olden days, faces were created by hand. They're not too bad once you get a hold of them and they're surprisingly flexible.

Emacs faces can do all sorts of snazzy things like auto-detect whether they're on a terminal, change their color if the background color changes and invert themselves if they're on a black and white screen. Here's an example taken from the Emacs Elisp manual, section "Elisp/Display/Faces/Defining Faces".

(defface region
`((((type tty) (class color))
(:background "blue" :foreground "white"))
(((type tty) (class mono))
(:inverse-video t))
(((class color) (background dark))
(:background "blue"))
(((class color) (background light))
(:background "lightblue"))
(t (:background "gray")))
"Basic face for highlighting the region."
:group 'basic-faces)

Alas, you young punks don't wanna do it by hand. You'd rather use Emacs's built in customizer. Fair enough.

The Customizer.

I'm not going into a lot of detail here as there are other web resources dedicated to customizing Emacs.

For the sake of this article we need to know:

  • "M-x list-faces-display" shows you all the faces Emacs knows about.
  • Pressing when your cursor is on the face name will let you edit it.
  • In programming mode, all the fonts we care about begin with the wildly intuitive name "font-lock-".
  • When you're customizing in terminal mode don't forget about "Weight Bold" and "Weight Light". In most terminal emulators they give you extra colors to play with.


Multiple Customs.

If you're using a new version of Emacs, you can go into the customizer, click on the "state" button and select "Show All Display Specs". Then click on "Display" and choose "Check List". This lets you select the modes that you want the changes for. If this gets the job done, then you're done. I've had trouble with edits in TTY mode stomping on my edits in display mode, so I like to keep the variables separate.

After much (and I do mean multiple days) experimenting, I've chosen a more robust solution that is easier to maintain and is a lot harder to stomp on.

Whenever you customize a face and save it, Emacs replaces the function "custom-set-faces" with a new version that has your changes in it. The change is written into your custom file. This file could be the end of your ~/.emacs file or the file named in the "custom-file" variable. I'll use the generic "custom file" because I don't care where it actually is.

The way I handle multiple customizations is to customize them via the customizer. Then load the custom file back into Emacs and rename the custom-set-faces function so it only fires when you're in either terminal or display mode, but not both.

It's very easy to do and it's mostly cut and paste.

Load your Emacs custom file. Before any "custom-set-faces" commands, add these 2 functions:

(defun my-custom-set-faces-display (&rest faces)
"Load these faces if Emacs is in windows mode."
(when window-system
(apply 'custom-set-faces faces)))
(defun my-custom-set-faces-terminal (&rest faces)
"Load these faces if Emacs is in terminal mode."
(when (not window-system)
(apply 'custom-set-faces faces)))

Now look at your current "custom-set-faces" command. Is it set up for display mode? Then rename it to "custom-set-faces-display". If it's for the terminal then rename it "custom-set-faces-terminal". Now add an empty function call for the "other" function. If you set custom-set-faces-display, then add "(custom-set-faces-terminal)". If you set custom-set-faces-terminal, then add "(custom-set-faces-display)".

Mine looks like this:

(custom-set-faces-terminal
'(font-lock-function-name-face
((t :foreground "LightlyDepressed" :weight bold)))
'(font-lock-comment-face ((t :foreground "cyan"))))

(custom-set-faces-display)

Now fire up Emacs in terminal mode (emacs -nw) and edit a face. I'll make the font-lock-comment-face "Naval" colored for this example. Then save the change. Take a look at your custom file and you should see the color change in the function "custom-set-faces". Here's my example:

(custom-set-faces-terminal
`(font-lock-function-name-face
((t :foreground "LightlyDepressed" :weight bold)))
`(font-lock-comment-face ((t :foreground "cyan"))))
(custom-set-faces-display)
(custom-set-faces
;; custom-set-faces was added by Custom -- don't edit or cut/paste it!
;; Your init file should contain only one such instance.
'(font-lock-comment-face ((t (:foreground "naval"))))
'(font-lock-function-name-face
((t :foreground "LightlyDepressed" :weight bold))))

Delete the old custom-set-faces-terminal. Rename custom-set-faces to custom-set-faces-terminal, save your work and you're done.

If you wish to edit your display faces, just fire Emacs up in display mode and run through the same process.

There's no limit to the number of face sets you can add. You can have a different face set for every day of the week if you want. Just create a "my-custom-set-faces-" for any discriminator you want and rename custom-set-faces to match it.

Hopefully my absurd 10 day journey into Emacs's faces has been rendered down into something useful for you. Let me know if you found this helpful.

Monday, June 22, 2009

Emacs: P2 Color Me LightlyDepressed.

Now that we have some color theory under our belt, let's calibrate Emacs's concept of color with the reality of the terminal's.

First we have to get the real colors being displayed. I'm using gnome-terminal which has a built in color picker. If you're using a terminal that doesn't have it's own color picker, fire up "emacs -nw", do "M-X list-colors-display", then use something like "xmag" or gimp to get the color values.

My Color List.

From a gnome-terminal, select "Edit/Current Profile" from the menu. From the "Default" screen, click on the "Colors" tab. At the bottom of the screen you should see 2 rows of 8 colors. The first row is the 8 colors that make up the terminal's pallet. Left most is entry 0, right most is 7. The second row are the colors you get when you print the first row using "bold". Gnome-terminal has a 3rd row of colors that are the first row in "dark" mode, but you can't edit them.

If Emacs was smarter about terminal colors you could tell it about all 3 rows of colors and it could use "bold" and "dark" version to increase the chance of it's making a good color choice. Alas, were stuck with our one row of 8 colors.

Click on each color in order, and write down their Red, Green and Blue (RGB) values. For example, the 4th color in my pallet is kind of brown, with yellow below it. It's RGB value is 170/85/0, so pallet entry 3 is 170/85/0.

This is my list:

0 0 0 0
1 170 0 0
2 0 170 0
3 170 85 0
4 0 0 170
5 170 0 170
6 0 170 170
7 170 170 170

Once you have all 8 values, ask Emacs (in a terminal) for help on the variable "color-name-rgb-alist" (C-hv color-name-rgb-alist). The help should list all the color names that Emacs knows and their RGB values.

Scan the list for colors that match the gnome-terminal colors. If you find a *perfect* match, put the color's name besides it's color in your list. Only use the name if it's a perfect match. 0/0/0 was the only match for me. I labeled color 0 "Black".

For the rest of the colors, give them descriptive names that are not in color-name-rgb-list. The last thing we need is 1 name for 2 colors.

Here's my final list.

Black 0 0 0 0
Brick 1 170 0 0
Greeny 2 0 170 0
Brownish 3 170 85 0
Naval 4 0 0 170
DarkishMagenta 5 170 0 170
NeonPee 6 0 170 170
LightlyDepressed 7 170 170 170

Now we have to get the colors into Emacs. It turns out that that's pretty easy.

RCS

First, make a backup of your ~/.emacs, just to be safe. As an aside, because this series isn't nearly long enough, consider using RCS to backup any config files that you hand edit. Under Emacs RCS is trivial to set up and use. It's saved my monkey boy butt more times than I care to remember.

To set up RCS for your ~/.emacs, make a directory called ~/RCS. Then load your ~/.emacs file into Emacs. Hit C-xvv. That's it. You're done. Your ~/.emacs is now write protected and checked into ~/RCS. To check out your file so you can edit it, load it into Emacs and hit C-xvv.

Back to work.

Edit your ~/.emacs, and add the following code. If your ~/.emacs has a custom-set-variables or custom-set-faces function, place this code before either. Obviously you should use your own colors and names for the my-tty-color-define-8 commands.


;; Code for handling term based Emacs.
(defun my-tty-color-define-8 (name index rgb8)
"Set the tty pallet using 8 bit rgb values."
(tty-color-define name index
(mapcar (lambda (x) (+ x (* x 256))) rgb8)))

(if (and (not window-system) (= 8 (length (tty-color-alist))))
(progn
(tty-color-clear)
(my-tty-color-define-8 "Black" 0 '(0 0 0))
(my-tty-color-define-8 "Brick" 1 '(170 0 0))
(my-tty-color-define-8 "Greeny" 2 '(0 170 0))
(my-tty-color-define-8 "Brownish" 3 '(170 85 0))
(my-tty-color-define-8 "Naval" 4 '(0 0 170))
(my-tty-color-define-8 "DarkishMagenta" 5 '(170 0 170))
(my-tty-color-define-8 "NeonPee" 6 '(0 170 170))
(my-tty-color-define-8 "LightlyDepressed" 7 '(170 170 170))))

Save your ~/.emacs file and exit. Restart with "emacs -nw" Type "M-x list-colors-display". You should see your color names listed with the colors.

This might not seem like much of an achievement, but you've actually taken a pretty big step.

To check our your results, create a file called color_test.el in Emacs (-nw). It should put you into "Emacs Lisp" mode automatically. Now type in this program:


;; Comments are in 'comment-face'.
;; defun and defvar are in keyword-face.
(defun function-name-face (&optional is-in-type-face)
"string-face `constant-face' string-face"
:builtin-face
(error "warning-face"))
(defvar variable-name-face)

;; To see the "doc-face" go into "perl-mode".
=pod
This should be in doc-face.
=cut


The program itself doesn't work. It's not even syntactically valid. All it exists for is you show all various "faces" that Emacs uses when coloring code.

To see doc-face, use "M-x perl-mode".

How do you like them colors? If you're happy happy, then you can skip post 3 of this series. If, however, you're like me and think that Red is a horrible color for comment text, then await with baited breath the last installment of the Emacs color saga.

Sunday, June 21, 2009

Emacs: P1: What Color Is My Painbow?

Last week I had a classic "Monkey Boy" moment. I decided to adjust the colors in my text editor. 10 days later I'm finishing a 3 post blog on it.

I worry myself some days.

This first post is going to be mostly theory work. Post 2 and 3 are more hands on.

Laying the Ground Work.

I use an editor called Emacs for most of my programming. It's an old editor, but its one of the most powerful editors out there. It also lets you edit files in display (windows) mode and from the shell (command.com for you Windows folks).

Now days most of the editing is done in display mode. No real surprise there. However, there are times when working from the shell makes more sense.

I routinely log into distant machines across slow connections. I could pop up a virtual session and wait for the window in the virtual session and then wait for the editor in the window in the virtual session and then wait for the file in the editor in the window in the virtual session, or I can use text mode, where are complete screen refresh is around 2000 bytes.

I hate waiting. Its a no brainer.

The down side of terminal mode is that you can only use characters to draw and you have a limited number of colors. Both of these could be overcome with modern technology, but it ain't going to happen so we have to get used to it.

Why do you have limited colors? Well, the underling technology differences between a terminal from 20 years ago and a modern graphic display is pretty significant.

RGB

Colors are made by mixing various amounts of Red, Green and Blue (RGB) together. If you crank up the RGB, you get bright colors, dial it down and you get dark. Wikipedia has a nice write up on color depth so I won't go into it here. The only thing you need to know is that by adjusting the RGB values you can change colors.

On modern display you have absolute control of every dot on the screen. Each one has it's own RGB setting which is independent of it's neighbor.

Old school color terminals were more like "paint by numbers" projects. You were given a pallet of colors (usually 8) that were hard wired into slots. If you set the color to pallet slot 0 and then printed, you got black text. If you printed in color 4 you might get blue. Unfortunately for us, these are the terminals that most terminal emulators emulate.

We have two problems when we want use Emacs in both terminal and display mode: First is that Emacs's support of terminal colors is functional, but not much more. The second is that the friendly Emacs customizers don't like it when you're a switch hitter. In fact they gets down right medieval on your monkey butt. Well, this ain't monkey butt, this is monkey boy butt. Accept no substitutions.

Terminal Colors


As I said before, most terminal emulators model the old style, 8 color pallets. There are ways for a program to ask the emulator for the number of colors available, but there isn't any way to get the actual RGB of each color.

What does Emacs do? It guesses! If you don't tell it otherwise Emacs assumes that you have an 8 color pallet with the following colors:


Slot Name Red Green Blue
---- ------- --- ----- ----
0 black 0 0 0
1 red 255 0 0
2 green 0 255 0
3 yellow 255 255 0
4 blue 0 0 255
5 magenta 255 0 255
6 cyan 0 255 255
7 white 255 255 255


The numbers after the colors are how much Red Green and Blue that each color is supposed to have. 255 is the largest number you can express in 8 bits (1 byte) of data. There are places internally where Emacs uses 16 bit (2 byte) RGB values which go from 0 to 65535. I got bit by this more than a few times so I'll try to point them out, or gloss over them when I can.

Emacs cares about the RGB values because you (the user) set colors by name not slot values. If you set the color of something to "CadetBlue1" 152/245/255, then run in terminal mode, Emacs needs to figure out which of the eight colors CadetBlue1 is closest to. It uses the RGB values to figure it out.

Oh, by the way, the name "CadetBlue1" comes from a variable called "color-name-rgb-alist". To see it's contents, fire up Emacs in display mode and type "M-x list-colors-display". You'll see the colors and their names.

Let's do some hands on. From a terminal, type "emacs -nw". It should start an Emacs session in the terminal. In Emacs type "M-x list-colors-display". You'll get a listing of the 8 colors that Emacs knows about. Note: On some systems you get more than eight. Lucky you. The theory is still the same.

If you're like me, you notice one thing first off. These colors look nothing like their names! The Red might be brick colored. Yellow may look brown. And my white has tattle tale gray! What happened?

Easy. Emacs has no idea what colors your terminal's pallet is set to and it's guess stinks. How do we handle the miss-match?

One option is to change our terminal to Emacs's pallet. Then we can vomit and claw our eyes out. Basic colors tend to be rather harsh on the psyche.

The second option is to tell Emacs what our terminal is really packing. That's the subject of the next post.

Wednesday, June 3, 2009

Aw C'mon, be a Minion!

Come my minions! Gather 'round me and do my bidding!!!

Nuts! That never works.

I'm starting a minion list. It's not a list of minions, it's a list of projects for my minions.

I'm going Internet because minions are sparse on the ground around these parts. I believe it's because my readership is made up of selfish swine who won't surrender their sense of personal identity for the betterment of me. But I press on! I'm brave that way.

My minion list is a list of projects, usually computer projects, that strike me as a good idea, but I don't have time to do them. It would be nice if a minion would do them (hint, hint).

I've talked about the minion list in the abstract many times, but now I've decided to codify it. That way, when someone else commercializes one of my ideas, say "Feels on Wheels" (I'll explain later), I can sue them for plenty cash.

Wanna Taste?

My first entry for the minion list is "Voice Messaging". Voice Messaging is like Instant Messaging, augmented with sound.

The concept is simple. A person wants to send you a message. Instead of typing, they have the option to record a message. The message is sent to you, along with any typing they wish to do. On your side your VM program converts the sound file to text and displays it like a normal IM. If the speech conversion garbles it too badly, you just highlight the part you can't read and it plays that part of the sound file.

This would rid the world of emoticons. If you want to know how someone is feeling, play the sound file.

I Object!

* Speech recognition sucks.

You're right as far as you take it. However, in this case we're not producing a finished document. It doesn't have to be perfect, just readable. The recognition only has to be gud enuf 2 undrsAnd. If you're unsure what was said, play the sound file.

* What's to stop someone from saying one thing, but typing another?

The conversion is done on the receiver's end. The sender can send text along with the sound, but the final arbiter of the text is the receiver.

* What's to stop someone from screaming obscenities or pulling other "funny" jokes.

Before playing a sound file, the program would normalize the sound levels. It could even warn you if the original has loud points in it.

* Couldn't someone just use the phone?

Yea, but VM is closer to IM than a phone call. The sender can send a VM, but you can ignore it until you're ready, just like IM.

* What about people who don't stop yakking?

Put a size limit on how big you'll receive. Make it settable per person. Even if someone does get carried away, it's still better than voice mail.

Almost any sound player lets you replay selected parts of a file so you can deal with the witlings who leave 5 minute messages and wait till the end to mumble their contact information.

* What file format should we use?

This is getting a bit low level, but something open format is an obvious requirement. It turns out that there is an open format called Speex that gets the job done.

A 14 second PCM, 16 bit, mono 48000 Hz .wav file is 1.4 meg. Convert it to an .mp3 shrinks it to 123K. Speex mauls it down to 66k. All with no real loss of sound quality for what we're doing.

* What about security?

As long as an open audio format is used, and you trust your audio player, then you're all set. If you use an audio player that does odious stuff like pop up web browsers (oy!) then you're asking for trouble. Playback should be based solely on the contents of the file, not it's extension.

* What technology should the be built on?

I'd bet that most IM protocols allow file transfer. It would simply be a case of the client programs handling audio file differently.

I Submit Like the Schweinhund I Am!

You have convinced me oh minion master. I supplicate at your feet. What is the first step?

For a start, someone could write a standalone application that reads Speex files and transcribes the contents. Some college out there has to have information on phonetic translation. Lets see what the state of the art is.

After that, add playback of selected sections. Once we have that settled then the rest would be cake, and how many masters let their minions have cake?

Tuesday, June 2, 2009

Sound Bite Me!

One of the true pleasures of programming is finding some niggling little problem and solving it with a simple bit of code. If you can solve it in a couple of hours, even better.

When I blog I tend to speak what I'm typing. If it doesn't sound right when I say it, it probably wont sound right when you read it.

One of the problems I have when I blog is that I talk faster than I type. I talk faster than I think. I start typing, and then I get a flash of inspiration. I run the idea through my head, and then, half a virtual page later, I realize that I haven't written anything down.

Then I have to try to recreate the idea from memory, but the flash is gone. By the time I've rebuilt it, or accept that I've forgotten it, my original thought is out playing in the yard and won't come back.

It frustrates the hell out of me.

I started playing around with ideas to making blogging easier.

At first I tried to check out the state of computer speech recognition. I figured I'd just blab on in my blog, and then I'd go through and clean it up by hand.

Computer speech recognition is still slow, expensive and it still sucks. Trying to run a editing session without using a keyboard is slower than typing with 2 fingers. I also wanted to blog via Linux, so firing up a Windows product ain't getting the job done.

Next it tried to do some sort of integration of speech and text together.

I envisioned loading a sound file into a sound editor, where I could chop it up and move it around. While editing the sound, I'd join text to it. When I moved a blob of sound, text that went with it would move to. Eventually I'd piece a blog out of all my rattlings on.

I still think that this is an interesting solution, but man, it would be some work! I also I think I'd end up with a crappy sound editor linked to a crappy text editor. No dice.

I also had a minor epiphany. I'm not going to keep these sound files around forever. I'm just brain dumping to a file for a few minutes until I've finished typing my original though. After that I can replay the recording and transcribe anything I think is useful.

I already know how to record from a mic on my Linux box. Adding that to my new insight I wrote a shell script that turns on the sound recorder, dumps the contents of the microphone into a file and, when I stop recording, plays it back (that way I can tell if I forgot to turn on the mic or something).

It worked like a charm! Every time I needed to make a note, I just fired up the script, decided on a name for the sound file and away I went. It was a little awkward, but a big step forward.

I called the script "sound_bite".

After than I needed to come up with a way to play back my sound bites, so I started adding flags to play the last sound bite or the first sound bite or list the sound bites and let me pick. Then, another epiphany! They're just frigging .wav files! Maybe I could just double click on them in my file manager. Oooo. Me one smart monkey!

Actually, once I got the file manager into the game, it cleaned up a lot of code. I didn't have to tell the recorder where to put the sound files, I would always put them into the same directory and give them a time stamp for a name. If I wanted to organize them better, I'd use the file manager to rename them or move them elsewhere.

The only thing left was making it easier to use. Typing in the command every time is a minor pain. I needed a quicker way to access it. That was easy too.

I hooked up sound_bite to a shortcut which put all the .wave files into the directory "sound_bites". I made another shortcut to bring up the file manager in "sound_bites" directory. I'm sure I could come up with a few dozen little tweaks, I know enough to stop typing when I'm done.

I'm now an official audio driven blogging fool!

Here is the entire source code for sound_bite. You may have to play with it a bit because the blogger code likes to play with it.

Enjoy!


#!/bin/sh -

# Sound_bite: Written by Dale Wiles 6/2/09.

# Exit if an error occurs.
set -o errexit

if [ $# -eq 0 ]; then
  cat <<EOM
Usage: $0 directory

Move to DIRECTORY and start recording a wave file from the microphone.
The name of the wave file is yymmdd_hhmmss.wav.
EOM
else 
  sound_dir="$1"
  cd "$sound_dir" || exit 1

  # Make the output name based on the time, down to the second.
  # That way I can't overwrite existing files.
  # Alright, in theory during daylights savings time it could
  # overwrite.  I've added code for that almost impossable situation.
  while :; do
    out=`date +%y%m%d_%H%M%S`.wav
    if [ ! -e $out ]; then
      break
    fi
    echo "Waiting...."
    sleep 1
  done

  echo "Recording $sound_dir/$out"
  sox -t alsa default -v 7 $out
  echo "Playing $sound_dir/$out"
  aplay $out
fi

Tuesday, May 26, 2009

Schleprock says: Lets Open that Id.

Through out my digital travels I tend to trip over more than my share of quirks, bugs and "obvious" ideas for improvements. I'm gifted in a Schleprock sorta way.

Unlike Schleprock (probably a pinko!) I also try to be a good netitizen and report what I find.

The problem is there are too many barriers to being a good Samaritan. The biggest one I run into is having to set up an account just to make a bug report. I have 71 separate accounts to date, of which I really use 2 or 3. I'm reticent to add more accounts just for the privilege of helping someone fix their code.

I understand the reasoning for the accounts. It's to cut down on Spam. On the surface it looks like a good solution, but in reality it's almost as bad as the problem. It cuts down on Spam, but it also cuts down on user participation. No participation is killer for open source projects.

One solution that I've seen it to allow anonymous posting, but have each posting validated before being passed through. This is a great solution. It allows anonymous posting, but puts a minor impediment in the way. If you want fast turnaround, set up an account, if you can wait, then post anonymously.

The only reasons validation isn't more popular are self important administrators, who don't make the time to deal with the human component of their project, or huge projects that legitimately generate a large number of anonymous emails. These huge, anonymous, projects are fairly rare, and Spam filtering is easier for them than for normal mail. The size of my penis is not a bug.

But how do you deal with a site, that for some obsessive reason, wants you to set up an account. If only you could create one Id and have an *OPEN* way to hand it around? Open, Id? Hmmmm. OpenId? It's got a nice ring to it.

The OpenId project exists to solve the problem of having too damn many Ids. It's really clever on how it solves it too.

You start out by going to an OpenId provider and signing up for an Id. Arg! I know, you have to sign up for another account! But this is the wish for more wishes. Once you get an OpenId account, then you can use it on any OpenId enabled site that's out there.

Let's say I'm trying to log on to newsite.com. Newsite is OpenId aware.

I type in my OpenId (http://upcracky.myopenid.com in my case). Newsite then decodes my Id and redirects me to myopenid.com. I log into myopenid.com using the password I set up there. Once I'm logged in, I'm redirected back to newsite.com and I'm in. If the password is good enough for myopenid.com it's good enough for newsite.com.

In fact, I don't believe newsite.com ever sees my password. They're probably pinkos!

How do you get an OpenId. One of the nice things about OpenId is, you may already have one. If you have a gmail account, they're an OpenId provider (It's your email address and password.) Yahoo is another. If you don't have one of those, then web search for "OpenId Provider". They're plentiful. Hell, OpenIds are cheap. You can make one for work and one for home and one for the kids and dog.

The only real problems I've run into with OpenId are:

1) They're not robust enough for super secure sites. I don't use OpenId for banking.

2) Not enough sites use it. This one is easy to fix. Whenever you have a site that is OpenId aware, use it. If you have a site that's not OpenId aware, complain to the admins. The code to implement your own OpenId log on is freely available on the net. The more sites that have it, the more users that use it. The more users that use it, the more sites that will get it.

Seriously, if you have a ton of user IDs and passwords, then it behooves you to pester all the sites you know to set up OpenId. If not, you'll end up like me, stooped over from the pain of dragging around all my BugZilla accounts.

Wowzie wowzie woo woo.

Saturday, May 16, 2009

Clones Ate My Files: A Sci-Fi Geek Thriller.

OK, so I'm banging out code for the betterment of my corporate overlords. It's a conceptually simple program: You give it a command and a list of servers and it runs the command against each server.

Not impressed? Well Monkey Boy doesn't get the corporate shekels without doing major mojo. This program runs in parallel. It juggles 64 instances of the command at a time. It can ping all the severs on a netmare in less than a minute, keep track of the results and print them out in the order that they were input. I get wet just thinking about it!

All praise Monkey Boy right? We'll I did run into a problem. A sneaky problem that involved failed clones, suicidal files and a forgotten inheritance. It's good stuff. The problem is, if you don't program in Perl, you're not going to give a crap. Oh well, I've never let my complete lack of an audience slow me down before. Why start now?

Like I said before, the beast is written in Perl. Perl's not an elegant language, but if you need to leap out of the bushes, rape and strange a problem and get on with your life, then Perl's your language of choice.

The basic layout is: Start a sub-process for the first 64 severs and have each one write to it's own temp file. When one sub-process finishes, read it's results from the temp file and let the temp file disappear. Then add a new sub-process for the next server. Once all the servers are done, print out the results in order and accept smoochies from hot code groupies/naked underwear models. Simple.

The Boy of Monkey knew that he'd need lots of temporary files to hold command results. He'd also like the files to go away by themselves when he's done with them. Perl lept to his aid with File::Temp. File::Temp is kind of the anonymous underage prostitute of programming. When you say "Gimmie" it gives you access to the goodies and provides an assumed name. When you're done using it, it disappears into the aether. It all works great, until someone, or something, starts killing the temps prematurely. Then you get a mystery to solve. Foreshadowing!

I got the code working and was getting ready to document (Yes, Monkey Boy is a pro, not documenting makes you a douche bag), when I though, what happens if the sever command can't run? If the user misspells "ping" as "pong" will they get a reasonable message?

Does
Couldn't open file '/tmp/multi.1d834.pid': No such file or directory
strike you as reasonable? Me neither. Nuts!

The actual error message made sense to Monkey Boy. He spawned the beast. '/tmp/multi.1d834.pid' is one of the randomly generated temp file names. When a sub-process finished, the program asked the temp handle for it's file name. It tried to opened the file, but for some reason the file was gone! Somehow the files were being killed before Monkus Boyus could get to them. No one should know about these files. They have specially constructed names, known only to the monkey... or the monkey's clone.

The way you run a sub-process in Linux is using the fork() command. You're running along happy as a clam, as if a mucus coated bivalve is your apotheoses of happiness, and then you hit fork(). At that point your program is cloned. You have 2 running copies of the code. The only difference is that fork() will tell the parent the ID of it's child. The child is handed the ID of 0, which tells it that it's the clone.

This is kind of Star Trekie at this point ain't it? We've got parents making 64 clones (top that Octomom!) and we've got children that are one bit away from being perfect copies of their parent. It gets better. The clone's next job is to call exec() which completely obliterates it and replaces it with another program. It's this second program, the sever command, which does the real work I want done.

A clone has one job. It's job is to die and be forgotten. Programming ain't for wussies!

When all goes well, the program runs like a well oiled roach motel. The clones check in, but never checkout. They disappear on the spot and are never heard from again.

That's when all goes well. What happens when the exec() command fails?

It's simple really, the clone lives on! It also keeps it's copy of the temp files, which it believes it owns. When it dies, it takes the temp files with it. Clones can be selfish little pricks.

Eventually the grieving parent checks on the child. It notices that it's died and then tries to check the temp file for the reason. The temp file is gone baby gone, it died at the hands of junior. You can only die once in temp file land.

As for the reason the exec() failed? It was written into the temp file. You know, the temp file that's in temp file heaven? Hmm. What to do? What to do?

Suddenly Monkey Boy (you remember Monkey Boy, he's the hero of this epic) has an insight. When you delete a file it doesn't really disappear until the last program that has a hold of it lets go. It's removed from the directory, so it can't be seen, but it's still out there, in limbo, awaiting for the sweet kiss of digital death.

Who else is holding on to the file? The parent of course! The question was, could Monkey Boy get to the parent to cough up the handle and could it be used for reading?

Detective Monkey Boy began investigating. He checked the usual suspects. "perdoc File::Temp" didn't provide much. It was higher level than a kite. The Internet tubes were blocked by flame wars and almost naked pictures of some platinum blond from California. No go. Detective Monkey Boy knew what he had to do. He had to go (non-prequil) Jedi. "Use the Source Luke!" is the rallying call of the Open Source movement. But Monkey Boy's name ain't Luke.

Into the source goes the hero. Past lines of documentation. Past obscure code references. Further he goes, until he finds, what he knows in is heard must be, "use IO::Handle". Rocken!

For those that don't know, deep in the belly of the beast, a file comes down to little more than a number. When you open up a file, voodoo happens, and an entry is put in a table called the File Descriptor Table. What you deal with, either directly or through some Perl interface, is an entry in this table. If you can figure out the index number to this table, you can find your file. File::Temp was a cold fish, but what about it's ancestors? File::Temp inherits from IO::Handle. To get to the real power, you got to seduce grandma.

"Hey there Granny!"

Once I go my hands on Granny's nodes (yech!) she gave up IO::Handle. IO::Handle has the fileno() function. You got the number, you get the data.

After that it was just a hop, skip and a file dupe to get to the data so ingloriously killed off by the wayward clone. It takes more than the death of a temp file to stop a motivated Monkey Boy!

All praise Monkey Boy.

Sunday, March 29, 2009

Death By Wiki

One the biggest software concepts of the past decade is the wiki. It's a really great concept that should have been invented 15 years ago. We had the technology, but it took Ward Cunningham to figure it out. Yay Ward!

A wiki allows almost anyone to contribute to a project, and you get instant gratification when you do. If you're reading a page, and see a typo, you can edit the page yourself, without waiting for a blessing from above. Once you edit the page, then your edited page becomes the page everyone else sees. Lookie what I did!

Unfortunately one of the biggest strengths of the wiki is also it's Achilles' heel. The great unwashed masses can maintain it, so, too often, the great unwashed masses are left to maintain it. That doesn't work.

Projects need leadership. The leadership needs to hop out of the ivory towers now and then and spend some time hanging out with the peasants and documentation.

I know it's boring. I'd rather be spend time writing code than documentation. Almost everyone would, but the docs got to be done.

I'm seeing more and more projects that aren't really projects. They're just inbred code excretions. A new version of the code comes out, and you have no idea what it does, or how to use it. Those in the know know how to use it, but the rest of us are stranded.

In the inbreeder's mind once the project becomes popular someone will swan in and write the documentation. Sometimes it's even worse and they try to make a virtue out of bad documentation by claiming that anyone that uses their code is a l33t h@x0rs and should read the source code.

Adding to the problem, a project opens up a wiki. This is supposed to magically fix all the documentation problems. Users are supposed to gather like worker bees and create documentation. The problem is that one user can't tell if another user's solution is good or a bad. They have no high level view of the project.

For a wiki to work, the project heads need to invest time in checking what the wiki writers are writing. If someone puts something up that's a bad idea, it needs to be edited out and the writers need to know why. If it's a good idea, it needs to get official praise so other users know it's a valid solution.

There also needs to be an editor who chops out entries that are no longer relevant. If version 2 has a bug, and version 3 fixes it, then a 2 page discussion on the version 2 bug is no longer needed. Especially when the current code is version 10 and version 2 hasn't existed for 5 years.

Wikis are a great tool for empowering non-technical users, but they're not babysitters. If you write code for others, you have to listen to them. If you don't you'll find your babysitter goes psycho and your project will die from "shaken code" syndrome.

Friday, October 10, 2008

Gnome Needs a Scripting Language.

Gnome needs a scripting language that fills the same scripting role in Gnome that Bourne shell fills for the command line.

A few days back I wrote a little script that pops up a window, gets some input from the keyboard, runs a program and then exits. I banged it out in about 2 hours in Perl using the Gtk2 library. Piece o' cake.

The problem is that I really can't distribute the program. Even if I assume that all Gnome systems have Perl, it's not safe to assume that they all have the Gtk2 libraries installed.

If I want to make my little script available I'm going to have to rewrite it in C? That can't be right! Linux always has scripts. It's one of the things that separates Linux from the lower beasts!

I decided to check out the current state of Gnome scripting on Google. Um, we're exactly where we were a decade ago. 10 years ago people were bitching about the lack of an official scripting language for Gnome and we're still bitching today.

Let's stop bitching and fix this insult to Gnomehood. For now we'll call the language "gscript" to save typing. If there is already a "gscript", then accept my apologies and lets get on with life.

What Gscript Needs to Be.

Here is a list of what I think gscript needs, in approximate order of importance.

  • It has to exist. This first entry isn't as stupid as it sounds. Good Gnome interfaces have existed for many of the existing scripting languages for years, but none of these scripts/interfaces are *guaranteed* to be there. Seriously, in the past decade, how many GTK/script have you used? For it to be useful, gscript has to come with Gnome. If you're using Gnome, then gnome-terminal may be optional, gscript is required!
  • It needs to be a Gnome scripting language, not just another scripting language that comes with Gnome. This is *important*! There has to has interfaces to all major parts of Gnome, not just GTK. Things like GConf and EggTrayIcon and user menus have to be accessible to gscript. When a new version of Gnome comes out, it comes with a synced version of gscript. It's not optional.
  • It should have useful defaults. Banging out a simple script should be, well simple. About as simple as this:
#!/usr/bin/gscript

init;
guts = gtkbutton(label => "Hello World", callback => exit);
; Note: A default also menu gets created here.
start_app(body => guts);
  • It should work with both GTK and Gnome. I'm not really sure what separates GTK from Gnome, but if the script doesn't need the Gnome stuff, then it should be able to use just GTK.
  • It should do things the Gnome way. One of the big bitches about Gnome is there's always a new way to do something. No one ever explains, in simple terms, why the new way is better and how to use it properly. If the right way to read a keypress from an international keyboard is to use the foobar interface, then gscript uses the foobar interface.
  • Gscript should be simple. We're not trying to replace C or Python or COBOL. The day will never come when 90% of Gnome is written in gscript. It should exist to fill the niche that shell scripts do on the command line. Anything else is gravy.
  • Gscript shouldn't be too simple. We've all worked with languages that are so simple that they're useless. It should have all the control structures of a real language with scoped variables and real data types. At a minimum it should have native lists, hashes, numbers and strings (anything else?). It should also have modules and classes to handle medium size programming. GTK-Bash would be a lousy choice.
  • It should be embeddable. Gnome is moving to things like cellphones. Why not kill two birds with one stone? Embeddable languages are useful and this will help keep the language small and simple.
  • Screw the ASCII code. Enough already! International support should be the default, not an add on. Those strange voices across the big salty pond are other programmers. If they speak Chinese then they should be able to script in Chinese.
  • Screw C! Most scripters don't come from a C background anymore because they don't need to. Computers are a zillion times faster than they were when I was a whelp, and no one is going to put gscript code in the kernel. Gscript should be fast enough to be useful. After that ease of use and flexibility are paramount.
  • Lets not create a new language unless we have to. I believe we can leverage an existing language.
  • If we have to create a new language, lets go for it! A lot of current languages are anachronistic retreads of boring old languages. Why strive to suck?
My Opinion on Existing Options.

Let's run through some of the existing options. Despite my list above, most of what's needed already exists. It's just not included with Gnome by default.

  • Perl or Python : Both of these play very nice with Gnome/GTK. The problem I see with these two languages are that they're both serious overkill for this problem. Also choosing one over the other would come with a pre-built pissing contest. I don't want to see their GTK libraries curtailed. I personally use Perl::Gtk2 and love it. I just don't think they would be the best fit for gscript.
  • Tcl/GTk : Tcl tries too hard to be simple, which makes it painful for all but the smallest of programs. If it scaled better it would have taken over the world by now. Gscript should strive to be what Tcl/Tk failed to be.
  • Zenity : Its great for popping up windows in shell scripts, but it doesn't really let you run actual scripts. A new gscript programmer would be able to implement zenity in about a day.
  • lua-gtk, guile-gnome and ruby-Gnome2 : Of all the languages I talk about, these 3 strike me as the most plausible. They fit most, if not all of the requirements, they have and existing community and, according to http://www.gtk.org/language-bindings.html, they already have viable bindings. Anyone with experience care to chime in?
  • Squeek : Aw c'mon. The official Gnome scripting language being Smalltalk would rock! We could write Gnome scripts that expand programmer's minds and then go give the KDE folks noogies. Pull this off and we P0WN by existing. It may not be the most practical of the choices, but I'll bet it would be the most fun.
  • SomeOther : I'm open to suggestions. Give me your opinions. Any other projects already started? Just remember Gscript is not trying to replace all the other scripting languages. We just need a good choice that we can get officially distributed with Gnome.
In Summary.

Gnome not having an window aware scripting language is a decade old embarrassment. It goes against all that is Linux. The embarrassment is compounded by the fact that we have a lot of options that are ready to go, or could be ready to go in, literally, a matter of days. We just need to choose one.

Once a language is chosen, then the only big step is getting it included with the Gnome base packages. Anyone have an in with the Gnome "powers that be"? If not, then wanna write a script to storm a castle?