Revenge of CookbooXUL

NOTE: None of this code is available. It was stolen by Ninjas. Sorry.

I did a very simple little Ruby On Rails application a while back using the XPFE (Cross Platform Front End). For those people out there with TLA overload the XPFE is what Firefox, Thunderbird, Mozilla and many other products sits upon. It has the XUL (XML UI Language) as well as many other components which make doing clients so rich they make Bill Gates look like a Calcutta leper. To most people this is a “XUL Application” since they don’t realize the full extent of the XPFE platform. This is a lot like saying that you wrote a “Ruby Application” when really you used Ruby On Rails and lots of other libraries.

The original CookbooXUL was nothing more than a XUL face put onto Curt Hibbs’ article Rolling With Ruby On Rails a while back. It didn’t do very many fancy things but did demonstrate that you can easily do a XUL interface with Rails.

This new version of CookbooXUL is significantly different. First off it is still a Rails application, but the Rails part is very thin—really nothing more than a REST backend if that. The remaining functionality for the application is all held on the client side. Don’t worry though, there’s nothing required besides an XPFE/Gecko based browser such as Firefox, Mozilla, or Epiphany. Just load your browser and check the application out.

One of the main detractors that you get from people who rant on Rails applications is that-~~since it is good at making CRUD applications~~-it must only be good for making CRUD applications. I’m hoping that CookbooXUL will demonstrate enough to show that Rails can do more than CRUD. The interesting thing in this case is that Rails isn’t really doing much more than serving and partially controlling the XUL interface. You could easily implement this same application in a number of languages and platforms. The key to my demonstration (apart from showing off a neat Finite State Machine library in JavaScript) is that I did it in Rails.

On the client side I’ve implemented a small JavaScript Finite State Machine (FSM) library which really makes writing asynchronous interactive applications a snap. I’m still refining the application, but I threw in enough demo code that it should get people interested in the technique.

Now With Fresh Odeum Flavor

The main backend feature of CookbooXUL is its use of Ruby/Odeum to search through 10,000+ recipes in RecipeML format. This provides fast as hell access to the entire catalog and demonstrates an application that isn’t really a CRUD application. If you wanted to do the same kind of searching within a CRUD application you would have to troll through tons of records and compare all the fields. It would take forever, and there wouldn’t be any relevance ratings included.

In the Java and Python world there is Lucene for full-text indexing. Lucene is a great library and does nearly the same things as Ruby/Odeum but is more mature right now. One of the things Lucene does not do is support Ruby so for now Ruby/Odeum is the only game in town.

If you were to use Lucene you could easily do this application in a simple little Java servlet. Again, the point isn’t to say that Java or other languages couldn’t do an application like this, but rather to demonstrate that a Rails application does not have to be “just another CRUD application.”

Other Features

There’s lots of other weird features that came out from me trying out new ideas and playing with the frontend code. The main features you should check out are:

The entire application logic is contained in the JavaScript code for the XUL interface.
The interface/application logic is controlled by a Finite Sate Machine that controls the asynchronous actions from both the user and the backend server.
The FSM was built using a little framework library I wrote while developing CookbooXUL. It’s something I’ve been tinkering with for about a week so it’s a little rough.
The FSM easily handles multiple events from multiple sources. For example, when you do a search it handles either hitting enter or pressing the SEARCH button, events from a timer that controls a spinner, the asynchronous data reply from the server, and an event passed back by the server in the HTTP headers. Without the FSM getting this right would drive you insane.
There are lots of handy functions in the FSM framework for handling common things like dealing with the DOM, doing client side XSLT, and other processing jobs.
Because the whole thing is controlled by an FSM doing everything asynchronously, the UI seems more responsive. The requests and processing still take the same amount of time, the difference is that the UI doesn’t “freeze” when the user clicks a button, but instead stays responsive.

Checking Out The App

Before you get into how the application is built you should go and play with it. I have a temporary setup running which you can play with.

If you want to check out a screenshot of the application then try this one as an example.

If you want to get the code then you can download the source and extract it. You’ll then need to do the following:

Go to the LargoRecipes RecipeML archive and grab all of the recipes you can. They are held in zip files.
Go into the public directory and start extracting each zip file. I didn’t want any of them to step on others so I made a bunch of directories matching the numbers in the zip file. You can skip this step though.
You need to either translate or clean out any files with # in the name. I did some shell magic to remove the # character, but you can just do find -name "*#*" -exec rm {} \; to zap them. Those recipes suck anyway.
Then run: ruby indexer.rb ../db/recipe_index xml from the public directory (not the xml directory).
Once the index is created (it could take a while) you’ll be able to go to the cookbooxul top-level directory and run script/server to start the server. If you have lighttpd installed (very most latest) then you can also try script/lighttpd.

That should get you started no problem, but let me know if you have issues. The main thing is getting the files indexed, and making sure they are in the public/xml directory. The other key point is to get rid of any files with # in the name.

Architecture

CookbooXUL is kind of an experiment in pushing all of the logic out to the client and keeping the backend as simple as possible without losing features. One major disadvantage of doing fatter clients is that you have to struggle with deploying them to clients and doing a controlled roll-out. This is a huge pain in the ass and is the major driving force behind HTML only interfaces.

With XUL and XPFE you can do a completely “thick client” application that uses a ridiculously simple backend to get the data it needs, but still get the deployment heaven of an HTML application. In many cases it isn’t necessary to use a XUL application since HTML user interfaces these days can handle most situations just fine. A XUL interface with a fat XPFE based client though has a lot of advantages including a wider range of client side technologies available, and an application that looks like it’s a desktop application.

For CookbooXUL I went with all of the logic sitting on the client in the web browser, and then had the backend provide all of the services needed. These services simply return XML results which are converted into UI elements and alter the client’s interaction. It turns out that this was pretty darn simple, and I even wrote a full framework to support the application at the same time.

Rails Background

You should have a Rails understanding before you dive into this little article. I’m not going to go over Rails at all, but take a look at the recipes_controller.rb file for the code. It’s pretty simple and completely documented.

There are three things you should realize about the controller which are specific to XUL interfaces, and particular to the FSM Framework:

You have to set the HTTP header ‘content-type’ to application/vnd.mozilla.xul+xml or else your user interface won’t work. It will usually render but just won’t function right. You’ll see lots of weird permission errors for things you shouldn’t have problems accessing.
The XML you return for any XUL elements needs the namespace xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul" or else it won’t get recognized. This includes even little partial elements you paste in later with JavaScript.
The session_store_defaults function in recipes_controller.rb has a strange line of code: response.headers['X-AppState'] = "END" There’s three of these setting the X-AppState HTTP header. This is a special response header which, if set, is interpreted by the FSM framework as an event for the FSM to processes. This lets the backend pass important events to the client without having to embed them in returned data.

Once you get this and how the Ruby/Odeum library is being used then you should be ready to get into the actual CookbooXUL code.

The Frontend Client

The CookbooXUL consists of a simplified Rails backend and a much fatter frontend client that uses XUL, JavaScript, XmlHttpRequest, and XSLT to create the UI. As I wrote the CookbooXUL interface I also wrote a small framework that simplifies creating an FSM based logic controller that handles asynchronous events from the user and the server at the same time.

The framework simplifies doing a lot of things, but you’re probably better off reading the documentation and playing with the functions. Here’s a quick list of what it does:

Makes it easy to construct and use Finite State Machines.
Has functions for processing XmlHttpRequest objects asynchronously.
Let’s you easily paste in client-side XSLT results.
Includes an in-memory logging facility for easy debugging.
Gives you a full set of callbacks for intercepting transitions, missing events, etc.
Simplifies working with the DOM by giving you some nice methods like find, replace, setattr. These rock compared to document.getElementById("someid").setAttribute("someattr", "somevalue").
Can process events passed back by the remote application in the X-AppState header as mentioned previously.
Allows you to register a timer that will send back an event after a given amount of time.

The framework doesn’t deal with fancy stuff like faders, drag-n-drop, or other stuff. It’s purpose is to make it easier to hook the logic of the client into the backend and make the interactions consistent. It wouldn’t be hard at all to use another framework, but you might run into function clashes since I made many of the functions first level for convenience.

Finally, the framework isn’t cross browser. I only care about doing XUL applications with it, so that’s all I worked on. If you want IE you should look at some of the Ajax libraries out there as they have much better support for this stuff under that platform.

The Client FSM

If you take a look at the recipes.js file you can see the FSM definition right at the top. All you need to use the FSM framework is to specify the states, events, and the code to run for each. The states are implemented as a hash map that is passed into the new FSM({}) constructor. Each state then has another hash consisting of each event that it can handle. Finally each event has the function to run when the event is triggered.

Here’s an example from the recipes.js file:

    start: {
        INIT: function(fsm, data) {
            find("query").focus()
            fsm.request('/recipe-transform.xml', 'processing', 'READY')
            fsm.logShowing = false  // keep this around for the log toggle
        }
    },

In this code we’re saying the following:

There’s a ‘start’ state. The state named ‘start’ is selected by default and you must have it.
The ‘start’ state answers one event named ‘INIT’. I do the events in uppercase to differentiate them, but do as you wish. It’s the name that matters.
The INIT: function(fsm, data) part attaches a function that takes the fsm and any necessary piece of data. In this case we just ignore it. You actually could drop the second parameter if you wanted since JavaScript will safely ignore it.
The onload="fsm.handle('INIT', this)" is found in the app/vies/recipes/index.rhtml file in the first window tag. This basically sends the fsm we defined the ‘INIT’ event when the window is first loaded, and thus kicking it into gear.
When the fsm receives this ‘INIT’ event from the window onload handler the FSM object calls the attached function.
Inside this function, you’ll notice we use the fsm.request('/recipe-transform.xml', 'processing', 'READY') which grabs the XSLT we need from the server. What’s interesting is the second two parameters, which say to transition into the ‘processing’ state and then fire the ‘READY’ event once the request returns.
Once the ‘INIT’ event is processed the fsm stays in the ‘start’ state until the request that it sent comes back. That’s when the fsm transitions into the ‘processing’ state and starts processing with the first ‘READY’ event.

That’s basically how the entire system works. You create a collection of states and events, and then use either XUL tag handlers (onload, oncommand, onclick, etc.) or asynchronous request/response calls to transition the FSM and handle events.

Other Important States

You should also review these other state:EVENT functions as they have interesting functionality in them:

processing:LOG—This actually displays or clears and hides the internal log file. This is tied to the ? button on the interface so you can put log("debug", "some message") function calls in your code to debug a problem. This is much easier than using alerts.
paging:RESPONSE—On line 54 you can see fsm.handleRequestEvent(req, 'MIDDLE'); This is how you handle the X-AppState HTTP header so that it gets processed as an event. You need to transition into the state you want to handle the event, but it will transition for you. All you need to do on the backend is set the X-AppState to the string for the event you want. See above where I talk about it in the recipes_controller.rb file.
viewing:VIEW—This function handler uses the replaceXSLT() to take an XSLT you’ve registered with the FSM and replace a part of the XUL interface with the results.
processing:READY—The XSLT file is registered with the fsm in the processing:READY handler using fsm.configureXSLT('recipeDisplay', req.responseXML). This lets you asynchronously gather XSLT files and then use them later indirectly, rather than having to remember URLs and download them all the time.
waitig:TICK—This is responsible for firing off a timer tick that rotates a cheesy little ASCII art spinner in the SEARCH button. You’ll notice that the waiting:TICK handler doesn’t have any transitions. What happens is that it just keeps firing once after each time it is triggered. When the request finally comes back it transitions to another state that does not have a ‘TICK’ event. The FSM will simply ignore events that aren’t available for a state, so when the timer fires the next ‘TICK’ event it gets dropped on the floor and things move on smoothly.

You should be able to figure out how the rest of the FSM works from this. The main thing is to play with it then use the ’?’ button to look at the log. I’ve setup a bunch of the event handlers at the end to log transitions and unknown states/events so you can see what’s going on.

Supporting Code

After the FSM you’ll find a bit of supporting code. This code is typical JavaScript that is used to turn buttons on, disable them, alter DOM objects, etc. After this code are a set of handlers that I’ve setup for which log various events. Normally you wouldn’t put code like this in production, it’s just there for demonstration.

You should also take a look at the app.js file to see the rest of the available functions. There’s a lot of handy stuff, but not much of it is required. If you prefer to torture yourself with the DOM API then you can skip the simplifying functions I’ve got.

Using The FSM Framework

To actually use the app.js code you need to of course include it prior to your own FSM definition. Otherwise you won’t have any of these functions.

You may use the FSM Framework under the LGPL license for now. Nothing too formal, just please send me back any bug fixes or enhancements you may have done. I’m interested in doing a small chat client with this so if someone wants to beat me to it then let me know.

Driving Home Drunk With A 6’8” Transvestite

I wanted to include a small blurb about my thoughts on this little XUL experiment. These are totally my opinion, so take them as you want.

One thing I found about XUL applications is that they are stuck between being “web apps” and being “desktop apps”. I realized this when someone commented, “The text in your lower panel is too wide for my 1600×1200 pixel screen.” Turns out he had his browser completely maximized and it just didn’t dawn on him that he should probably try shrinking the window down before he used the application. For whatever reason, he thought that since the application is in a web browser it should try to defend against all possible window-maximize-stupidity and give him a nice printed page layout.

But it’s more like a desktop app and is meant to be used in a thin configuration, which is hard and annoying to control since it’s a web browser really. Damn.

Now, imagine this same fellow has accidentally maximized the tools panel in his Adobe™ Photoshop™ application. He’d see it was huge and ugly, realize this isn’t normal, and then shrink it down. He wouldn’t complain about the ugly typography of the menus, the weird striped background of his OSX panels, or the strange window organizations. He’d just deal with it and move on since he’s using the application to get something done.

Desktop applications are held to different standards of usage and appearance than web applications. I’m not sure why. People will use ugly ass Vim or Emacs and not shed a single tear, but show them anything other than a Zen Garden layout on a web site and they instantly are experts on how much your site sucks.

I see now that XUL has a distinct disadvantage as people don’t know whether to think of it as a desktop application or a web application. It looks like a desktop application. It feels like a desktop application. But wait, it sizes like a web application. Oh, and look it has web browser controls on top.

It’s like people are checking out CookbooXUL’s ass, roaming their eyes over its nice rack, then bam! They see facial hair and an adam’s apple. They suddenly become confused and don’t know how to evaluate it. “Oh no, am I using a web application? Does that make me gay? Holy crap, this thing is suddenly ugly for a web application. I mean sure, it’s got a sexy simple fast as hell body, but damn that’s one ugly mug. Could I switch over? What’s the point really? I’ll just stick to HTML I guess.”

Oh well, it was fun.