Latest News >> 2008-09-29

I read Obie’s most recent post about his intense passion for Loverboy’s quintessential anthem, “Lovin’ Every Minute Of It”. I find the early 1980’s music is inspiring and uplifting and definitely suited to such important things as corporate culture, recruiting, and motivating the troops to do better. Yes, nothing gets a worker working better than a little Loverboy right in their ear.

2008-09-25

Don’t forget folks, the FU NYC show will be in a few hours (7pm-9pm). I’ll be icecasting this one at http://zedshaw.com:8000/fu_nyc and as usual you can use VLC, mplayer or many other players to play the stream.

2008-09-17

I got into music school last week and I’m going to study guitar exclusively for the next year. This is something I’ve always wanted to do, but just never had the chance. Either I wasn’t good enough (being self-taught for so many years) or I just didn’t have the money. After being laid off and getting a small package I decided to practice my ass off on the guitar, do a few live shows to get ready, and then audition for a school in the city.

2008-09-04

The Freehackers Union NYC show went insanely well. I managed to pull off a full live internet feed of the audio to people in FU and the show at the same time. We had about 10 newbies show up to give their first five minutes and 11 listeners on IRC/icecast. Some people showed up just to hang out, so we relaxed the rules and let them stay to build an audience. Overall, there were some cool projects presented and everyone had a good time.

Ruby/Event is Now Dead

Hi folks. After struggling with problems getting libevent to play fair with Ruby’s threading, I decided that it was too much effort for the gain and am dropping Ruby/Event. Feel free to take the code here and use it. You can contact me if you feel like maintaining it.

People looking for the SCGI Rails Runner then go to it’s new project page for more information.

Update: Sep 4 v. 0.4.2

As mentioned on the index page this release fixed a couple little error reporting oversights for SCGI, and includes a complete scriptable telnet implementation that should be very correct. These instructions still mention 0.4.1 as I want to test 0.4.2 more and clean up the telnet implementation to make it an actual included Myriad library similar to SCGI.

You can download the tar.bz2 or the rubygem if you want to test it.

Update: Sep 1 v. 0.4.1

The latest release is 0.4.1 which fixes a problem with SCGI exiting, and corrects documentation for running SCGI Rails under lighttpd. This release is also lightly tested with Apache.

Ruby/Event and Myriad

Ruby/Event is a Ruby extension to the libevent library by Niels Provos to multiplex IO in the fastest way possible for your system. Ruby/Event tries to be as faithful to the libevent API while still looking like idiomatic Ruby so that both experienced libevent users and Ruby programmers can learn the API quickly. Finally, it completely wraps the library including the bufferevent structures, timeouts, and signals.

In addition to this basic functionality Ruby/Event includes many examples and extensive RDoc documentation for your reference. I actually think the documentation for Ruby/Event is better documentation for libevent in general. I made sure that I read through the code to find out what functions did and documented any inconsistent behavior. You should hopefully be able to get started with Ruby/Event and probably even learn a little bit about libevent at the same time.

Some people read “Ruby/Event” and think it’s for doing calendars even after reading that it uses libevent and handles buffers and all sorts of weird crap that has nothing to do with calendars.

Just to be clear, this is for writing clients, servers, and the protocols
between them.  If you want calendars then you're in the wrong place.

Getting The Goods

You can get:

There is a rough getting started guide coming soon, when I find more time.

The Myriad Framework

Ruby/Event also includes a framework called Myriad that I’ve probably done in every language I know. The purpose of Myriad is to make it easier to work with TCP/IP events and data without killing your performance or having to learn yet another massive framework architecture. Myriad works kind of the way you probably already think about a protocol—as a series of expect/receive/send operations.

Myriad does this by adding two more layers to the current TCP/IP protocol stack:

  • Application—Implements the application operation, sometimes not required.
  • Control—Works Translator to control protocol flow and operation.
  • Translation—Converts TCP vomit into clean messages and events.
  • Transport—TCP
  • Internet—IP
  • Interface—Ethernet

The Translation layer is already implemented and you usually don’t need to bother with it at all. The part you write is the Controller and your Application if you have a more complicated system. Many times you can mash the Control and Application layers together, but other times there is a clear separation between the two.

I think this latest Ruby incarnation is the best as the features of Ruby really make it easy to implement protocol processors easily and quickly. Combined with Ruby/Event and Ruby I’ve managed to make Myriad a very nice clean library while still keeping the performance high compared to other Ruby frameworks. I really hope you’ll like it.

Major Unfinished Parts

While Myriad supports abstractions for reading messages, most events, timeouts, signals, and lots of grunt work, it is missing one big abstraction: writing messages. Currently I found it easier to just write a simple function to send the messages, but I would like to make the analog to the Reader for writing.

Any ideas are welcome, but for now simply writing to the buffer seems to work great. Just keep in mind that this will change in the future, but your current use of writing will not be impacted.

Another thing is that I’ve only used it for TCP/IP and not UDP or any other protocols. It probably would work but I just haven’t bothered. I don’t use UDP so let me know if you get it to work and have any suggestions. Don’t bother me though if you just want to tell me that I should only support UDP because it’s faster. Tell me it’s faster after you’ve implemented all the things TCP/IP provides you and then we’ll talk (after I demonstrate your UDP speed hitting 0 over a bad satellite link).

Many Built-in Readers

Myriad includes a lot of the standard message formats that you find in many protocols these days, and makes it easy to create your own without having to modify the Myriad::Translator class to use them. The list of default supported formats are:

  • FixedLengthReader—Reads a fixed length chunk of data.
  • LineEndReader—Reads until it finds a line ending, which can be anything you want.
  • NetStringReader—Reads a netstring as described by D.J. Bernstein for transmitting pre-defined length data in a simple text-only format.
  • ReadClosedReader—Just reads until the input is closed or another reader is requested.
  • VRecReader—Reads a vrec as described in Effective TCP/IP Programming with the modification that it uses an unsigned short in network byte order as the length header. A vrec is a great way to safely send data, and there’s an included examples/vrecho.rb.html script that lets you work with vrec protocols using text input.
  • UnpackReader—Super powerful as it lets you use the Ruby String.unpack function on any fixed length received input. The only downside is that it requires a fixed length message since it’s not easy to know the length of a pack expression from the expression.

A Complete SCGI Implementation Supporting Rails

There is also a complete SCGI implementation included that is agnostic to the web framework, but implements a sample runner for Ruby On Rails suitable as an alternative to the FCGI system in use today. It will let you run a Rails application with a set of child processors, stop them, and get their status from one convenient command line utility. Simply check out the scgi_rails command with:

scgi_rails --help

And you should be able to figure out the rest. There is also an example lighttpd.conf file that will run lighttpd against your SCGI children off a single port. Some folks claim to have problems with Apache’s SCGI implementation, so I’m looking into that.

I’m also interested in writing a similar command for Og+Nitro and any other web frameworks out there. Hell I’m interested in writing a whole new web hosting framework to replace the complete aging WWW stack that all these frameworks are built on. I really think Myriad could do this well and clean up a lot of the cruft that is lying around.

The SCGI implementation also supports the client side as well, which should help people test their SCGI servers and maybe even give them ideas for SCGI based systems.

There’s more information in the Getting Started with SCGI guide.

More Information

The best place to check for complete installation information and getting started is the README that comes with the source distribution. Keep in mind that the project is licensed under the LGPL license so if you make modifications and distribute them then you should send them to me.

Who the hell am I kidding, open source “users” are the absolute worst at giving back code they write. If you’re a good person you’ll give me back neat things you write, and I’ll of course give you credit. If you’re an evil bastard you’ll keep good stuff to yourself and pretend you’re an open source advocate.