Hacklog: Blogamundo — poking holes in the language barrier since approximately 1 month from now

An essential piece of software behind Blogamundo is a full-fledged multi-user RSS (and Atom) aggregator. We’re building the web UI with Rails, so we were naturally looking forward to writing our feed poller in Ruby. As it turns out, though, there aren’t really many RSS parsing libraries for Ruby, and the best ones still would require quite a bit of fiddling in order to be suitable for our purposes. After playing with a bunch of libraries I was quickly convinced of something that in reality I already knew: there isn’t anything better (or as hackable) as Mark Pilgrim’s Universal Feed Parser.

At first thought, having our web UI written in Rails and our feed poller in Python involves a trivial interoperability scheme: the poller loads parsed feed data into the database which the web UI can then query afterwards. The problem arises when we need to parse feeds directly from within the web UI and provide the user with immediate data. We needed a reliable way to make Rails communicate with the Python feed poller.

After much pondering about inter-process communication (and how they scale) I figured the simplest and perhaps most scalable way to do this would be to use HTTP. Make our poller run in a local HTTP server which Rails can query. The data format I chose, of course, was YAML. Currently our poller is running in Apache with mod_python but it can be made infinitely scalable via FCGI or SCGI.

So, here’s a code demonstration of what just decribed:

# python code
import sys
sys.stdout.write('Content-Type: text/yaml\r\n\r\n')
sys.stdout.write('{yaml_test: Hello, some_property: 2}')
	
# ruby code
require 'yaml'
res = YAML:: load open('http://127.0.0.1/test.py').read

Staggeringly simple (and stupid, some might say). This could result in a slew of different errors, so we also needed a reliable error handling scheme. It quickly made sense to have our error codes defined in an external file, and inspired by this post from Sam Ruby, I’ve written code to parse and dynamically generate exception classes in runtime, both in Python and Ruby. Here’s the relevant code:

# config file
MyExceptionX:
    code: 1
    message: This is MyExceptionX
MyExceptionY:
    code: 2
    message: This is MyExceptionY
MyExceptionZ:
    code: 3
    message: This is MyExceptionZ

# python code
import new
import syck # YAML parser for Python
	
ERROR_CODES = syck.load(open('config/errorcodes.yml').read())
	
class MyException(Exception):
    pass
	
for k, v in ERROR_CODES.items():
    nexc = new.classobj(k, (MyException,), {})
    nexc.code, nexc.message = v['code'], v['message']
    tostr = lambda self: \"{code: %s, message: %s}\" \
             % (self.__class__.code, self.__class__.message)
    setattr(nexc, \"__str__\", new.instancemethod(tostr, None, nexc))
    globals()[k] = nexc

# ruby code
	
ERROR_CODES = YAML::load open(\"config/errorcodes.yml\").read
	
module MyExceptions
    class MyException < StandardError
    end
    CODES = []
    for k, v in ERROR_CODES
        CODES[v['code']] = const_set(k, Class.new(MyException) {
            define_method :to_s do
                \"{code: #{v['code']}, message: #{v['message']}}\"
            end
        })
    end
end

I hope this proves useful to someone. I’m really not certain what the best approach to this problem is, but this one proved to be very reliable during testing. Suggestions are of course very welcomed.