I left my room last night headed for a night of gweeping in the free-WiFi lobby at my hotel (compared to $10/night in the room). In the elevator, I ran into Jason Hoffman from Joyent (Spotstory is a customer) and was promptly invited to a Joyent event at the Lucky Lab Brew Pub.

I can’t comment on the stronger stuff, but the root beer at the Lucky Lab was wicked good (sorry, I need to keep my provincial New England self in check)!

Thanks so much to Jason, David, Kristie, and all the other folks from Joyent for hosting a great event.

This morning, my karma has been balanced, as Jason is using my HDMI-to-VGA converter for his Scaling a Rails Application from the Bottom Up Thursday morning tutorial.

Aron pointed me to Josh Kenzer’s interview last month with Twitter Developer Alex Payne.  The newsworthy bit is that Alex states that Rails applications don’t scale well:

Running on Rails has forced us to deal with scaling issues – issues that any growing site eventually contends with – far sooner than I think we would on another framework.

We’ve actually been told this a lot by Rails folks.  Hopefully, someday the demand for Spotstory will be so great that we’ll have some first hand knowledge on the matter!

Having said that, there are already a couple of areas where we’ve already bumped into issues while using Rails.

Issue 1: Fixtures break down

A single set of fixture data just breaks down after you’ve created a certain number of tests.  I’ve talked about this before and it’s what caused us to create Isolated Fixtures.  So, we innovated our way around that.

Issue 2: The trap of CRUD-is-King

This is an issue we’ve encountered in the past couple of months, and a lot of what we’re doing now is geared towards ameliorating this condition.  But first I should probably explain what I’m talking about!

Rails lets you get a lot done quickly in the sense that you create a model, lots of code gets created generated, and you start seeing features work almost immediately.  Awesome!

Now, everyone warns you that you’ll throw away a lot of this generated code, and you do.  You refactor something here, change the presentation of something there, and you think you’re pretty much done.

But you’re not.  Because what you’ve done (at least in our case)  is expose features in a way that makes sense in terms of pure CRUD, but it has little to do with how regular people think about doing what they want to do.

I think what I’m trying to say is this: since so much of the application is automatically derived from the data model, the interface you end up exposing is tied to closely to the data model.

Maybe if we were more skilled designers we wouldn’t have gotten ourselves into this situation.  Perhaps we needed a better design methodology.

Also, I concede that Rails doesn’t force you to do this, or keep you from fixing the problem.  This is not the end of the world.  Very likely, this is also a problem in other frameworks.

But Rails’ CRUD-is-king mindset does do a lot to steer you down this path.  Again, I’m not saying this is a reason to not use Rails, just pointing out our own experience.

In closing

In closing, we’re happy with Rails and Ruby.  But we also understand that they’re tools, and all tools have their limitations.

I hope we’re not excommunicated from the Ruby on Rails community for this! :)

If you have been following this weblog for a while, you have probably figured out that Spotstory is a Ruby on Rails application.

Over the past few days, I have been working to upgrade Spotstory from Rails 1.1.6 to Rails 1.2.3.

There is very little comprehensive information about some of the problems encountered when upgrading a Rails application from the 1.1.6 release to 1.2 (specifically, Rails 1.2.3).

Once You Freeze, Step Cautiously

Upgrading your Rails application is easy! Update, deploy, then go enjoy your apple pie! Right? Wrong.

% rake rake rails:freeze:edge TAG=rel_1-2-3

So far, so good. Let’s see if anything runs.

rake rails:update --trace
rake aborted!
uninitialized constant ActionController
/usr/local/gems/rake-0.7.1/lib/rake.rb:1948:in `const_missing'
config/../config/environment.rb:32

Why did this happen? Did the Rails installation get corrupted during the upgrade? No. Far, far simpler. On line 31 of config/environment.rb, I had a raw reference to ActionController. Due to major dependency changes in Rails 1.2, this no longer works.

# Rails 1.1.6 technique for setting session expiry:
ActionController::Base.session_options[:session_expires] = Time.mktime(2037)

In Rails 1.2, it appears that this sort of configuration can not appear within the Initializer block. After moving session configuration out of the Initializer block, things started running.

Deprecations & Warnings

I decided that every deprecation message would be squashed and every warning removed. I worked on most of the warnings before even trying to get all our tests to pass; they were just too noisy for my aesthetics.

DEPRECATION WARNING: The :dependent => true option is deprecated and
will be removed from Rails 2.0.  Please use :dependent => :destroy
instead.  See http://www.rubyonrails.org/deprecation for details.
See http://www.rubyonrails.org/deprecation for details. (called from
has_many at vendor/rails/activerecord/lib/active_record/associations.rb:558)

(Yes, that link is actually printed twice!)

These deprecation messages are great — they tell me what has changed and how to improve my code, but they reference source files and line numbers within the Rails core. This is not helpful. I was able to find some instances of deprecated usage by recursively grepping our source (from within Emacs, which makes results electric). Very quickly, I turned to a heavier hammer.

ActiveSupport::Deprecation.debug = true

Toggling this debug variable exposes the full callstack which caused the deprecation message.

Here are the changes I needed to make in order to eliminate all our deprecation messages:

1. Some model helpers have gone away or have changed in call syntax. Rewrite model methods in the following way:
   

   Rails 1.1.6	Rails 1.2.3
   Model.find_first	Model.find(:first)
   Model.find_all	Model.find(:all)
   Model.type	Model.class.to_s
   Model.count(conditions)	Model.count(:all, :conditions => conditions)

2. Has many associations no longer provide the has_ASSOCIATIONs? and ASSOCIATION_count methods. Rewrite:
   

   Rails 1.1.6	Rails 1.2.3
   Model.has_ASSOCIATIONs?	! Model.ASSOCIATIONs.empty?
   Model.ASSOCIATION_count	Model.ASSOCIATIONs.size

3. Rewrite controller/view instance variable references:
   @params, @session, @request, @response all become params, session, request, response.
4. Move to the new form generators.

   form_tag ... end_form_tag

   becomes:

   form_tag do ... end

   The same is true for form_remote_tag references. In addition, the form methods no longer need output-generating RHTML rules (remove the “=”).

5. The link_to helper prefers a different way for specifying that a link will generate a POST operation.

   link_to ... :post => true

   becomes

   link_to ... :method => :post

6. Association dependencies now support more granularity, which means that the old-style:

   has_many ... :dependent => true

   becomes:

   has_many ... :dependent => :destroy

7. Add parentheses and remove whitespace between parens and method names to prevent ambiguity. Not clear why these issues didn’t cause issues in Rails 1.1.6.

Fixing Broken Tests

With our application beaten into quiet submission, it was time to turn to our tests. A good majority of our tests still worked, but a few changes were needed to get everything passing.

1. Content-Type headers now include charset information; make test helpers resilient to the presence of charset.
2. assert_tag with a string value for the :content option now performs a strict match against the content for any sub-tag. As a result, some :content options were converted to regular expressions.

   Determining when a regular expression is necessary is not obvious, especially when the match text is embedded within another markup tag.

At this point, all of our tests passed.

Dependencies

Unfortunately, we weren’t done. The development server would unexpectedly give errors like:

User expected, got User

In addition, our User model would get missing_method errors for methods which clearly exist. The problem is an interaction between the Rails 1.2 dependency rewrite and plugins which have dependencies upon application models. Ramble On discusses a similar dependency issue.

We use a number of plugins in our application. One of these is a customized version of acts_as_commentable. This plugin defines the Comment model and expects a User model to be defined by your application.

In development mode, plugin models are not reloaded. The dependency infrastructure unloads the User model, but doesn’t realize that there are still live references from the plugin-defined Comment model. Subsequent references to a comment’s user information will generate an AssociationTypeMismatch exception giving the cryptic “expected User, got User” message. Because this is a reload problem, errors don’t happen on initial load. The errors start the second time a comment-displaying page is loaded.

Web searching didn’t reveal much at first. Some people recommended explicitly requiring the User model in application code. Others recommended replacing references to a User object with references to its id. Another solution called for explicit :class_name in the association declarations. Some of it worked some of the time; none of it worked consistently.

I started looking through the Rails dependency code and found that Nicholas Seckar (ulysses on IRC and in SVN) was responsible for much of the rewrite. After explaining my missing_method errors in an email, he replied:

See http://api.rubyonrails.org/files/vendor/rails/railties/CHANGELOG.html
Look under the changes for 1.2.0, 4th one down.

This references a late change to Rails 1.2 by Rick Olson:

Ensure plugins are in the Dependencies.load_once_paths collection by default. [Rick] If you really want your plugins to reload, add this to the very top of init.rb:

Dependencies.load_once_paths.delete(lib_path)

Ok. I need to reload my plugins. Bleh. Further, this has to happen in environment.rb, not in environments/development.rb. Bleh.

# Array of plugins with Application model dependencies.
reloadable_plugins = ["acts_as_commentable"]
# Force these plugins to reload, avoiding stale object references.
reloadable_plugins.each do |plugin_name|
  reloadable_path = RAILS_ROOT + "/vendor/plugins/#{plugin_name}/lib"
  Dependencies.load_once_paths.delete(reloadable_path)
end

None of this forced plugin reloading feels right. Rails should just figure it out, yes? More mail to Nicholas, leading to his response:

The problem is related to associations — the class object is cached and thus the class is never garbage collected, even though it is ‘removed.’ This causes all sorts of problems…

Since it only happens in development mode, it really doesn’t matter.

He’s partially right. It doesn’t affect production applications, but it really does affect peace-of-mind. I’m sure lots of people are freaked out when they bring up their upgraded app in development mode and find all sorts of things unexpectedly broken.

Update: I was wrong about the session configuration changes in Rails 1.2. I have updated this post accordingly.

It is quite easy to extend your Ruby on Rails test infrastructure so every document is checked for validity.

1. Install the plugin assert_valid_markup.
2. Install my test_validation_helper.rb in your test directory.
3. Require the validation helper from your test/test_helper.rb:

   require File.expand_path(File.dirname(__FILE__) +
       "/test/test_validation_helper.rb")

After those three simple steps, your HTML and RSS documents are now being validated each time you run tests.

I am giving a brief presentation on this topic at the April 10, 2007 Boston Ruby Group meeting.

You can flip through a PDF of my slides if you would like more details.

Update: In addition to the PDF, my presentation is now hosted on slideshare and shown below. If you are reading this post in an aggregrator, you will need to click through for the show (or just use the PDF version).

As I posted on Sunday night, I participated in the BarCamp Boston 2 programming contest.

Mike Walsh has been gracious enough to edit and host a video of the contest presentations. Many thanks to Mike for his work and to everyone else for making BarCamp Boston 2 a great experience!

I’d like to describe and provide code for what I’m calling “isolated fixtures”. Isolated fixtures allow you to have an independent and separate set of fixtures for Test::Unit::TestCase in your Ruby on Rails project.

I’d already run into the situation a couple of times where adding info for my new test into existing fixtures broke existing tests, either due to assumptions of about the fixture data, or because multiple tests “borrowed” fixtures from previous test.. According to some other pieces I’ve read on the net, the situation would only get worse as our test suite grew (and ours does grow). So, instead of becoming adept at the sort of limbo-dancing required to finesse new tests into the testsuite, I decided to come up with a way to let each set of tests have their own fixtures.

Now, I don’t know if this is considered a hack, or another demonstration of the flexibility and power that is Ruby on Rails. I’m sure you’ll all let me know.

This works for me with Rails 1.1.6. I have no idea if it will work in 2.0. (2.0 might even have this built in for all I know!)

Usage

Let’s assume you have a test class called AccountControllerTest. In this test where you normally use fixture instead user isolated_fixture instead:

  isolated_fixture :customers, :accounts

Instead of putting the fixture data into test/fixtures it will create a subdirectory there based on the name of the test class. In this case the path of the directory and the fixture data will be:

  test/fixtures/account_controller/accounts.yml
  test/fixtures/account_controller/customers.yml

Code

Here is the code. Put it in your test/test_helper.rb, or do what I did and put it in test/test_helper/fixtures.rb and require it into test_helper.rb.

class Test::Unit::TestCase

  ## Each subclass needs its own copy 'fixture_path' so replace
  ## the existing single class variable.
  remove_class_variable :@@fixture_path
  class << self
    remove_method :fixture_path
    remove_method :fixture_path=
  end

  class_inheritable_accessor :fixture_path
  self.fixture_path = RAILS_ROOT + "/test/fixtures/"

  ## Add specialized fixture class methods and alias them.
  def self.standard_fixtures(*table_names)
    self.fixture_path = RAILS_ROOT + "/test/fixtures/"
    self.fixtures_without_fixture_path(table_names)
  end

  def self.isolated_fixtures(*table_names)
    fixture_subdir = self.to_s.ends_with?("Test") ? self.to_s[0..-5].underscore : self.to_s.underscore
    self.fixture_path = RAILS_ROOT + "/test/fixtures/#{fixture_subdir}/"
    self.fixtures_without_fixture_path(table_names)
  end

  class << self
    alias_method :fixtures_without_fixture_path, :fixtures
    alias_method :fixtures, :standard_fixtures
    alias_method :original_method_added, :method_added
  end

  def setup_with_fixtures_and_with_fixture_path
    setup_with_fixtures_and_without_fixture_path
  end

  ## Provide a default 'setup' method.
  alias_method :setup_with_fixtures_and_without_fixture_path, :setup_with_fixtures
  alias_method :setup, :setup_with_fixtures_and_with_fixture_path

  ## Handle the case where the user provides a 'setup' method for the tests.
  def self.method_added(method)
    case method.to_s
    when 'setup'
      unless method_defined?(:setup_without_fixtures_and_without_fixture_path)
        alias_method :setup_without_fixtures_and_without_fixture_path, :setup
        define_method(:setup) do
          setup_with_fixtures_and_with_fixture_path
          setup_without_fixtures_and_without_fixture_path
        end
      end
    else
      self.original_method_added(method)
    end
  end

end

Drop a line and let me know if you found this useful, or if you found a bug. I'll post any updates.

Update

Here is a slide presentation covering the same material.

Look at our way-cool custom markers! Drop shadows and everything!

We are more than just a little excited.

I recently needed to import a large number of DBF files into a MySQL database. These DBF files may end up providing a large set of test data for spotstory.

After searching far and wide and encountering many stale and broken links, I came across a version of dbf2mysql published by Josh Drew.. I had a great deal of success with this program.

I ran into one hiccup with dbf2mysql; errors were not reported if the DBF file contained a column name which conflicted with a MySQL keyword. This is resolved by changing the calls to mysql_query as follows:

    if (mysql_query(SQLsock, query) != 0) { /* error handling */ }

Previously, only return values of -1 were identified as errors. According to the MySQL documentation for mysql_query, all non-zero returns indicate an error.

Hopefully, Josh will incoorporate this improvement into his version of dbf2mysql. It truly saved me hours of time!

At some point during the day today Aron made the 100th commit to the spotstory source repository. (He’s usually the one who is committing code while I’m staring dumbfounded at my screen.)

Yes, it’s a small milestone, but the path from our starting point to success will be marked at irregular intervals by lesser and greater acheivements: we try to celebrate even the smallest accomplishment before moving on — even if its just a congratulatory IM.

So, here’s to the 100th commit. Hooray for us.

(p.s. We’re hoping to have some larger milestones to announce in the next several weeks.)

I’ve spent too much of my day installing SUSE Linux 10.1 as a guest OS using VMware Player on my Dell Latitude D620 running Windows XP. It’s not rocket science, but having never used VMWare before I sent myself down a couple of long, meandering dead-ends. Should you want to do this, hopefully, this will keep you from making the same wrong turns.

But first, a side note! I can hear you asking now “Why do you want to do this?” Well, I do. I hate developing under Windows, hate Cygwin even more (call it personal preference), but still prefer using a text editor (emacs) and a unix command line. Yeah, I should have bought a Mac instead. That’s another story. Anyway, here is the process.

Preliminaries
First, go get all the stuff you’re going to need:

• Download the SUSE Linux 10.1 ISOs and burn them onto CDs. I used Roxio Creator Plus which came with my machine to do the burning.
• Download VMware Player. Install.
• Download this zip file (courtesy of this fine post which was quite helpful.) Unzip this into a directory on a disk with plenty of space. Rename all of the files from “OS” to “sl10-1″.

Modify the VMware configuration file
Make this series of edits to “sl10-1.vmx”.

First, we need to tell VMware that we do not have a floppy drive. Remove this line:

floppy0.fileName = "A:"

Replace it with:

floppy0.present "FALSE"

Now we need to tell VMware to load the operating system from the CDs we will put in the CD drive. Change these lines:

ide1:0.fileName = "c:\image.iso"
ide1:0.deviceType = "cdrom-image"

to:

ide1:0.fileName = "auto detect"
ide1:0.deviceType = "cdrom-raw"

Finally, change this line:

ide0:0.fileName = "OS.vmdk"

to:

ide0:0.fileName = "sl10-1.vmdk"

Run VMware and install SUSE
Put the first first SUSE CD in the CD drive. Run VMware and load the “sl10-1.vmx”. At this point in time, you should be able to install SUSE as normal. When I did this, almost everything worked, except my wireless network.

Getting the wireless network to work
After you’ve installed SUSE if you type “lspci” you will see that SUSE knows about an AMD networking card, but not your actual wired or wireless network card. THIS IS CORRECT.

What is happening is that VMware shows this dummy network card to SUSE, and patches it through to an actual network device on the host machine. If you have more than one active network connection, VMware will pick which one of those network connections for you. In my case, it chose the wired network connection, which gave the impression that the wireless network connection was not working.

You can tell VMware which networking connection to use by running the Virtual Network Editor:

<Program Directory> > VMware > VMware Player > vmnetcfg.exe


Go to the “Host Virtual Network Mapping” tab. Change the value of VMnet0 from “Bridged to an automatically chosen adapter” to your wireless card (in my case a Dell 1490). Hit “Okay” and everything should be okay!

Have fun
Now you should be able to have all sorts of wireless unixy fun. Good luck!