I left my room last night headed for a night of gweeping in the free-WiFi lobby at my hotel (compared to $10/night in the room). In the elevator, I ran into Jason Hoffman from Joyent (Spotstory is a customer) and was promptly invited to a Joyent event at the Lucky Lab Brew Pub.

I can’t comment on the stronger stuff, but the root beer at the Lucky Lab was wicked good (sorry, I need to keep my provincial New England self in check)!

Thanks so much to Jason, David, Kristie, and all the other folks from Joyent for hosting a great event.

This morning, my karma has been balanced, as Jason is using my HDMI-to-VGA converter for his Scaling a Rails Application from the Bottom Up Thursday morning tutorial.

Right now, I’m in Portland, Oregon getting ready for RailsConf 2007. Even though I’ve been here for less than an hour, I’ve already had an encounter with Internet celebrities!

If you follow the Ruby on Rails community (and who doesn’t), you have probably seen the Ruby on Rails vs. Java commercial by the RailsEnvy.com guys. We STOOD TOGETHER waiting for our luggage! I was in complete, geeky fan mode.

I introduced myself and mentioned that I work on both Spotstory and a.placebetween.us. They had heard of our site! Does this make me cool? It makes me a geek.

A wicked geek.

Aron pointed me to Josh Kenzer’s interview last month with Twitter Developer Alex Payne.  The newsworthy bit is that Alex states that Rails applications don’t scale well:

Running on Rails has forced us to deal with scaling issues – issues that any growing site eventually contends with – far sooner than I think we would on another framework.

We’ve actually been told this a lot by Rails folks.  Hopefully, someday the demand for Spotstory will be so great that we’ll have some first hand knowledge on the matter!

Having said that, there are already a couple of areas where we’ve already bumped into issues while using Rails.

Issue 1: Fixtures break down

A single set of fixture data just breaks down after you’ve created a certain number of tests.  I’ve talked about this before and it’s what caused us to create Isolated Fixtures.  So, we innovated our way around that.

Issue 2: The trap of CRUD-is-King

This is an issue we’ve encountered in the past couple of months, and a lot of what we’re doing now is geared towards ameliorating this condition.  But first I should probably explain what I’m talking about!

Rails lets you get a lot done quickly in the sense that you create a model, lots of code gets created generated, and you start seeing features work almost immediately.  Awesome!

Now, everyone warns you that you’ll throw away a lot of this generated code, and you do.  You refactor something here, change the presentation of something there, and you think you’re pretty much done.

But you’re not.  Because what you’ve done (at least in our case)  is expose features in a way that makes sense in terms of pure CRUD, but it has little to do with how regular people think about doing what they want to do.

I think what I’m trying to say is this: since so much of the application is automatically derived from the data model, the interface you end up exposing is tied to closely to the data model.

Maybe if we were more skilled designers we wouldn’t have gotten ourselves into this situation.  Perhaps we needed a better design methodology.

Also, I concede that Rails doesn’t force you to do this, or keep you from fixing the problem.  This is not the end of the world.  Very likely, this is also a problem in other frameworks.

But Rails’ CRUD-is-king mindset does do a lot to steer you down this path.  Again, I’m not saying this is a reason to not use Rails, just pointing out our own experience.

In closing

In closing, we’re happy with Rails and Ruby.  But we also understand that they’re tools, and all tools have their limitations.

I hope we’re not excommunicated from the Ruby on Rails community for this! :)

Howdy, Spotstorians!  We did a major system update on April 24, 2007 around 11:45 ET.  We hope you enjoy.  This release contains:

• KML support See your Spots in Google Maps and Google Earth!
• Rails 1.2.3 We’ve updated to the latest and greatest Rails!
• Bug fixes Always some bug fixes for you!

KML

You can now view your Spots using either Google Earth or Google Maps.  There are a couple of ways to do this.

Google Maps

Currently, the only way to see a Spot is to provide a URL in the search box.  When Google has crawled the site (the next day or so), you’ll be able to find your Spot in the Google search results.

To see your Spot right now, go to a Spot, take its URL, copy it into the Google Maps search box, and append the suffix “kml” onto it like so:

  http://www.spotstory.com/spots/show/120.kml

To see what this looks like, just click here.

As I mentioned, Google hasn’t sucked our KML into their index yet, but when they do, you should start seeing your Spots appear in Google Maps under the “See user-created content” link at the bottom of the left-hand side of the page.

We’ll let you know when this happens.

Google Earth

To see your spots on Google Earth, you can go to the Spot, and click on the “KML” link under the map.  If you have Google Earth installed, it should load automatically.  (If you don’t have Google Earth installed you can can download it from here.)

You can also right-click to save the content to disk, and load it into Google Earth from there.

Rails 1.2.3

This isn’t a change you’ll notice, but we get a lot of traffic from the Ruby on Rails community, so we thought it made sense to mention that our production environment is now using Rails 1.2.3.  For details, you can see Aron’s great post about his Rails 1.2.3 upgrade travails.

Thank You!

That’s all for this update.  Please let us know if you find any problems.

Thanks to everyone who has contributed to the site either with content, comments, or attention!

If you have been following this weblog for a while, you have probably figured out that Spotstory is a Ruby on Rails application.

Over the past few days, I have been working to upgrade Spotstory from Rails 1.1.6 to Rails 1.2.3.

There is very little comprehensive information about some of the problems encountered when upgrading a Rails application from the 1.1.6 release to 1.2 (specifically, Rails 1.2.3).

Once You Freeze, Step Cautiously

Upgrading your Rails application is easy! Update, deploy, then go enjoy your apple pie! Right? Wrong.

% rake rake rails:freeze:edge TAG=rel_1-2-3

So far, so good. Let’s see if anything runs.

rake rails:update --trace
rake aborted!
uninitialized constant ActionController
/usr/local/gems/rake-0.7.1/lib/rake.rb:1948:in `const_missing'
config/../config/environment.rb:32

Why did this happen? Did the Rails installation get corrupted during the upgrade? No. Far, far simpler. On line 31 of config/environment.rb, I had a raw reference to ActionController. Due to major dependency changes in Rails 1.2, this no longer works.

# Rails 1.1.6 technique for setting session expiry:
ActionController::Base.session_options[:session_expires] = Time.mktime(2037)

In Rails 1.2, it appears that this sort of configuration can not appear within the Initializer block. After moving session configuration out of the Initializer block, things started running.

Deprecations & Warnings

I decided that every deprecation message would be squashed and every warning removed. I worked on most of the warnings before even trying to get all our tests to pass; they were just too noisy for my aesthetics.

DEPRECATION WARNING: The :dependent => true option is deprecated and
will be removed from Rails 2.0.  Please use :dependent => :destroy
instead.  See http://www.rubyonrails.org/deprecation for details.
See http://www.rubyonrails.org/deprecation for details. (called from
has_many at vendor/rails/activerecord/lib/active_record/associations.rb:558)

(Yes, that link is actually printed twice!)

These deprecation messages are great — they tell me what has changed and how to improve my code, but they reference source files and line numbers within the Rails core. This is not helpful. I was able to find some instances of deprecated usage by recursively grepping our source (from within Emacs, which makes results electric). Very quickly, I turned to a heavier hammer.

ActiveSupport::Deprecation.debug = true

Toggling this debug variable exposes the full callstack which caused the deprecation message.

Here are the changes I needed to make in order to eliminate all our deprecation messages:

1. Some model helpers have gone away or have changed in call syntax. Rewrite model methods in the following way:
   

   Rails 1.1.6	Rails 1.2.3
   Model.find_first	Model.find(:first)
   Model.find_all	Model.find(:all)
   Model.type	Model.class.to_s
   Model.count(conditions)	Model.count(:all, :conditions => conditions)

2. Has many associations no longer provide the has_ASSOCIATIONs? and ASSOCIATION_count methods. Rewrite:
   

   Rails 1.1.6	Rails 1.2.3
   Model.has_ASSOCIATIONs?	! Model.ASSOCIATIONs.empty?
   Model.ASSOCIATION_count	Model.ASSOCIATIONs.size

3. Rewrite controller/view instance variable references:
   @params, @session, @request, @response all become params, session, request, response.
4. Move to the new form generators.

   form_tag ... end_form_tag

   becomes:

   form_tag do ... end

   The same is true for form_remote_tag references. In addition, the form methods no longer need output-generating RHTML rules (remove the “=”).

5. The link_to helper prefers a different way for specifying that a link will generate a POST operation.

   link_to ... :post => true

   becomes

   link_to ... :method => :post

6. Association dependencies now support more granularity, which means that the old-style:

   has_many ... :dependent => true

   becomes:

   has_many ... :dependent => :destroy

7. Add parentheses and remove whitespace between parens and method names to prevent ambiguity. Not clear why these issues didn’t cause issues in Rails 1.1.6.

Fixing Broken Tests

With our application beaten into quiet submission, it was time to turn to our tests. A good majority of our tests still worked, but a few changes were needed to get everything passing.

1. Content-Type headers now include charset information; make test helpers resilient to the presence of charset.
2. assert_tag with a string value for the :content option now performs a strict match against the content for any sub-tag. As a result, some :content options were converted to regular expressions.

   Determining when a regular expression is necessary is not obvious, especially when the match text is embedded within another markup tag.

At this point, all of our tests passed.

Dependencies

Unfortunately, we weren’t done. The development server would unexpectedly give errors like:

User expected, got User

In addition, our User model would get missing_method errors for methods which clearly exist. The problem is an interaction between the Rails 1.2 dependency rewrite and plugins which have dependencies upon application models. Ramble On discusses a similar dependency issue.

We use a number of plugins in our application. One of these is a customized version of acts_as_commentable. This plugin defines the Comment model and expects a User model to be defined by your application.

In development mode, plugin models are not reloaded. The dependency infrastructure unloads the User model, but doesn’t realize that there are still live references from the plugin-defined Comment model. Subsequent references to a comment’s user information will generate an AssociationTypeMismatch exception giving the cryptic “expected User, got User” message. Because this is a reload problem, errors don’t happen on initial load. The errors start the second time a comment-displaying page is loaded.

Web searching didn’t reveal much at first. Some people recommended explicitly requiring the User model in application code. Others recommended replacing references to a User object with references to its id. Another solution called for explicit :class_name in the association declarations. Some of it worked some of the time; none of it worked consistently.

I started looking through the Rails dependency code and found that Nicholas Seckar (ulysses on IRC and in SVN) was responsible for much of the rewrite. After explaining my missing_method errors in an email, he replied:

See http://api.rubyonrails.org/files/vendor/rails/railties/CHANGELOG.html
Look under the changes for 1.2.0, 4th one down.

This references a late change to Rails 1.2 by Rick Olson:

Ensure plugins are in the Dependencies.load_once_paths collection by default. [Rick] If you really want your plugins to reload, add this to the very top of init.rb:

Dependencies.load_once_paths.delete(lib_path)

Ok. I need to reload my plugins. Bleh. Further, this has to happen in environment.rb, not in environments/development.rb. Bleh.

# Array of plugins with Application model dependencies.
reloadable_plugins = ["acts_as_commentable"]
# Force these plugins to reload, avoiding stale object references.
reloadable_plugins.each do |plugin_name|
  reloadable_path = RAILS_ROOT + "/vendor/plugins/#{plugin_name}/lib"
  Dependencies.load_once_paths.delete(reloadable_path)
end

None of this forced plugin reloading feels right. Rails should just figure it out, yes? More mail to Nicholas, leading to his response:

The problem is related to associations — the class object is cached and thus the class is never garbage collected, even though it is ‘removed.’ This causes all sorts of problems…

Since it only happens in development mode, it really doesn’t matter.

He’s partially right. It doesn’t affect production applications, but it really does affect peace-of-mind. I’m sure lots of people are freaked out when they bring up their upgraded app in development mode and find all sorts of things unexpectedly broken.

Update: I was wrong about the session configuration changes in Rails 1.2. I have updated this post accordingly.

We had a lot of fun presenting at the Boston Ruby Group last night.  We’d like to thank Tom Dyer and everyone else involved with organizing, setting up, providing space, and providing pizza.

Once again, the meeting was well attended: folks were standing along the walls and sitting on the floor.  This month’s program was more Rails-oriented than last month’s, but few people seemed to mind that.

Eric Mill from thoughtbot gave a talk on REST and ActiveResource.  The highlight was definitely Jester, a JavaScript REST implementation  developed at thoughtbot.  It’s modeled after ActiveResource and, boy howdy, does it look neat!

Jeremy Durham did a presentation on Memcache.  Jeremy has spoken at both of the meetings I’ve attended, and on both occasions he’s provided a lot of practical, meat-and-potatos info you can use.

You can find Aron’s presentation here, and mine here.

The group is always looking for speakers, so please volunteer!  If you’re in the Boston area and working with Ruby/Rails, it’s a great way to learn about the subject and to meet folks in the community. 

It is quite easy to extend your Ruby on Rails test infrastructure so every document is checked for validity.

1. Install the plugin assert_valid_markup.
2. Install my test_validation_helper.rb in your test directory.
3. Require the validation helper from your test/test_helper.rb:

   require File.expand_path(File.dirname(__FILE__) +
       "/test/test_validation_helper.rb")

After those three simple steps, your HTML and RSS documents are now being validated each time you run tests.

I am giving a brief presentation on this topic at the April 10, 2007 Boston Ruby Group meeting.

You can flip through a PDF of my slides if you would like more details.

Update: In addition to the PDF, my presentation is now hosted on slideshare and shown below. If you are reading this post in an aggregrator, you will need to click through for the show (or just use the PDF version).

Aron and I will be presenting at the upcoming meeting of the Boston Ruby Group on on Tuesday, April 10th, 7:00 PM at One Broadway, Cambridge, MA.

Aron will talk about the regression testing infrastructure we use to validate all of our markup.  I’ll briefly go over another component of our Rails testing environment that we call Isolated Fixtures.

More details can be found here.  Come on by and say hello (not to mention last time there was pizza!)

Matthew claims that I am a superstar, but it’s really less glamorous.

Part of the BarCamp Boston 2 conference was a programming contest.

During the whole first day, people wrote words on a blackboard. At the end of the day, some 11 words were chosen. From these words participants needed to build software which used at least four.

My entry, Pixels to Penguins was one of about seven different entries from different teams of different sizes. All the presentations were very well done, and all quite light-hearted; a great end to the weekend.

Pixels to Penguins took user input, performed a flickr search for related tags, then performed another flicker API call to find images which matched those related tags. The images were mapped onto an ASCII-art rendering of Tux, the Linux penguin; JavaScript made these images appear gradually. To accompany this beautiful scene, all of the data was converted into an integer stream and a 4-track MIDI file was generated. What beautiful music!

The judges named Pixels to Penguins the winning contest entry. The top two teams (three of us in total) won a helicopter ride with Phil Greenspun. I’ve never been in a helicopter!

The (Ruby on Rails) code for Pixels to Penguins is here. I’ve found that it works best under the FireFox browser. My presentation is here.

Update: I’ve had reports that this hack hangs some browsers, so buyer beware. I have been using it successfully on Firefox/MacOSX, but Safari hangs. Firefox/Windows seems to also have problems. If you figure it out, please add a comment to this post!

Update2 I think the browser hangs have been resolved. I was toggling visibility by altering opacity from 0 to 1. A number of browsers were not happy with so much opacity changing so quickly. Now, I’m simply changing the visibility CSS attribute. Please let me know if you still experience troubles with my Pixels to Penguins hack. (Really, it’s nothing more than a cute hack.)

I’d like to describe and provide code for what I’m calling “isolated fixtures”. Isolated fixtures allow you to have an independent and separate set of fixtures for Test::Unit::TestCase in your Ruby on Rails project.

I’d already run into the situation a couple of times where adding info for my new test into existing fixtures broke existing tests, either due to assumptions of about the fixture data, or because multiple tests “borrowed” fixtures from previous test.. According to some other pieces I’ve read on the net, the situation would only get worse as our test suite grew (and ours does grow). So, instead of becoming adept at the sort of limbo-dancing required to finesse new tests into the testsuite, I decided to come up with a way to let each set of tests have their own fixtures.

Now, I don’t know if this is considered a hack, or another demonstration of the flexibility and power that is Ruby on Rails. I’m sure you’ll all let me know.

This works for me with Rails 1.1.6. I have no idea if it will work in 2.0. (2.0 might even have this built in for all I know!)

Usage

Let’s assume you have a test class called AccountControllerTest. In this test where you normally use fixture instead user isolated_fixture instead:

  isolated_fixture :customers, :accounts

Instead of putting the fixture data into test/fixtures it will create a subdirectory there based on the name of the test class. In this case the path of the directory and the fixture data will be:

  test/fixtures/account_controller/accounts.yml
  test/fixtures/account_controller/customers.yml

Code

Here is the code. Put it in your test/test_helper.rb, or do what I did and put it in test/test_helper/fixtures.rb and require it into test_helper.rb.

class Test::Unit::TestCase

  ## Each subclass needs its own copy 'fixture_path' so replace
  ## the existing single class variable.
  remove_class_variable :@@fixture_path
  class << self
    remove_method :fixture_path
    remove_method :fixture_path=
  end

  class_inheritable_accessor :fixture_path
  self.fixture_path = RAILS_ROOT + "/test/fixtures/"

  ## Add specialized fixture class methods and alias them.
  def self.standard_fixtures(*table_names)
    self.fixture_path = RAILS_ROOT + "/test/fixtures/"
    self.fixtures_without_fixture_path(table_names)
  end

  def self.isolated_fixtures(*table_names)
    fixture_subdir = self.to_s.ends_with?("Test") ? self.to_s[0..-5].underscore : self.to_s.underscore
    self.fixture_path = RAILS_ROOT + "/test/fixtures/#{fixture_subdir}/"
    self.fixtures_without_fixture_path(table_names)
  end

  class << self
    alias_method :fixtures_without_fixture_path, :fixtures
    alias_method :fixtures, :standard_fixtures
    alias_method :original_method_added, :method_added
  end

  def setup_with_fixtures_and_with_fixture_path
    setup_with_fixtures_and_without_fixture_path
  end

  ## Provide a default 'setup' method.
  alias_method :setup_with_fixtures_and_without_fixture_path, :setup_with_fixtures
  alias_method :setup, :setup_with_fixtures_and_with_fixture_path

  ## Handle the case where the user provides a 'setup' method for the tests.
  def self.method_added(method)
    case method.to_s
    when 'setup'
      unless method_defined?(:setup_without_fixtures_and_without_fixture_path)
        alias_method :setup_without_fixtures_and_without_fixture_path, :setup
        define_method(:setup) do
          setup_with_fixtures_and_with_fixture_path
          setup_without_fixtures_and_without_fixture_path
        end
      end
    else
      self.original_method_added(method)
    end
  end

end

Drop a line and let me know if you found this useful, or if you found a bug. I'll post any updates.

Update

Here is a slide presentation covering the same material.