18 11 / 2012

Friends don’t let friends hit the database (in tests)

Unit tests that hit the database are generally slower than those that don’t by an order of magnitude. Not hitting the database in specs often makes your code better as well. You are forced to stub out collaborators that would hit the database and stubbing things out forces you to think more about the actual interface. Suddenly things like Video.published.for_user(user).most_interesting_first.limit(4) become red flags when called from a controller. They make you do things like stub_chain which is usually just a tool to enable Law of Demeter violations. You’d likely be better served by a more specific scope or, even better, a query object.

Yes, you could use a sqlite in memory database to potentially speed things up but you’re still going to be much slower than a test that simply stubs the database access.

To this end, we’ve decided to prevent access to the database from specs that don’t explictly request or “naturally” need it. A test will fail if it hits the database.

Why not just be diligent about it? A couple reasons. First and foremost, we like to encourage the Pit of Success. The more we can do to make it difficult to do bad things the better. Also, Rails makes it pretty hard to know when you’re hitting the database.

To make sure a test fails when it hits the database, we freedom patch the Mysql2Adapter and add some RSpec configuration. This could likely be ported to minitest or TestUnit or whatever other framework you fancy.


# place in spec/support/prevent_database_access.rb

ActiveRecord::ConnectionAdapters::Mysql2Adapter.class_eval do
  def execute_with_prevent_database_access(sql, name=nil)
    if prevent_database_access?(sql)
      raise "You should only access the database in model and acceptance specs.
If you really need to you can use :db to grant access for that spec.

Offending query: \"#{sql}\""
    end

    execute_without_prevent_database_access(sql, name)
  end
  alias_method_chain :execute, :prevent_database_access

  def prevent_database_access?(sql)
    return false unless ActiveRecord::ConnectionAdapters::Mysql2Adapter.prevent_database_access

    sql =~ /^(SELECT|UPDATE|INSERT|DELETE)/
  end

  class 

A few things to note:

  • Adding this to an existing project will take some work, but it will likely expose plenty of things to talk about as it did for us.
  • We only prevent SELECT, UPDATE, INSERT and DELETE. Things like stub_model need to hit the database to get column information. We don’t want to stop this.
  • Request specs, model specs and any spec with :db => true will be allowed to hit the database. You can customize this in the config.before :each block above.

You also may want to check out nulldb which will let you run tests against a fake database. This can speed things up when you want to test things like observers or after_save hooks.

29 3 / 2012

Creating RVM gemsets from TeamCity

I don’t know if our setup is weird or not, but this took me a while to figure out so I’m posting here for posterity. Just set RUBY_VERSION and GEMSET as project variables and add a build step as below.

rvm %RUBY_VERSION% do rvm gemset create %GEMSET% 

15 9 / 2011

Test Cookie deletion with rspec and Rails (plus AAA mocking)

So you’re working on a complicated authentication scenario that involves deleting a couple cookies on log out… or you want to be sure that your tracking cookie is deleted after a user has made it to a certain page. Either way, there’s no super obvious way to test that a delete actually happens in Rails.

See, cookies aren’t actually “deleted”. They’re simply set to no value and “expired” when you call cookies.delete. Rails’ CookieJar will give you back nil for a cookie that was deleted. The same thing it will give you for a cookie that never existed. 

So how do we test it? Our good friend mocking.

describe MyController do
  let(:cookies) { mock('cookies') }
  
  context "when my page is visited" do
    it "should delete my cookie" do
       controller.stub!(:cookies).and_return(cookies)

       cookies.should_receive(:delete).with(cookie, :domain => '.mydomain.com')

       get "my_action"
    end
  end
end

Not the cleanest thing in the world, but it gets the job done. Of course, it relies on you using delete instead of setting an expired cookie, but you should use delete any way, so that’s fine in my book.

As an aside, I’ve kicked and screamed in the past to get AAA (Arrange-Act-Assert) mocking to .NET and when I got to Ruby-land I couldn’t find it here. Recently though I’ve come across a couple frameworks that do it. The first I discovered was matahari. Unfortunately, it didn’t seem super baked and it was difficult to stub on something that you also mocked. I then came across bourne, which looks to be more mature and piggy-backs on mocha so it should handle the stub/mock hybrid just fine. I haven’t tried it out on a project yet, but plan to. Check’em out!

28 4 / 2011

Speed up your rspec + cucumber run

Did you know, if you run “rake spec cucumber” you load the rails environment three times? Did you also know that loading the rails environment is greatly dependent on the number of gems you have in your Gemfile? Did you also know that that time is nearly 30% more (or more) if you’re running on Ruby 1.9.2 (here too)?

We just learned all of these things at more along this journey from nearly 6 minutes to less than 3 minutes. All without changing any (or many tests).

How did we do it?

  1. First thing we did was calm the garbage collector down and let it run only every 2 seconds in between tests. Note that though the post is about rspec you can do the same for cucumber, just put the start in a Before block and reconsider in a After block.
    Savings: 35s
  2. Next, we sped up user creation by weakening devise’s encryption in our test environment.
    Savings: 30s, Total: 1m 5s
  3. Then we stopped calling ActionMailers during test runs. Even though they were configured not to actually mail, for some reason they stood out while profiling and saved us a decent amount of time.
    Savings: 20s, Total: 1m 25s
  4. After that, we switched our Akephalos cucumber scenarios (all 5 of them) to Selenium. Believe it or not, Selenium on Firefox running Headless.ly is much faster than HtmlUnit was.
    Savings: 30s, Total: 1m 55s
  5. With Selenium we had a few intermittent failures due to jQuery Animations. Turns out disabling jQuery Animations gives us a small speed boost too. Not big enough to mention normally but this project isn’t too animation or @javascript test heavy. This fix is worth fixing the intermittent failures any way. 
    Savings: 5s, Total: 2m
  6. Almost done. The next thing we did was disable forking for Cucumber. This causes it to run under the same Rails environment that Rake loads. To do this, edit your lib/tasks/cucumber.rake and change t.fork to false. Be sure to run rake with RAILS_ENV=test or your tests may run under development and slow waaaay down. Also, you’ll probably want to ensure your cucumber tests run after your rspec tests. This may not work for all, use at your own risk.
    Savings: 30s, Total: 2m 33s
  7. I figured that we could probably do the same thing with rspec. Unfortunately it’s not a simple flag in rspec, but there is a way. Simply throw that rspec.rake in your /lib/tasks and call fork:spec instead of spec in your build. I’d recommend creating a custom teamcity task like ours. This really may not work for all, so again, use at your own risk. 
    Savings: 30s, Total: 3m 

Obviously these things took some digging to figure out and they may not apply to your specific scenario. I’d recommend profiling your build and seeing what’s slow. Also, take a look at your Gemfile. Make sure you’re not loading any gems in test that you don’t actually need, some nasty gems can take a decent amount of time to load. Not as important if you get down to one rails environment load though.

Also, you should check out Gary Bernhardt’s screencast to get you rethinking how you even write the tests for even more speed, especially while coding.

Any of you have any other tips, hacks or tricks for speeding your build up?

21 3 / 2011

TeamCity personal builds from the command line with Git

Update: TeamCity 6.5 will apparently support this out of the box. In the meantime, this applies to 6.0 at least, maybe 5.0 as well. Thanks Alex.

TeamCity personal builds are pretty cool. If you’ve got a longer running test suite that you don’t want to bother your machine with you can fire off a build on your build server and have it tell you when it’s done.

That’s all well and good if you’re using one of the sanctioned IDEs, but what if you’ve shunned the bloat and moved to VIM? Command Line Remote Run Tool to the rescue. Awesome. Until you try to use it with Git.

Here are some steps to get it working:

  1. Install the plugin as described here.
  2. Download the tcc.jar as recommended in the instructions.
  3. Don’t bother “Configuring plugin via UI options” it doesn’t work with Git.
  4. Create a .teamcity-mappings.properties in the root of your git project with the following (be sure to replace the sha1 with any full sha1 in your git history. See the issue for more info):
    .=jetbrains.git://<any-full-sha1-in-your-git-history>||.
  5. Login to TC:
    java -jar tcc.jar login --host <TCServer:port>
  6. Get a list of the files that have changed so tcc can make a patch (change the commits to whatever you want, typically you want the diff between what it is currently running builds off of such as origin/master and your local copy):
    git diff --name-only HEAD origin/master > .tcdiff
  7. Run to start the build (you can find the BuildTypeId by inspecting your build configuration’s url query params in TeamCity:
    java -jar tcc.jar run --host <TCServer:port> -m "<build comment>" -c <buildTypeID> @.tcdiff
  8. Watch the magic.