Rake and Ant Together: A Pick It n’ Stick It Approach »

Created at: 19.02.2010 21:00, source: Engine Yard Blog, tagged: Technology ant java jruby rake ruby

Recently, I landed a new library for JRuby that will be part of JRuby 1.5. Before I start I want to conjure the image you see below this text: that’s Right!  Mr. Potato Head: a role model for us all. Something that delights us for hours (or at least, probably did, at one point in our lives), is flexible, and is not only a toy, but also a starchy food product.

Mister Potato Head
(Courtesy of Flickr user MyMollyPop)

Bear with me for a second, and excuse what must have just sounded like a wee bit of lunacy. The the truth is, we live in a world where, as programmers, we construct Mr. Potato Heads every day. We’re confronted with making software where not only are we required to stick various elements together, but also to arrange them in the most pleasing way possible.  Software design is really just like decorating a potato. The potato of this blog entry will be build software.

Build Tools

In the Java world, Ant is the 800 pound potato of build tools.  It’s in virtually every Java build environment on the planet.  To date, I have only really known one person who really really liked Ant.

For the most part, I think people respect Ant as a tool which is a bit syntactically gross, but which gets the job done reliably.  It is also lamented for having little support for imperative programming constructs. This seems to have been by design, but it doesn’t seem to make very many programmers happy.

In the Ruby world we have Rake. Rake, by contrast, has a much nicer syntax than Ant. It also allows any construct that the Ruby language allows, since Rake is just a DSL-like API for building software running in a Ruby interpreter. On the other hand, if you have the need to perform common tasks in the Java world, then it lacks a bunch of standardized cross-platform tasks that Ant contains. You find yourself breaking out into lots of icky shell commands (`javac -classpath #{my_files}`) and this works great until you try and build something on, gasp, Windows.

A pragmatic (but not so great) reality is that most Java shops may warm to switching to a different build technology, but are unlikely to switch over projects en masse. Even if they were enough in love with Rake to switch, they’d need to hobble together replacements for tasks that they take for granted in Ant. That is… until now.

Use Cases

JRuby’s Rake and Ant integration handles the following use cases:

• Call any ant task or type from within Rake
• Allow Rake to be invoked from within Ant
• Allow Rake tasks to be imported as callable Ant targets
• Allow Ant to be invoked from Rake
• Allow Ant targets to be called as Rake tasks from within Rake

Let’s break these down one at a time…

Call Any Ant Task or Type From Within Rake

The truth is, Ant is really just a built-in library in JRuby. You could just write a script and not use it in Rake:

require 'ant'

ant do
  build_dir = "java_build" # Regular Ruby variables interact fine

  # But defining and consuming Ant properties is fine
  property :name => "src.dir", :value => "java_src"

  path(:id => "project.class.path") do
    pathelement :location => "classes"
  end

  mkdir :dir => build_dir

  javac(:destdir => build_dir) do
    classpath :refid => "project.class.path"
    src { pathelement :location => "${src.dir}" }
  end

  jar :destfile => "simple_compile.jar", :basedir => build_dir
end

This example constructs an instance of an Ant project, then makes a directory, compiles some Java source, and finally, generates a jar file of the results. All of these are just Ant tasks. They’ll work on any platform. Sweet, I say! This does, however, lack dependency management.  So let’s use Rake to do it instead:

require 'ant'

build_dir = "java_build"
file build_dir

task :setup => build_dir do
  ant.property :name => "src.dir", :value => "java_src"
  ant.path(:id => "project.class.path") do
    pathelement :location => "classes"
  end
end

task :compile => :setup do
  ant.javac(:destdir => build_dir) do
    classpath :refid => "project.class.path"
    src { pathelement :location => "${src.dir}" }
  end
end

task :jar => :compile do
  ant.jar :destfile => "simple_compile.jar", :basedir => build_dir
end

task :default => :jar

[Quick note: since we can combine the best of both worlds, you don't need to ever set an Ant property if you don't want to. Just use a Ruby variable or constant. Different folks for different strokes...]

This snippet shows how easy it is to consume Ant tasks in a Rakefile. Really, JRuby’s Ant library is just a straight-forward set of APIs that map clearly to Ant’s original syntax. Looking up how to do something is a very straight-forward task.

The other benefit mentioned above is that Rake can use imperative programming in it. Let’s consider a snippet like this in Ant:

      <java classname="${mainclass}">
        <arg value="--command"/>
        <arg value="maybe_install_gems"/>
        <arg value="--no-ri"/>
        <arg value="--no-rdoc"/>
        %lt;arg value="--env-shebang"/>
      </java>

It can use imperative conveniences:

command = "--command may_install_gems --no-ri --no-rdoc --env-shebang"
ant.java :classname => "${mainclass}" do
  command.split(/\s+/).each { |value| arg :value => value }
end

So if you’re a Rake user already and you need to do Java things, then using this support should be a pretty simple decision. Heck, there are many other optional Ant tasks that may be useful even if you’re not doing Java things.

Allow Rake to be Invoked From Within Ant

If you have the ability to write the equivalent of an Ant project using Rake, then you may want to make part of your project driven by Rake. However, if you do this, you may still need to call it from within Ant. We can do this with the new ‘Rake’ task.

If we pretend the previous Rakefile existed parallel to a build.xml file we have then in that Ant’s build.xml file we could have:

<?xml version="1.0" encoding="UTF-8"?>
<project name="foobar" default="default" basedir=".">
  <description>Builds, tests, and runs the project foobar.</description>

  <target name="load-rake-task">
    <taskdef name="rake" classname="org.jruby.ant.Rake"/>
  </target>

  <target name="default" depends="load-rake-task">
    <rake task="jar"/>
  </target>

  ...
</project>

This Ant script’s ‘default’ target will load our Rake task and then call Rake (file defaults to ‘Rakefile’), and more specifically, call the task ‘jar’ (task defaults to ‘default’). There are a couple of cool scenarios to consider here:

1. Try Rake by only dipping your toes in the water

This strategy is great if you like Rake but are worried you don’t have enough influence to get your development team to switch their entire build suite. You can just stick a Rakefile off to the side for some new functionality and let your teammates evaluate how much they like it. If they do, then switch the rest later… or don’t. The idea that most Java shops will big-bang change their build software is extremely unlikely. An incremental strategy is your best bet.

2. Easier integration with Java tools

Even if you’re totally sold on Rake, you still need to know that software like NetBeans expects to see a build.xml file, so it can interact with your project. Having a small shim like the one above makes you play nice with any tools that expect Ant.

This is the simplest way to call Rake from Ant, but the next option may suit your needs better…

Allow Rake Tasks to be Imported as Callable Ant Targets

The big missing feature of the Rake task in the script above is that it’s only one-way. You can call Rake, but then the Rakefile you call has no meaningful interaction with the Ant side of things. Sure it can call Ant tasks, but it can’t see properties or Ant targets that are defined in the calling build.xml file.

To have better interoperability we have another Ant task: RakeImport. RakeImport will require the specified Rakefile and then register all of its tasks with Ant dependency management system. Let’s look at a simple example:

<?xml version="1.0" encoding="UTF-8"?>

<project name="foobar" default="top-level" basedir=".">
    <description>Builds, tests, and runs the project foobar.</description>

    <taskdef name="rakeimport" classname="org.jruby.ant.RakeImport"/>
    <rakeimport/>

    <target name="top-level" depends="its_in_rake" />

    <target name="its_in_ant">
      <echo message="ant: its_in_ant"/>
    </target>
</project>

In Ant, we specify that we want to use RakeImport and then immediately call it. This loads the following Rakefile and registers all of its tasks with Ant:

task :its_in_rake => [:setup, :its_in_ant]  do
  puts "it's in Rake"
end

task :setup do
  puts "setup in Rake"
end

Now let’s run ‘ant top-level’; we now see the following output:

Buildfile: build.xml
[rakeimport] (in /Users/enebo/work/akakamiari/samples/rake_import_example2)

setup:
setup in Rake

its_in_ant:
     [echo] ant: its_in_ant

its_in_rake:
it's in Rake

top-level:

BUILD SUCCESSFUL
Total time: 7 seconds

This output shows that it’s executing both Ant targets and Rake tasks in the desired order. its_in_ant executes as a dependency of its_in_rake which in turn executes because it is a dependency of the Ant target ‘top-level’.

The scenarios for this level of integration:

1. Choosing the best tool

Since Rake gives a full imperative programming environment there are some things that are trivial to do in Rake which are cumbersome (or impossible without writing a custom Ant task) in Ant. You can move that stuff into Rake, but still continue using Ant for everything else.

2. Toe wet… time to go waist deep

In the previous section we used the Rake task to demonstrate to your development group that Rake is useful. This allows you to start depending on the capabilities of Rake more by being able to inject Rake tasks into the Ant dependency graph. Your group is still using Ant as the main build tool, but you’re delegating more of the build to Rake.

Allow Ant to be Invoked From Rake

Let’s look at things from the other side of the coin. If you’re already a Rake user, but you need to interact with existing Ant build files, we’ve also got solutions for you. The first method allows you to just call Ant from within Rake:

task :call_ant do
  ant '-f my_build.xml my_target1'
end

Alternatively, you can also supply arguments as a list:

args = ['-f', 'my_build.xml', 'my_target1']
task :call_ant do
  ant args
end

Believe it or not, this doesn’t just execute `ant`; it loads it into your JRuby environment. This is nice because it doesn’t spawn a second JVM to run Ant.

As I mentioned at the beginning of this section, you may want to just call into Ant, but not have any more interaction than that. That’s ok, but if you want more….

Allow Ant Targets to be Called as Rake Tasks From Within Rake

‘ant_import’ beats ‘ant’ in flexibility because it ends up registering all of Ant’s top-level targets with Rake’s dependency management system. Once you import an Ant file into Rake, you can call the Ant tasks as if they were ordinary Rake tasks. A simple example:

task :ant_import do
  ant_import
end

task :compile => [:ant_import, :its_in_ant_setup] do
  # Do some compilation
end

This example will load ant_import when the compile task is executed which will in turn load the build.xml file in the current directory and load all of its Ant targets. I did not ant_import at the top-level of the Rakefile to show that you may only want to load the Ant targets if you are actually planning on using them.

Details Details

This code has just landed in JRuby trunk. All of the examples listed here should work, but this code is brand new. It’s a work in progress that will firm up by the time JRuby 1.5 is out. So this means two things for the reader:

1. You can help find problems and help improve the library

In other words, if this interests you, then you can get involved early and help address problems or enhance the library. Odds are you can make the difference between this integration being great or merely good.

Getting started note: the current practice for running the examples above is to copy jruby-complete.jar into $ANT_HOME/lib. You can manually set your classpath to include jruby-complete.jar as well, but I found it less error prone to just copy the jar.

2. If you don’t like unfinished software then wait for 1.5

Some people don’t like the hassles or have the time to play with unfinished software. If you’re in this position, then don’t worry. We’ll get things working nicely for 1.5. We don’t want an early test run to ruin anyone’s expectations.

Conclusion

Going back to our original Mr. Potato Head theme, I was hoping to show that sometimes combining things together really is the best strategy. In theory, it’s nice to start over and make something pure and wonderfully homogenous. In practice, there are always pesky real-life details that get in the way. Why rewrite everything when you can just attach the new technology on the side? If you do have the long term goal of replacing an older technology, why not do it incrementally? Every big-bang rewrite project I have ever been in has basically failed in a significant way. Change things one piece at a time and your chances of success are much higher.

Our new Rake/Ant integration is meant to allow incremental change, or at least to allow the best features of each tool to be used together. It embodies the reality that both tools may be neccesary and that replacing one for the other probably won’t be a 100% solution.

As always, questions and comments are welcome!

A Hint of Hubris »

Created at: 16.02.2010 20:00, source: Engine Yard Blog, tagged: Technology Haskell Polyglot Programming ruby

Ruby is a highly dynamic language with impressive capabilities for runtime redefinition of classes, objects, methods, and variables. Haskell, on the other hand, is a purely functional language that confines mutation within a sophisticated static type system. Given their many differences one or the other may be more suited for whatever problem you might be working on (see polyglot programming), but sometimes, a mix of both would be even better. Thanks to Hubris, you now have that option!

Hubris is a gem that allows you to call out to natively compiled Haskell shared objects. In this post, I’ll talk a bit more about why you might want to do that, how you’d go about doing that, and share some real world examples.

So Why Would I Want To Use Haskell?

It’s easy for any decent programmer to spot the many striking differences between Ruby and Haskell, but why a Rubyist might be interested in  Haskell is a question worth asking. There are two primary scenarios:

1. Native Extensions

When CPU-intensive algorithms are needed for a task in Ruby, you usually drop down to the C or C++ level to take advantage of the highly optimized code yielded by a C/C++ compiler. Unfortunately, for someone without significant experience in the language, it can be easy to run into memory management woes, not to mention the frustration of dealing with a more hands-on (with respect to memory and explicit typing) language like C.

Haskell is a managed memory language with one of the most powerful type inference capabilities around. What this means is that you can avoid being bothered with explicitly denoting types at function call-sites, and choose whether or not to add types to definitions for the particular function you’re working on. Additionally, GHC (Haskell’s most prominent compiler for production applications) produces compiled native code that has performance in the same league as C/C++.

2. Concurrency

Many popular languages (especially Object Oriented ones) allow unconstrained mutation of data (e.g. mutator! methods in Ruby). This has always had implications for the difficulty of reasoning in the resulting code. But lately, this has become a bigger problem because it makes reasoning about concurrent program execution a nightmare.

What differentiates Haskell from many other functional languages is that Haskell isolates side effects statically in types rather than dynamically in language constructs. In effect, this means you can statically determine a list of functions/places to look (and eliminate many of the places you don’t have to look) when problems related to side effects do come up.

3. And More

Haskell  as a whole has many other useful properties, such as implicit laziness, implicit argument currying, a typeclass system for generic programming, various concurrency constructs including an STM and a wonderful concurrency debugger… just to name a few. Rubyists who enjoyed Why’s Poignant Guide to Ruby may enjoy parsing through Learn You a Haskell for Great Good.

Okay, I Buy It: Now How Can I Do It?

Now that we’ve seen why you might want to use Haskell, we’ll take a look at how to use Hubris from Ruby. There are two main modes of operation: a way to embed a Ruby string containing Haskell code inline, and a way to load code from a plain Haskell .hs file. Like so:

class Example
  hubris :inline => "foo :: Integer -> Integer; foo x= 1330 + x"
end

Example.new.foo # => 1337

class Example
  hubris :source => "MyHaskell.hs"
end

Example.new.my_awesome_function "go ye forth into a world unknown"

Note that due to support for the :inline option, you can even experiment with adding new functions dynamically in an irb session (compile errors like type conflicts get spit out for you to read)! In the background, Hubris will compile code to /var/hubris/cache and then link it in to your running Ruby program.

The first time you run the code, it will be a bit slower as it compiles the Haskell, but subsequent runs will load the cached .so files directly. Another nice property is that the already compiled code is compared against a saved hash in order to only recompile when changes have been made.

As the Hubris README demonstrates, a third option is load modules already installed on the system:

hubris :module => "Data.ByteString", :packages => ["bytestring"]

Hubris comes with an example Rack app that exposes a Haskell implementation of the Fibonacci function. I’ll show it to you here as a compact Sinatra app. A likely use case for Rubyists will be to leverage Ruby’s mature web development library ecosystem by exposing an API for a compact and optimized Haskell algorithm/program .

require 'sinatra'
require 'hubris'

helpers do
  hubris :inline => <<-HASKELL
fibonacci :: Int -> Int
fibonacci n = fibs !! n
  where fibs = 0 : 1 : zipWith (+) fibs (tail fibs)
HASKELL
end

get '/:number' do
  fibonacci(params[:number].to_i).inspect
end

As you can see, sensible conversions between data types in both languages are done automatically. Below is a table of the mappings between Ruby and Haskell types (see the Hubris specs for example uses). Try  searching Hoogle (no, that’s not a typo) to find Haskell functions supporting types you’re interested in.

Hubris currently lacks support for calling functions with multiple arguments (or tuples or custom algebraic types)—which is a serious drawback. However, this is not by design and you can track the Github issues for those missing features via the links in the table above.

In the future, fine-grained interaction between both languages will be possible. For example, unit testing Haskell functions with RSpec, including those with multiple arguments. For the time being, you can take a more coarse-grained approach by passing a hash of many values to a single interface function in Haskell.

A Real-World Example

I’ll end my overview by linking to a more complicated example that exposes a Ruby webservice to a coarsely-grained Haskell function. Here, a Sinatra web service is passed a JSON string, which is parsed into a Ruby Hash (actually using C via the JSON gem), then passed to a Haskell function, which is converted on the Haskell side to a Map and then returns an [[Int]] List that gets converted into a nested Ruby Array of integers to finally return the web response as a String. (That made for a long and hard to read sentence, but go through it slowly: it makes sense.)

Rather than calling out to a simple Fibonacci function, you call a basic implementation of a Multi-layer Perceptron Neural Network. HUnit is used for unit testing in Haskell. These tests correspond to values in a handwritten derivation of the first epoch of learning the XOR problem. A Ruby integration test drives the whole stack via rack-test on the XOR problem. A rake task runs both sets of tests to conveniently validate expected behavior. Explaining the algorithm would be enough content for at least a separate post, but hopefully Neurosis can serve as a useful Hubris reference implementation to fiddle around with in the meantime.

Conclusion

There are some serious limitations that prevent fine-grained interaction between Ruby and Haskell. Nevertheless, Hubris has strong potential as a useful library and can be used today via coarse-grained interfaces. Note that Hubris has been tested to work on MRI 1.8 and 1.9, but should work with any Ruby that conforms to MRI’s C API.

Installation of the Haskell-side of the library is a bit of a pain right now, but this wiki page is recording the team’s progress as they work on fixing it.

So here we are: it’s time to go out and write your own Haskell code, to be called from Ruby (or find a favorite Haskell library to reuse)… that is, if you can make it past installation ;)

As always, leave any questions and comments!

Let Them Code Cake! »

Created at: 09.02.2010 20:00, source: Engine Yard Blog, tagged: Technology activerecord cake datamapper refactoring

In TDD and BDD we write small, focused, technical tests, sometimes called micro-tests. One of the core ideas is that these tests should run fast, really fast—each one measured in milliseconds. If you’re writing plain Ruby code, that’s pretty easy to accomplish. However, when you’re using something like Rails or Merb and DataMapper or ActiveRecord, it can get a bit more challenging.

Why do we end up with model tests/specs that run slowly? In looking at how we write and test our code, let’s ignore controllers and views, and focus on models. After all, models are where you put your business logic.

There can be several reasons for a slow test suite: database access, unnecessary objects, a massive setup that takes a while to load—just to name a few. This time though, I want us to focus on a new reason, one that’s a result of operating in the context of ActiveRecord or DataMapper. One that tricks us into thinking we’re doing well. One that exposes a few somewhat major flaws in the design of AR and DM.

With ActiveRecord and DataMapper, each model class is responsible for its own persistence, hence DHH using the name “Active Record.” It was inspired by Fowler’s writing on the pattern in Patterns of Enterprise Application Architecture and is the enabler of our model addiction.

OverActive Record

Used properly, the active record pattern is great for persisting data. These objects are great at handling their own persistence. Accessors, associations, and validation make a good wrapper around the transactional nature of a database. This is fine when the model is a simple data object, but we all run into problems when we start adding other behavior to the model class.

Weighted Companion Cake: The cube was the cake!

(Compliments to flickr user Mandrake88)

Recall the Single Responsibility Principle: “There should never be more than one reason for a class to change.” As we commonly use them, ActiveRecord and DataMapper¹ classes almost always violate this principle. There are usually at least two responsibilities handled by every AR/DM model: persistence and business logic. Carrying around that persistence behavior, and all the dependencies that go along with it, is what bloats and slows down our specs.

There have been various attempts at dealing with this: in memory databases, stubbing parts of the DM/AR frameworks, etc. None of these are ideal. Either they don’t speed things up as much as we’d like, they’re awkward, or they bulk up the specs an objectionable amount.

Is There a Better Way? Why, Yes! Yes There Is.

DM/AR lets you define a model’s properties, its persistent parts, and generates the associated accessors and mutators. All access to the persistent properties is handled through these accessors and mutators. This gives us a perfect seam along which to split the class. All of the persistence-dependent functionality is on one side of this seam, and the business logic of the model is on the other.

We need a good saving throw against this frosting.

Courtesy of Mike’s Amazing Cakes

Pretend… It’s a Cake.

Yes, you read that correctly. Keep on reading—it makes sense!

Each of our models is like a frosted cake. We have the nice fluffy cake batter, baked with our properties, validations, and persistence. Then we frost the outside with our business logic. We can take a simple ActiveRecord cake and make it into almost anything, with a good amount of frosting.

Now herein lies the problem: the frosting starts to take on a life of its own. Before we know it we’re baking a few 100 ActiveRecord cakes each time we need to test the frosting behavior… There are two really good ways to get around this: cupcakes and cardboard.

Strategy 1: Oh Look! Cupcakes!

When we build a web application with ActiveRecord or DataMapper, it’s really hard to think outside the persistence box. If something is persisted on that model, or if it uses something on that model, we automatically add the code to the model. This leads quickly to a gigantic cake that does everything.

This is especially bad with primary models like User. The poor bloated User gets tons of code stuffed into it because our app is usually focused around users. Users do things in our app. Users get affected by things. They control the app and most things belong to the user in some way. But this doesn’t mean our User class should be 1000 lines.

This cake does too much (yes that’s cake)

The solution is to break out chunks of code into their own classes. There are many common patterns that we can find in our classes: Factory, Provider, Policy, Strategy, and plenty of types that are custom to each domain (Cake Decorator?). Let’s lump these all into something that we’ll call Cupcake Classes.

At Engine Yard, one of the places we found an opportunity for a Cupcake was our Deployment initialize. The method had to do a lot of data mutating before it could create an instance. Blank strings needed to become nil, string representations of a type needed to be converted to the actual type, and defaults needed to be applied to fill out the incoming data. The initialize method was a ridiculous 50 lines long. This needed to become its own class.

The result is a really clean initialize:

  def initialize(params={})
    merger = Deployment::DefaultsMerger.new(params)
    merger.each_attribute { |att, val| send("#{att}=", val) }
  end

…and a class that knows all about how to merge incoming params and defaults. It has a clear API, and it’s easily testable. We know what comes in and what comes out and we don’t have to create a database record every time we test it.

class Deployment
  class DefaultsMerger

    def initialize(params)
      @attributes = merge_with_defaults(params)
      scrub_blank 0, :data_volume_size, :app_server_count
      scrub_blank nil, :db_volume_id, :data_volume_id
      coerce_value(:instance_size, :db_instance_size) do |val|
        Instance::Size.coerce(val)
      end
      # ...
    end

    def each_attribute
      @attributes.each do |k,v|
        yield k, safe_to_i(v)
      end
    end

  private

    def defaults
      Mash.new({
        :data_volume_size => Volume::DEFAULT_SIZE,
        :app_server_count => 1,
        # ...
      })
    end
    # ...
  end
end

Cupcake Coding Rule of Thumb: Messy Code Makes for Good Cupcakes!

When you want to clean up your models, look for messy code. Long methods or sets of utility methods (methods that barely, if at all, touch instance state) are just begging to be cut up into smaller classes.

What tipped us off was that the initialize method was long, nasty, and really had nothing to do with the class it was in. Mostly, it dealt with params, a hash. Almost every line in the old method did hash operations, not operations dealing with the actual model.

We clearly needed a class that would know how to work with merging our defaults into an incoming hash. The resulting class would also improve our real test coverage. It’s much easier to test a bunch of branches when they don’t have to pass through the model life cycle.

describe Deployment::DefaultsMerger do
  describe_attribute :data_volume_size do
    it_defaults_to Volume::DEFAULT_SIZE
    it_converts ["", nil, 0, "0"], :to => 0
    it_converts [123, "123"],      :to => 123
  end

  describe_attribute :app_server_count do
    it_defaults_to 1
    it_converts ["", nil, 0, "0"], :to => 0
    it_converts [2, "2"],          :to => 2
  end
  # ...
end

With a focused class, it’s easy to write focused specs. With a few custom methods in your specs, you have beautifully concise descriptions of behavior to match your simple code.

Looks yummy!

Thanks to flickr user hazellterry

Strategy 2: Cardboard Cake

The next way to improve your models and your testing is to extract the model’s behavior from the persistence. The idea is to test the frosting without having to bake a cake every time. Our example class after the behavior has been extracted:

class Oven
  include DataMapper::Resource
  include Oven::Behavior

  ## Properties

  property :id,           Serial
  property :oven_type,    String
  property :temp_setting, String

  has 1, :heating_element
  has 1, :latch
  ## ...

end

Here’s the frosting, or behavior, that has been extracted from it:

class Oven
  module Behavior

    def check_lock
      latch.engage! if dangerous_temperature?
    end

    def dangerous_temperature?
      Oven::Temperatures.dangerous? temp_setting
    end

    def turn_on(temp = 350) # ...
    def turn_off # ...
    def preheating? # ...
  end
end

Once this is done, you can write specs that focus on the logic in isolation. Instead of testing every aspect of behavior on top of the persistence framework (slow), you can isolate the behavior (fast). Sort of like making the cake out of cardboard so you can test our frosting without the baking time (See? Told you it would make sense!).

How Do We Achieve This Isolation? Mock Object Trickery, But of Course!

Let’s create a mock to stand in for the model object being tested. Let’s call these Cake Mocks. Then, we’ll extend the mock object with with the Behavior we’re testing.

  @oven = mock("Oven").extend(Oven::Behavior)

It still acts like an oven, but instead we’ve frosted the outside of a cardboard cake with the same business logic that we would add to a real ActiveRecord object. The trick is to have the mock object extend the module that contains the business logic. The result is an object containing the behavior of the model but not its persistence.

Now we can stub accessors on the mock (but only those involved in the example) and set expectations on the mutators. Then the example can call methods on the mock oven as if it was a real Oven instance. Those calls invoke the actual business logic methods that we want to test. When those methods access the internal behavior of our persistence layer, our mock answers with the methods we stubbed. The mock merely fills in the center of the cake.

describe Oven::Behavior do
  describe "preheating" do
    before(:each) do
      @latch = mock("latch")
      @oven = mock("Oven", :latch => @latch).extend(Oven::Behavior)
    end

    it "isn't locked when temperature setting is less than 500" do
      @oven.stub!(:temp_setting).and_return(350)
      @latch.should_receive(:engage!).never
      @oven.check_lock
    end

    it "isn't locked when temperature setting is 500" do
      @oven.stub!(:temp_setting).and_return(500)
      @latch.should_receive(:engage!).never
      @oven.check_lock
    end

    it "is locked when temperature setting is greater than 500" do
      @oven.stub!(:temp_setting).and_return(505)
      @latch.should_receive(:engage!).once
      @oven.check_lock
    end
  end
end

This cake mock style encourages another good testing practice: integration testing. When the model is completely mixed up with behavior, the need for integration testing is easier to ignore, but no less important. With separated behavior and persistence we can write simple integration tests that make sure the seam between the cake and the frosting is still intact.

Give Your Models Some Attention

When looking at models this way, whether you’re breaking them up in to cupcakes or separating the frosting, it’s easier to see the difference between behavior and persistence. If the logic can stand on its own, make a new class. If it’s more like an extension of the model, make a module and mix it back in. Allow ActiveRecord and DataMapper models to take care of what they do best without all the extra weight.

Take a few minutes and look at the models in your project. If the classes weren’t backed by the database, would you still have all that stuff jammed into one place? Are your models more complex than they need to be? If so… do something about it!

Rails 3 Beta is Out — A Retrospective »

Created at: 04.02.2010 22:00, source: Engine Yard Blog, tagged: Technology Rails 3

The Rails team has finally released the Rails 3 beta, after more than a year since the Rails and Merb teams started working on this release. You can read all about it at the official Rails blog, but I figured I’d take the opportunity to share my take on the release.

First of all, you’re probably sick of hearing this, but we’ve done far, far more than we ever expected. A lot of that happened in the last few weeks.

One of the things that most surprised and impressed me is the Rails core team’s (and especially DHH’s) attention to detail and the experience of the beta release. For weeks, we’ve been “this close” to releasing, but the experience of starting up a new Rails app or upgrading from a Rails 2.3 app still felt too unpleasant. In this kind of situation, it’s tempting to say “it’s just a beta—people who use beta software know what they’re getting,” but that would have been a major cop-out. For many people on the leading edge, a poor beta experience will shape their perception of the product as a whole. So we waited a bit, but now we’re finally here.

In addition to a surprising (in a good way) amount of polish, we touched nearly every part of Rails: I’ll walk you through the top-level blocks.

Dependencies

Rails gets a bunch of new dependencies, stepping out of the “virtually zero dependencies” box it has lived in since the beginning. In order to achieve this, we wrote a brand new dependency management system (called Bundler) from scratch, remedying many of the problems with Rubygems that have made using a number of dependencies difficult (and which we encountered in Merb when taking a more-dependencies approach). Taking another step into the future, we built Bundler itself as a standalone component, so you can use it for Rails, Merb, Sinatra, or even a plain old Ruby script.

And even though the Rails happy path uses Bundler to manage dependencies, that happy path is limited to the generated code only. The Rails library itself does not rely on Rubygems, Bundler, or any package manager. This makes using a different package manager, like Debian or rip, as simple as changing a few lines of the generated code.

Instrumentation

A number of companies and projects have sprung up over the years attempting to provide useful information about the runtime status of a Rails process. In order to facilitate this work going forward, we’ve baked instrumentation into Rails itself. Instead of having to monkey-patch an appropriate method to learn about which templates Rails renders, for instance, simply listen for an event. While Rails has never officially supported the monkey-patching approach, we will commit to supporting instrumentation events in the same way as we support any other public APIs, making them future-proof stable ways to learn about what’s happening inside of Rails.

Better yet, Rails’ logging system uses these very events to emit information into the log!

Every Framework is an Extension

In what is possibly the biggest conceptual shift in Rails 3, we’ve exposed an extension API that allowed us to make ActiveRecord, ActionController, and ActionMailer simple Rails extensions. In other words, the DataMapper ORM can take a look at the part of ActiveRecord that implements its Rails extension, implement similar features, and know for sure that it’ll work. Additionally, users can completely remove the ActiveRecord gem, and be rid of the generators, rake tasks and other peripheral elements. An example: if DataMapper implements a db:test:prepare rake task, a Rails developer can replace ActiveRecord with DataMapper, and the test:units rake task will use DataMapper’s new tasks.

In short, we have completely removed references to other frameworks from Railties itself, making the frameworks of Rails true extensions to Rails itself. This makes it possible to replace any part of Rails without having to worry about entanglements with Railties.

Rack

Rails 2.3 took a very firm stance in favor of Rack. Rails 3 completes the story. In Rails 3, the Application itself, the router, and controller actions themselves are all Rack applications. Rails ships with a standard config.ru, and can boot using the rackup tool that ships with Rack, or with Passenger or Glassfish in standard Rack mode. In fact, any Rails host or server that supports the Rack protocol can run a Rails 3 application unaltered.

Because so many parts of Rails are Rack, you can also insert standard Rack middleware in config.ru, in your Rails application, in the router, or even in a controller’s actions. Rails itself ships with a number of standard middlewares, for exception handling, cookies, parameter parsing, flash notifications, sessions and a bunch more. You can use any of these middlewares in standard Rack applications (without Rails).

ActionController

ActionController got a major overhaul, dividing up a monolithic component into multiple pieces.

ActionDispatch

You can think of ActionDispatch as rack-ext. Essentially, we packaged up all of the parts of ActionController involved in the Rack dispatching process into a single reusable component. This includes ActionDispatch::Request, ActionDispatch::Response, our mime type handling, a bunch of middleware, and integration testing support.

It also includes a wrapper around rack-mount a new powerful routing library. Josh Peek, a member of the Rails Core Team, wrote rack-mount as a standalone component, so that it could be used in standard Rack projects without any Rails dependencies. By default, it simply dispatches inbound requests to Rack endpoints.

Because Rails 3 controller actions are Rack endpoints, Josh could easily wire up a thin shim around rack-mount to dispatch inbound requests to Rails controller actions. But because we built the Rails router on top of rack-mount, Rails applications can route inbound requests to any Rack application, like Sinatra or Cloudkit.

AbstractController

The disparity between ActionController and ActionMailer has long been a personal pet peeve of mine. For Rails 3, we refactored ActionController and ActionMailer to inherit from a common superclass: AbstractController. AbstractController includes the basic concept of controllers, rendering, layouts, helpers, and callbacks, but nothing about HTTP or mail delivery. This means that ActionMailer can no longer drift away from ActionController, leaving us with two distant cousin APIs.

ActionController

ActionController got a big overhaul, completely eliminating alias_method_chain in favor of a module-based system that extension author can more easily extend. For instance, we exposed a Renderer interface, allowing extension authors to easily add render :pdf or render :csv without having to override the render method in brittle ways.

While the API did not change much in general, we did add a few new features, such as the ability to add middleware directly to a controller’s actions or the ability to wrap up common idioms more cleanly using responders. Additionally, we improved the overall architecture to reduce the need for certain hacks.

For instance, ActionController will now look up the layout for an action using the details (such as format and locale) that its template used. In Rails 2.3, an RJS template could end up using an HTML layout, requiring us to exempt RJS templates from layouts altogether. However, this was a band-aid (what if you wanted an RJS layout?) and the improved architecture eliminates the possibility for other, similar problems.

ActionView

ActionView got a significant overhaul, improving its internal architecture, increasing performance, adding pervasive XSS protection, and making all JavaScript helpers unobtrusive.

In terms of architecture, Rails 3 defines a clear boundary between ActionController and ActionView. Now, ActionView exposes a single API entry point for rendering templates and partials, making the code path easier to understand, eliminating a number of latent (but unusual) bugs, and making it easier to reason about caching lookups for performance.

When doing a performance audit of Rails applications, ActiveRecord and ActionView (specifically, template rendering) pop up as major costs. In particular, ActionView has had a fair bit of overhead that’s impossible to avoid without throwing certain features out the window. For instance, partials and especially collections of partials had enough overhead to become noticeable on profiles of real applications.

In Rails 3, we optimized both the template lookup logic (which should now take up extremely small amounts of time) and the logic inside the loop of partial collections. In particular, partial collections should often be much faster in Rails 3 than a manual loop, and not much slower than manually inlining and unrolling the loop. Finally, the improved architecture has made additional optimizations possible, so expect further performance improvements in Rails 3.1.

We also switched to Erubis, a fully compatible ERB implementation with much better extensibility characteristics. In particular, it allows us to easily hook in the code that it generates for any ERB construct. This will allow compile-time performance optimizations in the future and enables the next feature: pervasive XSS protection.

Rails has always shipped with a simple way to escape any user-generated content. Unfortunately, this required constant vigilance. A number of highly visible, successful attacks against websites on a number of platforms taught the world that successful mitigation of this threat requires treating all Strings as suspicious, and allowing developers to mark Strings as safe. Rails 3 escapes all Strings, allowing developers to convert Strings into safe Strings easily. We have also modified all of our internal helpers to mark Strings that we create as safe, while not modifying Strings passed in by the application developer.

Finally, we have overhauled all of the JavaScript helpers to emit plain HTML markup instead of JavaScript. As part of the process, we have created drivers for jQuery and Prototype that bind events to the appropriately marked up elements in the DOM. This means that Rails can now support any JavaScript library as a first-class citizen. We will support and maintain the markup and JavaScript developers can develop pure-JavaScript libraries to make use of the markup (without ever having to touch Ruby). If other frameworks emit the same markup, the same JavaScript libraries will work with their helpers.

ActiveModel

Over the years, people have created a number of ActiveRecord lookalike libraries for things like MongoDB or CouchDB. In Rails 3, we have extracted elements of ActiveRecord unrelated to RDBMS persistence, such as validations, callbacks, dirty tracking, and serialization into a separate library. Other libraries, or even plain old Ruby code, can use these components on an à la carte basis, choosing to look as much or as little like ActiveRecord as they wish.

Additionally, ActiveModel defines an API that any object can implement in order to work seamlessly with ActionPack. It provides drop-in implementations of the API, but Ruby libraries that wish to participate may implement the APIs themselves. This means that ActionPack itself is agnostic as to the ORM used, and even supports hand-rolled models.

ActiveRecord

ActiveRecord has undergone a large internal change, integrating Nick Kallen’s ActiveRelation codebase. This replaces the internal ad-hoc query generation with query generation based on relational algebra. In essence, it means that you can chain queries for refinement before Rails builds the actual SQL query.

For instance, instead of Post.scoped immediately connecting to the database and retrieving all posts, it creates a relation object representing the idea of “all posts”. You can then refine further, such as Post.scoped.where(:author => "Yehuda"). This means that you can pass around relations, refining them as appropriate in helper methods before the view finally kicks off the query when it iterates over the objects.

This also unifies similar concepts in ActiveRecord, like named scopes, into a single relational concept that pervades the entire ORM.

ActiveSupport

In order to facilitate use of ActiveSupport by other libraries, ActiveSupport 3.0 makes it easy for developers to cherry-pick specific parts of the library they wish to use. In particular, we have made all dependencies between ActiveSupport files explicit, so that if you require a particular extension, you can be sure that you get everything that you need. In the past, in order to use ActiveSupport extensions reliably, you needed to require the full library.

ActiveSupport also comes with some new and retooled modules, such as ActiveSupport::Concern (which makes it easier to manage dependencies between modules) and a significantly faster ActiveSupport::Callbacks.

Railties

We made the biggest user-facing changes in Railties. A Rails 3 application is an object (of the class Rails::Application), which also doubles as a valid Rack application. Instead of configuring your application in a block, you configure the Application itself (which now lives in config/application.rb).

The Rails::Application class also inherits from an Engine class, making Engines behave almost identically to Applications, as appropriate. And Engines are a special kind of Railtie, again simplifying and unifying the concepts in the Rails glue code.

The rails command itself now has special powers inside of Rails applications, replacing the commands in script with subcommands of rails. For instance, use rails server instead of script/server.

In Conclusion…

It’s been an incredibly busy and productive year. Rails 3 is the culmination of thousands of man-hours by numerous dedicated open source contributors. It’s been a long journey, filled with debate, feedback, and writing code, refactoring it and  then refactoring it yet again. While this release is just a beta, it represents the bulk of things to come in a stable and tested state.

Plugin authors should take a look at the new exposed APIs and start porting plugins to work on Rails 3. The community is excited, and we expect adoption to be rapid, aggressive and just as importantly, vocal. We’ll be listening to that feedback to help get Rails 3 to its final perfected state, and you should be listening too, so your plugins can make the jump with us.

Leave comments here and on the official Ruby on Rails blog; we’ll be here to respond!

Homebrew: OS X’s Missing Package Manager »

Created at: 02.02.2010 20:00, source: Engine Yard Blog, tagged: Technology Homebrew

Managing software packages on Unix has always been, to put it politely, a giant pain, and most Linux distributions are built around the different ways we’ve all been trying to alleviate that pain. In this post, I’ll walk you through Homebrew, a fantastic new option for package management made simple.

Pre-Homebrew, there were various attempts to create effective package managers for OS X. The two most popular efforts were Fink and MacPorts, but they each had their frustrations. In both cases, creating packages or portfiles was still complex and difficult.

Max Howell’s done a great job with Homebrew; it’s easy to edit, and creating new packages is a breeze. Let’s dig in!

What Does It Do?

The pitch is simple: Homebrew alleviates the drudgery and repetition of downloading and installing Unix software packages on OS X. If you’re sick of ./configure && make && make install, Homebrew can help.

Why Homebrew?

As previously mentioned, OS X already has two package managers: Fink and MacPorts. If one of those is working for you, great. But if you’ve been frustrated by them in the past, I strongly suggest you give Homebrew a try. It’s easy to create and edit formulae, and even to edit Homebrew itself, since the core is just a few hundred lines of Ruby code.

It doesn’t impose external structure on you: the default is to install it to /usr/local, but you can install it anywhere. Inside your Homebrew directory, software is installed in subdirectories inside Homebrew’s cellar, like Cellar/git/1.6.5.4/. After installation, Homebrew symlinks the software into the regular Unix directories. If you want to hand-install a package or version that isn’t officially part of Homebrew yet, it can happily coexist in the same location.

That’s usually not necessary, though, since formulae can install directly from version control. If a package has a public git, svn, cvs, or mercurial repository, you can install the latest development version as often as you’d like with a simple brew install.

Installing packages is faster, too, because Homebrew also works hard to avoid package duplication. No more installing yet another version of Perl as a package dependency when you already have a working install of Perl built into OS X. Best of all, Homebrew has a basic philosophy that you shouldn’t have to use sudo to install or manage software on your computer.

Sounds Pretty Great… How Do I Get It?

The first (and only) dependency that Homebrew has is the OS X Developer Tools, which are on the OS X installer disc, and available from Apple as a free download.

Unless you have a reason not to, the easiest place to install Homebrew is in /usr/local. You can do that in just a few steps on the command line:

# Take ownership of /usr/local so you don't have to sudo
sudo chown -R `whoami` /usr/local
# Fix the permissions on your mysql installation, if you have one
sudo chown -R mysql:mysql /usr/local/mysql
# Download and install Homebrew from github
curl -L http://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C /usr/local

Once you’ve done that, you’re good to go! Assuming /usr/local/bin is in your PATH, feel free to try it out:

brew install wget
brew info git

The Homebrew wiki also has more about integrating with RubyGems, CPAN, and Python’s EasyInstall.

Keeping your copy of Homebrew up to date is easy, too:

brew install git
brew update

Once you have git installed, you can just run brew update any time you want to pull down the latest formulae.

Contributing

Creating a new formula is almost that easy. If Homebrew didn’t have a formula for wget, you could create one like this:

brew create http://ftp.gnu.org/gnu/wget/wget-1.12.tar.bz2

After you save your formula, you can test it out with brew install -vd wget, to enable verbose logging and debug mode. If you need help getting your formula working, there’s more documentation on the Homebrew wiki. You can also learn by example from already existing formula, like git or flac.

You can check out lots of example formulae, as well as the internals of Homebrew, by running brew edit. The code is pretty straightforward. If you have questions, or are interested in future plans, the contributors to Homebrew tend to hang out in the #machomebrew channel on Freenode.

Once you have a working new formula, it’s easy to create your own fork of Homebrew on GitHub to push your new formula to, by using the github gem:

git add .
git commit -m "Added a formula for wget"
gem install json github
github fork
git push <your github username> mastergitx

After pushing your change to GitHub, go to the Homebrew issue tracker and create a ticket with the subject “New formula: “. Assuming everything checks out, your formula will be added to the main Homebrew repository and available for everyone else to use.

Wrapping Up

Homebrew is a compelling alternative to MacPorts and Fink. The Homebrew core and all the formulae are written in Ruby, so it’s easy to add new packages or even new features. If you’re looking for more control over the Unix software you have installed on your Mac, or you’ve been frustrated by other package managers in the past, check it out. I think you’ll be happily surprised.