bgror

Introducing JRubyConf 2010 »

Created at: 25.08.2010 23:02, source: Engine Yard Blog, tagged: Uncategorized

We're excited to join forces with our friends at EdgeCase to co-host the second annual JRubyConf, taking place October 1-3 at Quest Conference Center in Columbus, Ohio. This year, we've expanded the event to include three days of JRuby-filled goodness. JRubyConf will showcase the growing use of Ruby in the enterprise while also highlighting elements of the Java language that Ruby developers can benefit from via JRuby. We've got a fantastic speaker lineup including: Tom Enebo, Chad Fowler, Jeremy Heingardner, Rich Kilmer, Keavy McKinn, Charles Nutter, Joe O’Brien, Nick Sieger, Brian Swan, Glenn Vanderburg, Jim Weirich, and more to be announced soon. If you're curious about what JRuby can do for you, or if you're someone who has been using it for years - join us! We've got something for everyone. JRubyConf will begin with Java and Ruby specific talks before progressing to more advanced sessions that demonstrate the possibilities of using both languages with JRuby – all focused on bringing the Ruby and Java communities together in a collaborative environment to share best development practices. Topics to be covered include: • Introduction to JRuby • How to use Java in Ruby applications • Best practices for introducing Ruby to Java development teams • Effectively managing large agile teams that use Ruby • Large scale testing with Ruby • How to scale Ruby on Rails Our growing sponsor list includes EdgeCase, 8th Light, ELC Technologies, Kinetic Data, O'Reilly, Terremark, and WyeWorks. Registration is now open. Take advantage of an early bird discount for registration before September 1. If you would like information on user group discounts, give us a shout! To register to attend or to participate as a sponsor, visit the JRubyConf event site. Follow @JRubyConf on Twitter to stay on top of announcements. Hope you can join us in Columbus!

more »

A Gentle Introduction to Isolation Levels »

Created at: 21.07.2010 13:16, source: Engine Yard Blog, tagged: Uncategorized

Bob opens a database transaction and selects everything from the books table. Tom comes along and adds a new book, then Bob, in his same transaction, repeats his same query for all the books. Does Bob see the new book that Tom added? The answer is that you get to choose! It's important to understand what your choices are (and what choice your preferred database makes for you) so that you can ensure your code executes in a way that you intend. The SQL standard specifies levels for how "isolated" transactions running at the same time are, all the way from being able to see uncommitted changes (not isolated) to effectively running the transactions in serial (full isolation). Academically there are eight levels of isolation, but for most purposes you only need to worry about the four defined by the standard. MySQL implements all four, PostgreSQL only two. You can specify a global isolation level for your database, but also override it for individual transactions. The easiest to understand are the extreme levels: no isolation and total isolation. The first of these is known as *read uncommitted*, and it allows Bob to read the new book that Tom is adding _even before Tom has committed his changes_. As you can imagine this level is mostly useless, however it can very occasionally be handy in some reporting situations. At the other end of the spectrum is full isolation, known in the spec as **serializable**. Bob will never see the new book that Tom is adding until he starts a new transaction. The database Bob sees is consistent---within the one transaction, the same query will always return the same result. At first glance this level seems like a great option but there's a lot of overhead involved, it drastically reduces the amount of concurrency you can achieve, and for most purposes the serializable level is overkill. There are two isolation levels in between read uncommitted and serializable, they are *read committed* and *read repeatable*, and this is where it gets interesting. Read committed is the default isolation level in PostgreSQL and Oracle, and is one step up from read uncommitted. It is the most "common sense" level: Bob will not see any changes made by Tom until Tom commits them. MySQL defaults to *read repeatable*. In this level, Bob will not see any _updates_ Tom commits, but will see any _inserts_. Say in Bob's first select he sees one book titled "The Odessey". Tom then fixes the spelling mistake to "The Odyssey", and also add Homer's other epic poem "The Iliad". When Bob selects all books again, he will see "The Odessey" (old title, no spelling fix) and "The Iliad" (the inserted book). To summarize, the four levels from least isolated to most isolated are: *read uncommitted*, *read committed*, *repeatable read*, and *serializable*. They define what types of changes made by Tom that Bob will be able to see within a single transaction. h2. In Practice Say the books we are selecting are ordered based on an arbitrary position column (they're on our bookshelf, for instance). Assume read committed isolation level.

Title       | Position
----------------------
The Odyssey | 1
The Iliad   | 2
The Nostoi  | 3

Bob wants to move "The Odyssey" to the bottom position. To do this, he needs to update its position to the bottom of the list (position 4), then subtract 1 from all positions. At the same time, Tom is adding a new book "The Cypria". Working this through: # Bob checks the bottom position, finds it to be 4 # Tom inserts "The Cypria" in the bottom position of 4 # Bob updates the position of "The Odyssey" to 4 # Bob subtracts 1 from all positions, and since he is using *read committed* he will "see" and update the newly inserted book. # Both "The Odyssey" and "The Cypria" have a position of 3

Title       | Position
----------------------
The Iliad   | 1
The Nostoi  | 2
The Odyssey | 3
The Cypria  | 3

If Bob had used the *serializable* level, the list would have remained consistent for his entire transaction, so his update would not have affected "The Cypria" that Tom inserted, and so would not have updated its position from 4 to 3. (In practice the way databases normally handle this is to actually abort one of the transactions with an error.) For those using Rails, you may have recognized the above scenario as a typical @acts_as_list@ scenario, and you'd be correct. In a default configuration, the @acts_as_list@ plugin makes the same mistake outlined above, and will leave you with inconsistent data. The quickest fix is to wrap all list operations in a serializable transaction.

Book.transaction do
  Book.connection.execute("SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE")
  @book = Book.find_by_name("The Odyssey")
  @book.move_to_bottom
end

(It may have occurred to you that some locking or a unique index on position could avoid the exact scenario above, but that breaks @acts_as_list@ and fails to address some other edge cases left as an exercise for the reader. The main point for the purpose of this article is to understand why it breaks under read committed, but works under serializable.) As a general rule, read committed is a sensible default. It's easy to reason about, fast, and forces you to be explicit about your locking strategy. Jump up to serializable when needed, usually when dealing with ranges. MySQL's repeatable read default can be confusing and deadlock in unintuitive ways, as such it is not recommended. This has been a very brief introduction to the four standard SQL isolation levels: read uncommitted, read committed, repeatable read, and serializable. Hopefully it has helped you get your head around them. I'll be going into much more detail with practical hands on exercises in my training days at Engine Yard's San Francisco office on the 24th and 31st of July. Visit www.dbisyourfriend.com for course and registration details.

more »

Engine Yard AppCloud CLI »

Created at: 20.07.2010 12:00, source: Engine Yard Blog, tagged: Uncategorized

At Engine Yard we've been helping developers ship Ruby applications for almost four years. Our approach to deployment has changed a few times but at its core our focus has always been helping people deploy and scale Ruby on Rails applications on virtualized hardware. Almost two years ago, we started experimenting with Amazon's AWS service and realized that people wanted more of a self service setup. For the first time, we decided to take a stab at providing the same kind of service on other people's hardware instead of our own. This has grown into our AppCloud offering. Today, we're happy to announce an awesome new addition to AppCloud that enables developers to ship code faster, easier, and straight from the command line.

A Bit of EY History

In our early days, we provided our customers with customized capistrano recipes to deploy their ruby applications to our clusters. A problem quickly arose because we also needed to help them maintain this recipe as we helped them scale their applications. We learned that keeping our customers' capistrano recipes up to date was a truly painful exercise, so when we built AppCloud we went with a more centralized approach.

Early AppCloud Direction

We thought that solving the problem of keeping most of the deployment related information in sync was so painful that we built a web based deployment strategy. It wasn't the worst idea ever, but the disconnect between leaving your shell and going to a web browser isn't really what developers want. In addition, we were so excited about the idempotency that chef offered at a configuration level that we felt it was imperative to "verify" the state of the system with a chef run each time we shipped code. This made pushing code slower than necessary and occasionally created panic situations if the chef run failed for some strange reason. People could still use capistrano with AppCloud, but it required them to re-download their deployment recipes every time their environment changed. There also wasn't an easy way to maintain customizations if customers kept having to re-download the capistrano recipe. Over and over again, we kept hearing the same complaints from customers. Customers liked the provisioning flexibility on AWS but shipping code on AppCloud was suboptimal. A few months ago, we finally admitted that our intentions were correct but we hadn't been doing the best things for our customers. We started working on a way to help our customers ship code more effectively.

Customer Feedback is Awesome

We accepted that idempotency is extremely important when it comes to system configuration but that doesn't mean you need to re-run chef each time you ship application code. We realized that people want to see their code running on their servers ASAP. Finally, we embraced the idea that people want to ship code with a command line tool similar to the way most people use rake to run their test suite. We're happy to introduce a more pleasant way to ship code to AppCloud, the engineyard gem.

A Better Workflow

The old way of deploying with chef works, but it forces you to reconfigure your servers every single time you deploy. The workflow looked like this: * Boot some instances (provision, configure, deploy) * Ship code (run configuration, deploy code) * Ship code (run configuration, deploy code) * Ship code (run configuration, deploy code) * Tweak system configuration (configure) * Ship code (configure, deploy) * Ship code (configure, deploy) * ... With the Engine Yard CLI, you can deploy without verifying your system's configuration, so it's quite a bit faster to ship new code. The new workflow looks like this: * Boot some instances (provision, configure) * Ship code (deploy) * Ship code (deploy) * Ship code (deploy) * Tweak system configuration (configure) * Ship code (deploy) * Ship code (deploy) * ... We really think our customers are going to prefer this approach because, let's face it, we ship code way more often than we reconfigure systems.

Get Started

* gem install engineyard * cd ~/myapp * ey deploy One of the things we like most about the new CLI is that it shows you, in real time, what's going on with your deploy. If something goes wrong, you don't have to scroll through a huge log in your browser; the error messages are right there in your terminal. When it succeeds, the process exits, so you know immediately that it's done. No more staring at the dashboard waiting for a spinning dot to turn into a green one. How about ey deploy && mpg123 woohoo.mp3 || mpg123 sad-trombone.mp3? That's immediate, unmistakable, annoying, audible feedback. You can't get that from a green dot.

Other Great Features

* Full Bundler Support * Maintenance Pages * Deploy Hooks for Extra Configuration * Advanced Deployment Customization * Ensure System Configuration is Current You can do a lot more than just deploy with the engineyard gem. Check out the docs and the FAQ. Go forth and ship!

more »

Concurrency and the AASM Gem »

Created at: 19.07.2010 12:13, source: Engine Yard Blog, tagged: Uncategorized

Your Ruby on Rails code is run concurrently, whether you like it or not. Concurrency is a staple term when talking about hosting infrastructure, but it is too often brushed aside when discussing actual code bases. This attitude is especially prevalent in the Ruby on Rails community: I can't name one popular plugin that gets it right. In this post I will address problems with the typical state machine pattern used by Rails applications, and show you how to address them and make your code bullet-proof.

The Problem

Consider the following controller action, backing a big green "ship button" next to a purchase order:

def ship
  @order = PurchaseOrder.find(params[:id])
  @order.ship!
  redirect_to order_path(@order)
end

Imagine two users both press the "ship" button at the same time. (Or as often happen, one user double clicks the button.) The two requests will hit the load balancer and be distributed out to run on different processes. What happens when the above code---typical of many rails applications---is run in two different places at the same time? Both processes will load the order from the database at line 2. At line 3 when the ship! method is run, both processes will check the attributes of the order and see that it is currently unshipped. As a result, both execute shipping code, which may include sending emails, updating caches, and transferring funds. As a result, the customer will receive duplicate emails, or worse, be charged twice. All versions of acts_as_state_machine (AASM) exhibit this behavior.

The Fix

Any time you read data from the database with the intention of making changes based on that data ("ship the order if it isn't already shipped") you must obtain an exclusive database lock on the row (or employ some form of optimistic locking strategy when updating, a topic not covered in this post). The database will block any processes trying to access that row until the session that obtained the lock concludes its transaction (COMMIT or ROLLBACK). ActiveRecord allows us to do this using the :lock flag:

def ship
  PurchaseOrder.transaction do
    @order = PurchaseOrder.find(params[:id], :lock => true)
    @order.ship!
  end
  redirect_to order_path(@order)
end

Working through the above example again, the first process to execute the find will issue the following SQL:

SELECT * FROM purchase_orders WHERE id = 1 FOR UPDATE

Notice the "FOR UPDATE" on the end; this instructs the database to place an exclusive lock on the row. When the second process executes the find and submits the above SQL to the database, the database will wait for the first transaction to complete (after calling ship! and updating the state of the order) before reading and returning the row. The returned row will now have a state of "shipped", and as such the ship! method will effectively be a noop (no operation). The customer will only receive one email. It is also possible using ActiveRecord to lock an object that has been already loaded from the database:

def ship
  @order = PurchaseOrder.find(params[:id])
  PurchaseOrder.transaction do
    @order.lock!
    @order.ship!
  end
  redirect_to order_path(@order)
end

This is equivalent to a reload, but adds the "FOR UPDATE" suffix necessary for a database lock. It is an extra SQL statement (the order is selected twice), but is an easier pattern to abstract away.

class Order < ActiveRecord::Base
  # This method is usually provided by AASM
  def ship!
    return if shipped?
  # Important emails and computations
  end
  def ship_with_lock!
    transaction do
      lock!
      ship_without_lock!
    end
  end
  alias_method_chain :ship!, :lock
end

With alias_method_chain, we can continue to use exactly the same controller code we started with (just a plain call to ship!), and locking is handled for us in the background. Lost updates or duplicate execution won't be a problem for every website, but if you are starting to worry about the concurrency of your hosting infrastructure, it's worth having a look over your code too. If you’d like to join me for some hands-on work with this, I’ll be running classes at Engine Yard's San Francisco office on the 24th and 31st of July. Visit www.dbisyourfriend.com for course and registration details.

more »

RailsConf, Here We Come! »

Created at: 08.06.2010 00:22, source: Engine Yard Blog, tagged: Uncategorized

It's another year, and time for another RailsConf! The Engine Yard crew is on-site in force, as usual, and we're excited to meet you! We're set up right in the center of the Exhibit Hall, with swag, demos, and a gaggle of Rails experts who'd love to meet you. On the schedule we've got Keynotes from Yehuda Katz and Evan Phoenix (Engine Yard's Open Source Love Affair), along with talks from Andre Arko (Bundler: Painless Dependency Management), Evan Phoenix and Charles Nutter (Ruby's Dark and Dusty Corners + Rubinius 1.0), Jon Crosby (Engine Yard's Community Powered Cloud) and last but not least, Nick Sieger (Rails 3 + JRuby: Awesome Framework, Awesome Platform)—so whether you're looking to get up to date on JRuby, Rubinius, Rails 3 or anything else, we've got all the info you'll need. Got a question about your app or setup? Curious about Rails 3? We've got engineers, sales reps, support techs—all on site and waiting to help. Just want the latest n' greatest Engine Yard tee (new design!)? We've got that too, complete with awesome attendee giveaways! Stop by and say hello—we'll see you there!

more »