bgror

RailsConf 2010 - Thank You! »

Created at: 11.06.2010 01:30, source: OnRails.org, tagged: Ruby on Rails

Wow, RailsConf is over. It ended nicely with @garyvee giving a powerful and entertaining keynote covering many subjects but mostly his standard spiel on connecting with your customers or audience. I read his book, crush it, a while back and enjoyed it. Note since, I haven’t been the most active on my blog or website…but I’m working on changing this.

Now of course RailsConf was way more that this keynote. I must admit I really enjoyed Baltimore and having the convention center, the hotel and the port at a five minutes walking distance. It’s a great area, I understand that not the whole city is like that, but that doesn’t remove anything from the fact that I really appreciate the place right now.

The best thing at this conference is the energy that transpires and I really feel energized and want to start some new Rails venture. It’s fun to see so many enthusiasts trying to learn, share, and push the community forward.

I must admit I usually enjoy smaller conferences, like the MoutainWest RubyConf or the forthcoming Moutain.rb, better as the tracks are more specialized and geekier. RailsConf tries to have several tracks for everybody but very few are very technical. Maybe the organizers should try to have an advanced track next time. That way the conference would stay appealing for the many people that have been doing Rails/Ruby for years and still be welcoming for beginners.

As usual there are always some talks that just suck. On the tutorial day I attended in the afternoon Rails 3 Deep Dive and it wasn’t about Rails 3 nor a deep dive. Another talk that didn’t turn out the way it should have been was Scaling Rails on AppEngine with JRuby and Duby where the speakers where knowledgeable but where note prepared enough and where counting on the wifi to work…The wifi never works at conference. This said the wifi was usually working pretty well at RailsConf this time.

I don’t wanna focus too much on the talks that didn’t work out as most of the talks where great and also not all my talks in the past always worked out they way they could have ;-)

One of the tutorial I really liked was Acceptance Testing With Cucumber. David Chelimsky rocked and just knows his stuff and Aslak hanged in there pretty well. They had an application prepared with multiple branches that let the audience follow right along. That was great. They didn’t have time to access a few of the advanced Cucumber subjects…next time make that a full day tutorial.

The first talk I attended on tuesday was Building an API with Rails which was a panel discussion with guys from Twitter, 37Signals, Github, the NYT and others. I really enjoyed that there was several distinct views on several aspects like on APIs like versioning, security, performance which relates exactly with some customer work I’m currently doing.

Then I went to the Metrics Magic by Aaron Bedra from Relevance which cover tools like RCov,Flog, Flay, Roodi, Reek and how to integrate them with your continuous integration build. For example why don’t you make your build fail when it reaches a certain threshold of uncovered code (let’s say 20%)…I like that idea. We do generate these stats in our ci builds but don’t fail the builds…yet.

Another great panel on tuesday was The State of Rails e-Commerce. I’m interested in ecommerce since the first of Rails project I worked on end of 2005.

Then I went to see Ilya Grigorik talk on “No callbacks, no threads: async & cooperative web servers with Ruby 1.9”…I just wish all the task where that awesome…Ilya tries to push the current stack of Ruby technologies to the next step to try to get the same benefits than what node.js provides…and he showed us how. Wow.

I’m not gonna list all the talks I attended here, but will provide a few more comments. There where two business oriented talks I enjoyed, the Million Dollar Mongo by Obie Fernandez and Durran Jordan from Hashrocket and the Agile the Pivotal Way talk by Ian MCFarland. Ian’s presented how his teams operates and I now see why Pivotal is on such a growth path. There are many cool aspects they are enforcing, one key I believe is to have really agile team members than can switch in an out of each team while still providing continuity to the customer by having one anchor member. Another essential aspect is the culture and how they propagate it buy doing peer programing to the extreme. Obie’s talk was an interesting retrospective on a very large project (10’0000 hours) they undertook. What surprised me is that he was pretty negative on his client and also went after some members of the Rails community…Well, maybe Hashrocket should stick to smaller projects ;-)

What else was cool? Rich Kilmer gave a nice talk about Authentication in a RESTful World. Again, that’s totally relevant to a customer project of mine. One of the best talks out there was Rocket Fueled Cucumbers by Joseph Wilk, but I saw only the last quarter as I selected another talk that I decided to leave…too late.

Matthew Deiters also gave a presentation I really enjoyed about “Recommendations in Rails”…something I may have to build in one of my apps very soon. I’m less exited about the fact that he recommends a java tool (Neo4j) to manage your graph of relations…but I trust him that it’s the best of the solutions out there for now.

Besides the talks Bluebox threw a party at a local bar, besides the fact that Fernand managed to get us lost on they way and I wasn’t sure we would survive the neighborhood we ended up in…the party was great and we met one of the Bluebox software developers and her mam and had a great time. At least I think so based on the trouble I had to wake up the next day.

I had fun with the keynotes. Derek Sivers gave a talk which I enjoyed even though it was not related to Rails but it was very entertaining. I really liked Yehuda Katz talk and he his certainly the driving factor behind Rails 3 but he gave a lot of credits to specific members of the community which took on many issues that most thought where impossible to address or change and fixed them.

The Ruby Heroe Awards Ceremony is always fun to watch, and Gregg puts lots of effort into making them happen. Next time just let the winners make a thank speech…that would stress them a little.

Overall there wasn’t enough coverage of Rails 3, I guess that’s gonna for next year when everyone migrated all there projects to it.

So reflecting on these 4 past days…it was a great conference. A big thanks to the organizers and all the presenters

Now time to kick off a new Rails 3 project.

Enjoy!
Daniel

Does Rails Performance Need an Overhaul? »

Created at: 09.06.2010 22:56, source: Phusion Corporate Blog, tagged: ruby Ruby on Rails

Igvita.com has recently published the article Rails Performance Needs an Overhaul. Rails performance… no, Ruby performance… no Rails scalability… well something is being criticized here. From my experience, talking about scalability and performance can be a bit confusing because the terms can mean different things to different people and/or in different situations, yet the meanings are used interchangeably all the time. In this post I will take a closer look at Igvita’s article.

Performance vs scalability

Let us first define performance and scalability. I define performance as throughput; number of requests per second. I define scalability as the amount of users a system can concurrently handle. There is a correlation between performance and scalability. Higher performance means each request takes less time, and so is more scalable, right? Sometimes yes, but not necessarily. It is entirely possible for a system to be scalable, yet manages to have a lower throughput than a system that’s not as scalable, or for a system to be uber-fast yet not very scalable. Throughout this blog post I will show several examples that highlight the difference.

“Scalability” is an extremely loaded word and people often confuse it with “being able to handle tons and tons of traffic”. Let’s use a different term that better reflects what Igvita’s actually criticizing: concurrency. Igvita claims that concurrency in Ruby is pathetic while referring to database drivers, Ruby application servers, etc. Some practical examples that demonstrate what he means are as follows.

Limited concurrency at the app server level

Mongrel, Phusion Passenger and Unicorn all use a “traditional” multi-process model in which multiple Ruby processes are spawned, each process handling a single request per second. Thus, concurrency is (assuming that the load balancer has infinite concurrency) limited by the number of Ruby processes: having 5 processes allow you to handle 5 users concurrently.

Threaded servers, where the server spawns multiple threads, each handling 1 connection concurrently, allow more concurrency because because it’s possible to spawn a whole lot more threads than processes. In the context of Ruby, each Ruby process needs to load its own copy of the application code and other resources, so memory increases very quickly as you spawn additional processes. Phusion Passenger with Ruby Enterprise Edition solves this problem somewhat by using copy-on-write optimizations which save memory, so you can spawn a bit more processes, but not significantly (as in 10x) more. In contrast, a multi-threaded app server does not need as much memory because all threads share application code with each other so you can comfortably spawn tens or hundreds of threads. At least, this is the theory. I will later explain why this does not necessarily hold for Ruby.

When it comes to performance however, there’s no difference between processes and threads. If you compare a well-written multi-threaded app server with 5 threads to a well-written multi-process app server with 5 processes, you won’t find either being more performant than the other. Context switch overhead between processes and threads are roughly the same. Each process can use a different CPU core, as can each thread, so there’s no difference in multi-core utilization either. This reflects back on the difference between scalability/concurrency and performance.

Multi-process Rails app servers have a concurrency level that can be counted with a single hand, or if you have very beefy hardware, a concurrency level in the range of a couple of tens, thanks to the fact that Rails needs about 25 MB per process. Multi-threaded Rails app servers can in theory spawn a couple of hundred of threads. After that it’s also game over: an operating system thread needs a couple MB of stack space, so after a couple hundreds of threads you’ll run out of virtual memory address on 32-bit systems even if you don’t actually use that much memory.

There is another class of servers, the evented ones. These servers are actually single-threaded, but they use a reactor style I/O dispatch architecture for handling I/O concurrency. Examples include Node.js, Thin (built on EventMachine) and Tornado. These servers can easily have a concurrency level of a couple of thousand. But due to their single-threaded nature they cannot effectively utilize multiple CPU cores, so you need to run a couple of processes, one per CPU core, to fully utilize your CPU.

The limits of Ruby threads

Ruby 1.8 uses userspace threads, not operating system threads. This means that Ruby 1.8 can only utilize a single CPU core no matter how many Ruby threads you create. This is why one typically needs multiple Ruby processes to fully utilize one’s CPU cores. Ruby 1.9 finally uses operating system threads, but it has a global interpreter lock, which means that each time a Ruby 1.9 thread is running it will prevent other Ruby threads from running, effectively making it the same multicore-wise as 1.8. This is also explained in an earlier Igvita article, Concurrency is a Myth in Ruby.

On the bright side, not all is bad. Ruby 1.8 internally uses non-blocking I/O while Ruby 1.9 unlocks the global interpreter lock while doing I/O. So if one Ruby thread is blocked on I/O, another Ruby thread can continue execution. Likewise, Ruby is smart enough to cause things like sleep() and even waitpid() to preempt to other threads.

On the dark side however, Ruby internally uses the select() system call for multiplexing I/O. select() can only handle 1024 file descriptors on most systems so Ruby cannot handle more than this number of sockets per Ruby process, even if you are somehow able to spawn thousands of Ruby threads. EventMachine works around this problem by bypassing Ruby’s I/O code completely.

Naive native extensions and third party libraries

So just run a couple of multi-threaded Ruby processes, one process per core and multiple threads per process, and all is fine and we should be able to have a concurrency level of up to a couple hundred, right? Well not quite, there are a number of issues hindering this approach:

• Some third party libraries and Rails plugins are not thread-safe. Some aren’t even reentrant. For example Rails < 2.2 suffered from this problem. The app itself might not be thread-safe.
• Although Ruby is smart enough not to let I/O block all threads, the same cannot be said of all native extensions. The MySQL extension is the most infamous example: when executing queries, other threads cannot run.

Mongrel is actually multi-threaded but in practice everybody uses in multi-process mode (mongrel_cluster) exactly because of these problems. It is also the reason why Phusion Passenger has also gone the multi-process route.

And even though Thin is evented, a typical Ruby web application running on Thin cannot handle thousands of concurrent users. This is because evented servers typically require a special evented programming style, such as the one seen in Node.js and EventMachine. A Ruby web app that is written in an evented style running on Thin can definitely handle a large number of concurrent users.

When is limited application server concurrency actually a problem?

Igvita is clearly disappointed at all all the issues that hinder Ruby web apps from achieving high concurrency. For many web applications I would however argue that limited concurrency is not a problem.

• Web applications that are slow, as in CPU-heavy, max out CPU resources pretty quickly so increasing concurrency won’t help you.
• Web applications that are fast are typically quick enough at handling the load so that even large number of users won’t notice the limited concurrency of the server.

Having a concurrency of 5 does not mean not mean that the app server can only handle 5 requests per second; it’s not hard to serve hundreds of requests per second with only a couple of single-threaded processes.

The problem becomes most evident for web applications that have to wait a lot for I/O (besides its own HTTP request/response cycle). Examples include:

1. Apps that have to spend a lot of time waiting on the database.
2. Apps that perform a lot of external HTTP calls that respond slowly.
3. Chat apps. These apps typically have thousands of users, most of them doing nothing most of the time, but they all require a connection (unless your app uses polling, but that’s a whole different discussion).

We at Phusion have developed a number of web applications for clients that fall in the second category, the most recent one being a Hyves gadget. Hyves is the most popular social network in the Netherlands and they get thousands of concurrent visitors during the day. The gadget that we’ve developed has to query external HTTP servers very often, and these servers can take 10 seconds to respond in extreme cases. The servers are running Phusion Passenger with maybe a couple tens of processes. If every request to our gadget also causes us to wait 10 seconds for the external HTTP call then we’d soon run out of concurrency.

But even suppose that our app and Phusion Passenger can have a concurrency level of a couple of thousand, all of those visitors will still have to wait 10 seconds for the external HTTP calls, which is obviously unacceptable. This is another example that illustrates the difference between scalability and performance. We had solved this problem by aggressively caching the results of the HTTP calls, minimizing the number of external HTTP calls that are necessary. The result is that even though the application’s concurrency is fairly limited, it can still comfortably serve many concurrent users with a reasonable response time.

This anecdote should explain why I believe that web apps can get very far despite having a limited concurrency level. That said, as Internet usage continues to increase and websites get more and more users, we may at some time come to a point where much a larger concurrency level is required than most of our current Ruby tools allow us to (assuming server capacity doesn’t scale quickly enough).

What was Igvita.com criticizing?

Igvita.com does not appear to be criticizing Ruby or Rails for being slow. It doesn’t even appear to be criticizing the lack of Ruby tools for achieving high concurrency. It appears to be criticizing these things:

• Rails and most Ruby web application servers don’t allow high concurrency by default.
• Many database drivers and libraries hinder concurrency.
• Although alternatives exist that allow concurrency, you have to go out of your way to find them.
• There appears to be little motivation in the Ruby community for making the entire stack of web frame work + web app server + database drivers etc scalable by default.

This is in contrast to Node.js where everything is scalable by default.

Do I understand Igvita’s frustration? Absolutely. Do I agree with it? Not entirely. The same thing that makes Node.js so scalable is also what makes it relatively hard to program for. Node.js enforces a callback style of programming and this can eventually make your code look a lot more complicated and harder to read than regular code that uses blocking calls. Furthermore, Node.js is relatively young – of course you won’t find any Node.js libraries that don’t scale! But if people ever use Node.js for things other than high-concurrency servers apps, then non-scalable libraries will at some time pop up. And then you will have to look harder to avoid these libraries. There is no silver bullet.

That said, all would be well if at least the preferred default stack can handle high concurrency by default. This means e.g. fixing the MySQL extension and have the fix published by upstream. The mysqlplus extension fixes this but for some reason their changes aren’t accepted and published by the original author, and so people end up with a multi-thread-killing database driver by default.

Is Node.js innovative? Is Ruby lacking innovation?

A minor gripe that I have with the article is that Igvita calls Node.js innovative while seemingly implying that the Ruby stack isn’t innovating. Evented servers like Node.js actually have been around for years and the evented pattern is well-known long before Ruby or Javascript have become popular. Thin is also evented and predates Node.js by several years. Thin and EventMachine also allow Node.js-style evented programming. The only innovation that Node.js brings, in my opinion, is the fact that it’s Javascript. The other “innovation” is the lack of non-scalable libraries.

Conclusion

Igvita appears to be criticizing something other than Rails performance, as his article’s title would imply.

I don’t think the concurrency levels that the Rails stack provides by default is that bad in practice. But as a fellow programmer, it does intuitively bother me that our laptops, which are a million times more powerful than supercomputers from two decades ago, cannot comfortably handle a couple of thousand concurrent users. We can definitely work towards something better, but in the mean time let’s not forget that the current stack is more than capable of Getting Work Done(tm).

Ruby on Rails 3 Security Updated »

Created at: 08.06.2010 15:13, source: Ruby on Rails Security Project, tagged: cross-site scripting rails Ruby on Rails security sql injection sqli web security XSS

I hold a talk about Rails 3 Security at the RailsWayCon10. It is about the new Cross-Site Scription protection in Rails 3, what is going to change in ActiveRecord and other Rails Security topics. You can find the presentation at Slideshare.

Rails Performance Needs an Overhaul »

Created at: 07.06.2010 17:32, source: igvita.com, tagged: Architecture Ruby on Rails performance rails

Browsers are getting faster; JavaScript frameworks are getting faster; MVC frameworks are getting faster; databases are getting faster. And yet, even with all of this innovation around us, it feels like there is massive gap when it comes to the end product of delivering an effective and scalable service as a developer: the performance of most of our web stacks, when measured end to end is poor at best of times, and plain terrible in most.

The fact that a vanilla Rails application requires a dedicated worker with a 50MB stack to render a login page is nothing short of absurd. There is nothing new about this, nor is this exclusive to Rails or a function of Ruby as a language - whatever language or web framework you are using, chances are, you are stuck with a similar problem. But GIL or no GIL, we ought to do better than that. Node.js is a recent innovator in the space, and as a community, we can either learn from it, or ignore it at our own peril.

Measuring End-to-End Performance

A modern web-service is composed of many moving components, all of which come together to create the final experience. First, you have to model your data layer, pick the database and then ensure that it can get your data in and out in the required amount of time - lots of innovation in this space thanks to the NoSQL movement. Then, we layer our MVC frameworks on top, and fight religious wars as developers on whose DSL is more beautiful - to me, Rails 3 deserves all the hype. On the user side, we are building faster browsers with blazing-fast JavaScript interpreters and CSS engines. However, the driveshaft (the app server) which connects the two pieces (the engine: data & MVC), and the front-end (the browser + DOM & JavaScript), is often just a checkbox in the deployment diagram. The problem is, this checkbox is also the reason why the ‘scalability’ story of our web frameworks is nothing short of terrible.

It doesn't take much to construct a pathological example where a popular framework (Rails), combined with a popular database (MySQL), and a popular app server (Mongrel) produce less than stellar results. Now the finger pointing begins. MySQL is more than capable of serving thousands of concurrent requests, the app server also claims to be threaded, and the framework even allows us to configure a database pool!

Except that, the database driver locks our VM, and both the framework and the app server still have a few mutexes deep in their guts, which impose hard limits on the concurrency (read, serial processing). The problem is, this is the default behaviour! No wonder people complain about 'scalability'. The other popular choices (Passenger / Unicorn) “work around” this problem by requiring dedicated VMs per request - that's not a feature, that's a bug!

The Rails Ecosystem

To be fair, we have come a long way since the days of WEBrick. In many ways, Mongrel made Rails viable, Rack gave us the much needed interface to become app-server independent, and the guys at Phusion gave us Passenger which both simplified the deployment, and made the resource allocation story moderately better. To complete the picture, Unicorn recently rediscovered the *nix IPC worker model, and is currently in use at Twitter. Problem is, none of this is new (at best, we are iterating on the Apache 1.x to 2.x model), nor does it solve our underlying problem.

Turns out, while all the components are separate, and its great to treat them as such, we do need to look at the entire stack as one picture when it comes to performance: the database driver needs to be smarter, the framework should take advantage of the app servers capabilities, and the app server itself can't pretend to work in isolation.

If you are looking for a great working example of this concept in action, look no further than node.js. There is nothing about node that can't be reproduced in Ruby or Python (EventMachine and Twisted), but the fact that the framework forces you to think and use the right components in place (fully async & non-blocking) is exactly why it is currently grabbing the mindshare of the early adopters. Rubyists, Pythonistas, and others can ignore this trend at their own peril. Moving forward, end-to-end performance and scalability of any framework will only become more important.

Fixing the "Scalability" story in Ruby

The good news is, for every outlined problem, there is already a working solution. With a little extra work, the driver story is easily addressed (MySQL driver is just an example, the same story applies to virtually every other SQL/NoSQL driver), and the frameworks are steadily removing the bottlenecks one at a time.

After a few iterations at PostRank, we rewrote some key drivers, grabbed Thin (evented app server), and made heavy use of continuations in Ruby 1.9 to create our own API framework (Goliath) which is perfectly capable of serving hundreds of concurrent requests at a time from within a single Ruby VM. In fact, we even managed to avoid all the callback spaghetti that plagues node.js applications, which also means that the same continuation approach works just as well with a vanilla Rails application. It just baffles me that this is not a solved problem already.

The state of art in the end-to-end Rails stack performance is not good enough. We need to fix that.

Making CRUD less "Cruddy", one step at a time  »

Created at: 28.04.2010 03:42, source: OnRails.org, tagged: Ruby on Rails Rails Tips

One of the great “new” features of Rails (as of 2.3) is accepts_nested_attributes_for, allowing you to build cross-model CRUD forms without “cruddying” your controller. There are some great examples out there about how to do this, but I’d like to walk thorough a particular use case — managing the “join” records in a has_many :through relationship.

Consider the following database schema:

class Villain < ActiveRecord::Base
  has_many :gifts
  has_many :super_powers, :through => :gifts
end

class Gift < ActiveRecord::Base
  belongs_to :villain
  belongs_to :super_power

  validates_uniqueness_of :super_power, :scope => :villain_id
end

class SuperPower < ActiveRecord::Base
  has_many :gifts
  has_many :villains, :through => :gifts
end

In our dataset, there are a relatively small number of super powers which we wish to present as a list of checkboxes on the villain management form. Checking/unchecking the boxes will manage the gift records for that villain, effectively managing the list of super powers available to the baddy.

To get started, we need to add accepts_nested_attributes_for :gifts to the Villain class — piece of cake. To complete the implementation, we need to change the params hash that our form generates. Let’s review the cases that we need to support and the associated params hash format needed to implement the correct functionality.

The first case is a super power record that is not currently associated with the villain. Here, the UI should display an unchecked checkbox. If we check it and submit the form, a gift record should be created linking the villain with the super power, making this bad guy that much badder. Here is an example of the params hash we should be sending to accomplish this:

  {
    'villain' => {
      'name' => 'Lex Luthor',
      ...
      'gifts_attributes' => {
        1 => { 'super_power_id' => 5 },
        2 => { 'super_power_id' => 7 },
        ...
      }
    }
  }

The alternate case is a super power this villain already possesses. In this instance, the UI should display a checked checkbox, and if we uncheck it, the existing gift record should be deleted, diminishing the villain’s capacity for evil. And our params hash needs to look like:

  {
    'villain' => {
      'name' => 'Two-Face',
      ...
      'gifts_attributes' => {
        1 => { 'id' => 101, '_delete' => true },
        ...
      }
    }
  }

Note that the keys for the gifts_attributes hash are arbitrary; we can use any scheme to generate unique keys for the hash.

So how can we craft a form that sends the params hash that Rails wants to see? Here’s my implementation:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%- if gift = @villain.gifts.find_by_super_power_id(super_power.id) -%>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][id]", gift.id %>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][_delete]", false, true %>
        <%= hidden_field_tag "villain[gifts_attributes][#{ index }][_delete]", true %>
      <%- else -%>
        <%= check_box_tag "villain[gifts_attributes][#{ index }][super_power_id]", super_power.id %>
      <%- end -%>

      <%= super_power.name %>
    </label><br />
  <%- end -%>

If the gift is detected, the villain has the super power, and we handle our second case from above, using the checkbox / hidden field hack Rails employs in the check_box helper method to make sure a value is sent whether or not the checkbox is checked. The else block handles the other case, setting up our params hash to create the gift if the checkbox is checked.

This works, but we probably don’t want to copy and paste that code everywhere we use this pattern. How can we reuse this in a DRY fashion? Here’s a helper method that encapsulates the logic:

  def has_join_relationship(model, join_collection_name, related_item, collection_index, options={})
    returning "" do |output|
      relationship_name = options[:relationship_name] || related_item.class.table_name.singularize + "_id"
      tag_prefix = "#{ model.class.class_name.underscore }[#{ join_collection_name }_attributes][#{ collection_index }]"

      if join_item = model.send(join_collection_name).find(:first, :conditions => { relationship_name => related_item.id })
        output << hidden_field_tag("#{ tag_prefix }[id]", related_item.id)
        output << check_box_tag("#{ tag_prefix }[_delete]", false, true)
        output << hidden_field_tag("#{ tag_prefix }[_delete]", true)
      else
        output << check_box_tag("#{ tag_prefix }[#{ relationship_name }]", related_item.id, false)
      end
    end
  end

Drop that in a helper, and then your form code becomes:

  <%- SuperPower.all.each_with_index do |super_power, index| -%>
    <label>
      <%= has_join_relationship(@villain, :gifts, super_power, index) %>
      <%= super_power.name %>
    </label><br />
  <%- end -%>

Much nicer … although I’m not sold on the name has_join_relationship. Any suggestions?