bgror

My Summer of (Open) Source »

Created at: 05.01.2012 22:25, source: Engine Yard Blog, tagged: Open Source ruby Technology

The last few months have been an great experience for me. I’m a graduate student from Potsdam, Germany. However, as some of you might already know, I’m also rather active in the Ruby community. This past year, I had an amazing opportunity.

Engine Yard sponsors a couple of Open Source developers to work full time on their projects. When I asked Dr. Nic Williams whether they would sponsor me spending three months in Portland, working together with Brian Ford on Rubinius, I expected nothing but a no. Turns out, Engine Yard was at least as thrilled about this idea as I was. A few days ago, I finally got back to Germany, and I wanted to give you a quick overview of what I’ve been working on during my time overseas.Like many others, I started contributing to Rubinius a while ago. However, I never really dared to play with the internals. So, my first stop was the Rubinius compiler. To make sure I really understood it and that it’s as flexible as it claims to be, I wrote a Smalltalk implementation using the Rubinius compiler infrastructure and looked into improving its API.

It’s a fun thing to do, as the Rubinius compiler is written entirely in Ruby. And, since Rubinius is bootstrapped, it also runs on other Ruby implementations. That is how you usually install Rubinius: You load the compiler from CRuby, it then compiles the compiler to Rubinius bytecode. If you want to look into this, there is some excellent documentation available on the Rubinius website.

This bytecode can then be executed by the Virtual Machine, which was my next stop. It took me a while to fully understand how things work within the VM. It is actually the only major part of Rubinius not written in Ruby, and the main reason for it’s blazing performance and excellent memory footprint. I am planning ton writing another blog post, or possibly even a series of blog posts about these internals.

Apart from bug fixes and API improvements, I used the gained knowledge to fix, for instance, one of Ruby’s least known and most confusing feature: the implemented flip-flops.

The last thing I worked on was Puma, a new web server for Rails/Rack/Sinatra applications. Rubinius 2.0 is about to be released, fully able to make the best use of all your CPUs. However, most web servers used for deploying Ruby applications are actually single-threaded. Since there is no real threaded option that is still maintained and not JRuby specific, Evan Phoenix and I started working on a new server.

Like many other servers, it uses the rapid HTTP parser that comes with Mongrel. It also uses a dynamically sized thread-pool for processing requests in parallel. With Puma, you now have a go to choice when it comes to deploying web applications on Rubinius. And since it does not contain any Rubinius specific code, it also works quite well on JRuby or CRuby.

To make sure we are heading in the right direction, I started working on a tool for benchmarking web applications under realistic load. The main issue with just using ab, the standard solution for measuring HTTP performance, is that it results in unrealistic numbers both on JRuby and Rubinius. When using ab, you just send the same request over and over again, causing the JIT and code inliner to highly optimize for exactly that request. This usually doesn’t reflect the actual production behavior, though. I therefore wrote code simulating a real browser session and, of course, running multiple of these sessions in parallel.

You think that’s all? Far from it! The Engine Yard OSS Community Grant Program enabled me to speak at six different conferences all over America. At Rocky Mountain Ruby, RubyConf Brazil and RubyConf Uruguay, I gave a talk on “Real Time Rack”. In San Francisco, at GoGaRuCo, I gave a presentation about “Smalltalk On Rubinius - or How To Implement Your Own Programming Language”. At this past year’s RubyConf in New Orleans, I spoke about “Message in a Bottle” and last but not least I gave a presentation titled “Beyond Ruby” at RubyConf Argentina in Buenos Aires.

rspec-rails-2.8.1 is released »

Created at: 05.01.2012 07:43, source: David Chelimsky, tagged: bdd rspec rails ruby

Bug fix release

The rails-3.2.0.rc2 release broke stub_model in rspec-rails-2.0.0 > 2.8.0. The rspec-rails-2.8.1 release fixes this issue, but it means that when you upgrade to rails-3.2.0.rc2 or greater, you’ll have to upgrade to rspec-rails-2.8.1 or greater.

Because rspec-rails-2.8.1 supports all versions of rails since 3.0, I recommend that you upgrade to rspec-rails-2.8.1 first, and then upgrade to rails-3.2.0.rc2 (or 3.2.0 once it’s out).

Changelog

http://rubydoc.info/gems/rspec-rails/file/Changelog.md

Docs

http://rubydoc.info/gems/rspec-rails
http://relishapp.com/rspec/rspec-rails

RSpec-2.8 is released! »

Created at: 05.01.2012 04:38, source: David Chelimsky, tagged: bdd rspec ruby

We released RSpec-2.8.0 today with a host of new features and improvements since 2.7. Some of the highlights are described below, but you can see the full changelogs at:

• http://rubydoc.info/gems/rspec-core/file/Changelog.md
• http://rubydoc.info/gems/rspec-expectations/file/Changelog.md
• http://rubydoc.info/gems/rspec-mocks/file/Changelog.md
• http://rubydoc.info/gems/rspec-rails/file/Changelog.md

Documentation

While not 100% complete yet, we’ve made great strides on RSpec’s RDoc:

• http://rubydoc.info/gems/rspec-core
• http://rubydoc.info/gems/rspec-expectations
• http://rubydoc.info/gems/rspec-mocks
• http://rubydoc.info/gems/rspec-rails

http://rspec.info is now just a one pager (desperate for some design love - volunteers please email rspec-users@rubyforge.org). All the old pages are redirects to the relevant RDoc at http://rubydoc.info. RSpec-1 info is still available at http://old.rspec.info.

We’ve still got Cucumber features up at http://relishapp.com/rspec, but we’re going to be phasing that out as the primary source of documentation. There are a lot of reasons for this, and I’ll try to follow up with a separate blog post on this topic.

rspec-core

Improved support for tags and filtering

You can now set default tags/filters in either RSpec.configure or a .rspec file and override these tags on the command line. For example, this configuration tells rspec to run all the examples that are not tagged :slow:

# in spec/spec_helper.rb
RSpec.configure do |c|
  c.treat_symbols_as_metadata_keys_with_true_values = true
  c.filter_run_excluding :slow
end

Now when you want run those, you can just do this:

rspec --tag slow

This will override the configuration and run onlly the examples tagged :slow.

–order rand

We added an --order option with two supported values: rand and default.

rspec --order random (or rand) tells RSpec to run the groups in a random order, and then run the examples within each group in random order. We implemented it this way (rather than complete randomization of every example) because we don’t want to re-run expensive before(:all) hooks. A fair tradeoff, as the resulting randomization is just as effective at exposing order-dependency bugs.

When you use --order random, RSpec prints out the random number it used to seed the randomizer. When you think you’ve found an order-dependency bug, you can pass the seed along and the order will remain consistent:

--order rand:3455

--order default tells RSpec to load groups and examples as they are declared in each file.

rspec –init

We added an --init switch to the rspec command to generate a “spec” directory, and “.rspec” and “spec/spec_helper.rb” files with some starter code in them.

rspec-expectations

We discovered that the matcher DSL generates matchers that run considerably slower than classes which implement the matcher protocol. We made some minor improvements in the DSL, but to really improve things we re-implemented every single built-in matcher as a class.

Special JRuby Release: 1.6.5.1 »

Created at: 28.12.2011 21:25, source: Engine Yard Blog, tagged: ruby Technology

For the Impatient

1. JRuby 1.6.5.1 is a single patch release of JRuby 1.6.5 to fix CERT advisory: CERT-2011-003.  ALL USERS: PLEASE UPGRADE
2. We talk about plans for the upcoming 1.6.6 release

CERT Details

Hashing 101

(For proper CSci vocabulary and a lot of fun details about hashing also read this wikipedia article)

Hash tables apply a math function (hashing function) to the key of a key-value pair. The result of the hashing function is a location to a hash bucket which stores the key/value pair internally:

a[:heh] = 1
hashing_function(:heh) -> store :heh/1 in hash bucket #3
a[:foo] = 2
hashing_function(:foo) -> store :foo/2 in hash bucket #13
a[:bar] = 3
hashing_function(:bar) -> store :bar/3 in hash bucket #1

Hashes have many buckets and in theory all key/value pairs added to a hash will get spread out evenly across the hashes buckets.  In practice, some number of keys will end up hashing into the same hash bucket (known as a hashing collision).  As you get more key/value pairs stored to the same hash bucket the time to access those particular key/value pairs will slow down.  This is because you need to walk some portion of the entries in the bucket to find the specific one you are looking for (hash structures will often make entries in an individual bucket a simple list structure).

a[:gar] = 4
hashing_function(:gar) -> store gar/4 in hash bucket #3 (same bucket as :heh)

In this example, accessing a[:gar] and a[:heh] may take longer than the other keys because they are sharing a hash bucket.

The Attack

The general application of the attack is for "the bad guys" to figure out a large set of values which will hash to the same hash bucket.  Once they create this list they will send all those values to a server.  The server will store them in a hash (think parameter list in Rack, for example).  The act of storing or accessing any of those values takes longer and longer as the number of entries in a single hash bucket grows.  The result will be a Denial Of Service (DOS) attack if enough values get stored.

hashing_function(:hostname) -> hash bucket #3
hashing_function(:aZ1) -> hash bucket #3
hashing_function(:cvg) -> hash bucket #3
hashing_function(:azr) -> hash bucket #3
... # many elided
hashing_function(:1fr) -> hash bucket #3
hashing_function(:yu3) -> hash bucket #3
hashing_function(:hyX) -> hash bucket #3

host = params[:hostname] # Uh oh! need to find this amongst many bucket buddies

The Fix

Adding a little bit of randomization to the hashing algorithm ends up making it much, much more difficult to figure out how to generate this type of attack.  JRuby 1.6.5.1 (and all later JRuby releases) all have this additional randomization built into the hashing algorithm.  The result should be decent hash bucket distribution that is difficult for attackers to predict.

More information

This vulnerability is not exclusively an issue of JRuby.  Other Ruby implementations also have a similar issue (also patched today).  In fact, Java and PHP also appear to be susceptible to this style of attack.  For more information, please see the CERT announcement.

Also, consider that language implementations are really only susceptible to this attack via frameworks which allow an external hacker to store arbitrary and/or unbounded key/values into a hash.  Ruby Rack had this vulnerability, but they have fixed things so that the amount of parameters stored is bounded by a size to remove the possibility of a DOS attack.  Rack users should upgrade to the latest version.

JRuby's First Security Fix-Only Release

We debated rolling what we have in our 1.6 branch along with the hashing vulnerability fix (mentioned above) and pushing out 1.6.6.  This was unappealing for a couple of reasons:

1. For stable environments deployed using 1.6.5 we would be asking them to evaluate this security fix and any other fix we placed on JRuby 1.6 branch in the last two months.  This seems like it would force more conservative users to perform their own build to manually patch just the security fix.
2. Of bugs we have fixed so far we felt we were about 10 short of what we wanted to have in JRuby 1.6.6

After consideration, we felt it best to give a security fix release now (A single security patch release JRuby 1.6.5.1 <--- update to this now please) to satisfy the cautious and to wait until we felt good about the quality of 1.6.6.  As they say, Open Source projects are ready when they are ready...

Hey! When will you be ready? What is missing?

It has been about two months since our last release and we suspect we can wrap things up in the next couple of weeks.  We plan on releasing JRuby 1.6.6 in mid-January.

As we have been saying all through the 1.6 series, we are primarily fixing 1.9 compatibility bugs.  Generally speaking, our 1.9 issue fixing has been dominated by encoding errors in Regexps, IO, and String.  Here is a list of what we have done so far.  It is also worth mentioned we fixed the regression which regressed Fiber (JRUBY-6170) in JRuby 1.6.5.  Also the dreaded missing 'read_nonblock' has been fixed (JRUBY-5529).

Here is the list of issues we are plan on settling for 1.6.6.  A few noteworthy mentions in this list is JRUBY-5657 (new 1.9 splat behavior), JRUBY- (new 1.9 to_ary behavior), and JRUBY-6067 (Windows YAML issue).

If there is some issue we don't have targetted but you think is drop-dead important then please let us know...We are willing to expedite other issues if presented with a reasonable case for why it should be fixed.  Please join the discussion.

Getting Started with JRuby and Java 7 »

Created at: 21.12.2011 23:38, source: Engine Yard Blog, tagged: ruby Technology

Unless you’ve been living under a rock, you’ve probably heard about the new hotness for JRuby: Java 7’s support for dynamic languages. You may also have heard about the huge perf gains that JRuby’s seeing when running on Java 7. How can you try it yourself?

Get Java 7

The reference implementation for Java is OpenJDK, and OpenJDK 7 has been out for almost six months now. The current version is 7u2 (’u’ stands for ‘update’), and includes a number of improvements over the GA (‘General Availability’) release.

Most platforms have easy access to OpenJDK builds. I’ll summarize the steps here.

Linux, Windows, and Solaris

Oracle provides binary downloads for Windows, Linux, and Solaris on its site. The JavaSE Downloads page links to both JDK and JRE download pages. You’ll probably want the JDK, since it includes other JVM-related dev tools, but the JRE will work too. Download, install, and you’re ready.

Additionally, package managers for Linux and Solaris will likely soon have OpenJDK 7u2 packages available, if they don’t already.

OS X

The official preview of OpenJDK for OS X lags behind, but you can get the current builds from theopenjdk-osx-build project. The build you want is currently labeled “OpenJDK-OSX-1.7-x64-u2-b21”, but any build labeled “1.7” and “u2” in the future will get what you need. The .dmg provides either a self-contained JDK for you to drop onto your system or a .pkg installer that does it for you.

Update: Henri Gomez, the primary guy behind openjdk-osx-build, has set up a page specifically for the update builds. Grab the JDK or JRE from his OpenJDK7JDK7UOSX page.

*BSD

The OS X work is based off the “bsd-port” branch of OpenJDK. There are links to Java 7 package information for FreeBSD, NetBSD, DragonFly BSD, and OpenBSD on the BSD Port wiki. These may not be updated to 7u2 yet.

Why Update 2?

We haven’t previously made a lot of noise about Java 7 and JRuby, nor assembled a blog post/tutorial like this, primarily because Java 7 GA was missing key optimizations in the invokedynamic subsystem. JRuby 1.7 will make heavy use of invokedynamic, and if we had released it before those optimizations were in place, it would have given people a bad impression of the power of invokedynamic.

Update 2 now has a small set of optimizations that make a very large difference. If you intend to start playing with JRuby 1.7 builds, we strongly recommend you use OpenJDK 7u2 or higher.

Update: Your jruby -v output should look something like this:

jruby 1.7.0.dev (ruby-1.8.7-p352) (2011-12-19 f404f75) (OpenJDK 64-Bit Server VM 1.7.0-u2-b21) [darwin-amd64-java]