bgror

Introducing Bloggy: A simple way to add a Jekyll blog to any Rails application »

Created at: 03.05.2012 20:22, source: Engine Yard Blog, tagged: Engine Yard Cloud Open Source ruby Technology

One of the most popular tools our customers use to help drive traffic to their site is a company blog. Blogs that are informative and helpful will attract your target audience and bring attention to your website, and more importantly, your work.

Here at Engine Yard, we have found that some of our best sources of traffic are the blog posts written by our team that help educate customers on our product.

In particular, there is one blogging platform that has really started to catch on with the hacker community and especially those of us who love Ruby.

Authored by Tom Preston-Warner, Jekyll is an open-source blog-aware static site generator. Unlike Wordpress, Tumblr, Posterous, Blogger and the like, it allows users to write posts in the editor of their choice with the markup they prefer and then commit the posts to git. I use Markdown to write my posts, but many others choose to write their posts in HTML because they were coming from WordPress and it felt more at home.

Currently, there are lots of posts out there detailing how to get started with Jekyll, how to run Jekyll as a Rack application, etc. What I want to talk about is just a little bit different from these posts.

Let's start with a simple concept. For SEO purposes it is better if your blog runs at http://mydomain.com/blog rather than http://blog.mydomain.com. We can go into detail here later but that's really for an entirely different post. Again there are numerous ways of accomplishing this task but I want to talk about doing it with Jekyll.

Finding a way to run your blogging platform within your existing application without many changes to the server configuration is a common cost cutting technique for bootstrapping startups, well-funded companies and even public corporations. One thing we know first hand is how time consuming and difficult that has proven to be for many of you over the years.

That’s why I wrote Bloggy. Simply put, Bloggy makes it easy to run a Jekyll blog right within your existing Rails application with no changes to your current configuration on Engine Yard.

Installing and configuring the Bloggy gem

Start by adding

gem ‘bloggy’

to your Gemfile in your repo, then just run

$ bundle

and it should be ready to go for you. Alternatively, you could install it by running

$ gem install bloggy

Once Bloggy is installed, all you need to do to get a working blog up and running is to use the provided Rails generator.

$ rails g jekyll:blog blog

BAM! This just generated your new blog and it’s live at http://yourdomain.com/blog

Go ahead start up your Rails server and check it out.

What Just Happened?

The static HTML generated by Jekyll (the magic behind Bloggy) goes to your public/blog directory, but the rest of the files live at config/jekyll and this is where you will create new posts, change the default look and feel of your blog and make any configuration changes you desire as you get acquainted with Jekyll.

If you’ve never used Jekyll before, you need to familiarize yourself with a few things.

1. Bloggy Generates static HTML from the markdown files that live inside the config/jekyll/_posts directory, so this is where you will create new posts (more on how to make the generation happen later).

2. The default layout of the Bloggy generated blog is found at config/jekyll/_layouts/default.html. This file is plain ole HTML and can be edited to your liking just like you would edit any other HTML file.

3. The config/jekyll/css directory contains, wait for it … your CSS files!

Configuration and new posts

If however, you are familiar with Jekyll already you will be delighted to know you can still choose from all of the options you enjoyed with the Jekyll gem previously.

The same familiar config elements of Jekyll can be accessed using the _config.yml file that is now neatly tucked away in the config/jekyll directory of your Rails application.

Now that your new blog is installed and serving pages at http://yourdomain.com/blog you probably are thinking:

“Zach this is great, but how do I write a new blog post? How do I get rid of the test post you provided me?”

Those are great questions. I’ll start with the simplest:

1. To get rid of the generated test post, simply delete the file from your config/jekyll/_posts.

$ rm appname/config/jekyll/_posts/2012-04-25-a-test-post.markdown

2. And to create a new post it’s not much more complicated. Just run the Rake task provided with Bloggy, which will automatically generate a post and open it so you can start writing right away.

$ rake np your_post_title

This will by default open up your new post in TextMate. If you don’t have TextMate or prefer another editor you can just change mate on this line at the end of the new_post.rake file located in your appname/lib/tasks directory.

‘mate #{path}’  - for example if you wanted to use vim you should use  ‘vim #{path}’

Now, if you’re looking carefully at the created post you will notice that your post was named with a slightly different scheme than just your title. This is critical to Jekyll being able to recognize and generate your posts into static HTML files that your application can serve, so please do not change this. For example if you ran that task today your post would look something like this:

2012-05-03-your-post-title.markdown

Feel free to change the date just make sure you leave it in the correct format, but keep in mind that YAML is required at the top of your post (including the dashes) for Jekyll to generate Metadata when generating the final HTML that you will see your posts rendered in. So the top of your file should always look something like this:

---
layout: post
published: false
title: A Test Post
---

The Rake task provided takes care of that for you but be sure to edit your title and add any other metadata you want included. Other than that, just be sure to change published: false to published: true so Jekyll knows your post is ready to go live on your blog for the world to see. Once you have written, edited and reviewed to your liking all you need to do is another Rake task.

$ rake generate

From there Jekyll will automatically regenerate your posts as static HTML files stored in the public/blog directory and these files will be served from http://yourdomain.com/blog with no additonal Nginx configuration.

You will have to hit CTRL + C on your keyboard after the files are generated to stop the Jekyll server as it wants to stay on and continually look for new posts to generate.

Now from your application directory you can just run a few commands to make sure you have added your changes to git and pushed them to your repo.

$ git add .

$ git commit -m ‘your commit message’

$ git push

Then deploy to Engine Yard using our awesome CLI tool! by running

$ ey deploy

That’s it! It’s really that simple.

So go check out Bloggy and be sure to fork and contribute if you want to see something added to Bloggy that would help you even more!

I have a sample application for you guys to play around with at our Engine Yard GitHub account.

Alternatively, you can just checking it out in action by clicking here.

Happy blogging!

OSS Grant Roundup: JRuby’s New Intermediate Representation »

Created at: 28.04.2012 20:28, source: Engine Yard Blog, tagged: Open Source

Between December 2011 and March 2012, Engine Yard provided me with a 3-month grant to put in focused time on JRuby's new runtime system based on a new intermediate representation, which will be driving performance optimizations in JRuby in the coming years. In this blog post, I will outline the state of this project so far and what lies ahead. While I am the one writing this blog post, a number of ideas here have come out of discussions with Tom and Charlie.

I started work on this project in 2009 (Charlie roped me in when I thought compilers were behind me) and did a first pass on building an intermediate representation (IR) from the AST, and outlines of a couple optimizations. Through 2010, Tom got the first version of an IR-based interpreter implemented. In 2011, Tom and I ironed out a lot of the bugs in the IR, refactored and cleaned up code. But, thanks to the grant, I was able to work a little more intensively with Tom to debug the IR in both 1.8 and 1.9 modes (using the new interpreter as a vehicle for this), with and without some preliminary optimizations in place. As of now, RubySpec runs are "green" (more about the air-quotes later), MRI tests are "green", Rails boots up and runs (yet to be tested extensively), gem functionality works, and IRB runs.

This project will be part of the 1.7 release in experimental form. We hope to have some version of the JIT (even if not functionally complete) as part of the release as well. But, with a default functionally complete and correct interpreter in place, we will now be ready to embark more seriously on performance work in the coming months and years. Thanks to Engine Yard for the grant and kudos for putting money into this project where the returns are not going to be immediate.

But, first things first ...

Why a new IR (intermediate representation)?

• The AST captures higher level structural semantics compactly, but its non-linear tree-structure makes it harder to perform standard dataflow analyses and code transformations on it, especially in the presence of blocks, exceptions, non-local breaks, non-local returns, and retries. There is a theoretical framework for dataflow and control-flow analysis based on control flow graphs and basic blocks with instruction sequences that can be used to analyze this IR, and implement performance improving code transformations on it. This would be harder to do with an AST. Inlining transformations are easier in this IR form than with an AST.
• This IR attempts to capture “operational semantics” (to use the term somewhat loosely) by enabling primitive operations on the JRuby runtime to be specified, whereas the AST captures “structural semantics” (once again to use the term loosely). For example, the AST captures a call via different flavors of call AST nodes, but looking at it, you don’t know what happens under the covers. But the IR can operationally specify what happens on a call: arguments are prepared, method is looked up, bindings are allocated, other values are pushed on the stack, arities are checked, arguments are received. Similarly, in the AST what are Colon2 and Colon3 nodes transform to identical InheritanceSearchConst, test for successful lookup, followed by ConstMissing instructions. All of these primitive IR operations are then subject to analysis and optimization which is harder in the AST since they are not surfaced explicitly.
• It is also a lot more straightforward to compile this IR to JVM bytecode when compared to the existing JIT. In addition, it is also possible to target new backends (ex: Dalvik) fairly easily while taking advantage of all optimization passes.
• An additional benefit of this instruction-based representation is that it enables JRuby to persist the IR to disk (similar to .s output for C) across runs.
• We also expect to be able to simplify pieces of the existing JRuby runtime once we switch to IR mode. For example, the IR-based interpreter localizes all non-local break and return handling to the interpreter loop and the break and return instructions. Implementation details of the VM dont leak out into JRuby code that implements Ruby libraries, unlike the existing runtime.

So, now that you know why we are going this direction, let us look at some IR output. The figure below shows a simple recursive fibonacci function and the corresponding textual AST and IR representations. The textual output of IR is far easier to read and understand as well.

Ruby Source code:

def fib_ruby(n)
 (n < 2) ? n : fib_ruby(n - 2) + fib_ruby(n - 1)
end

JRuby AST:

ArgumentNode:fib_ruby 0
ArgsPreOneArgNode 0
  ArrayNode 0
    ArgumentNode:n 0
NewlineNode 1
  IfNode 1
    NewlineNode 1
      CallOneArgFixnumNode:< 1
        LocalVarNode:n 1
        ArrayNode 1
          FixnumNode 1
    LocalVarNode:n 1
    CallOneArgNode:+ 1
      FCallOneArgNode:fib_ruby 1
        ArrayNode 1
          CallOneArgFixnumNode:- 1
            LocalVarNode:n 1
            ArrayNode 1
              FixnumNode 1
      ArrayNode 1
        FCallOneArgNode:fib_ruby 1
          ArrayNode 1
            CallOneArgFixnumNode:- 1
              LocalVarNode:n 1
              ArrayNode 1
                FixnumNode 1

JRuby IR:

LBL_2:
LBL_3:
 	check_arity(1, 0, -1)
 	n(0:0) = recv_pre_reqd_arg(0)
 	%block(0:1) = recv_closure
 	thread_poll
 	line_num(1)
 	%v_0 = call(< , n(0:0), [2:fixnum])
 	b_false(%v_0, LBL_0)
LBL_4:
 	%v_0 = copy(n(0:0))
 	jump LBL_1
LBL_0:
 	%v_1 = call(-, n(0:0), [2:fixnum])
 	%v_2 = call(fib_ruby, %self, [%v_1])
 	%v_1 = call(-, n(0:0), [1:fixnum])
 	%v_3 = call(fib_ruby, %self, [%v_1])
 	%v_1 = call(+, %v_2, [%v_3])
 	%v_0 = copy(%v_1)
LBL_1:
 	return(%v_0)
LBL_5:

Playing around in IR mode

All this code is on JRuby master. If you want to play around with this, the
notes below should get you going:

• -X-CIR enables interpretation in IR mode. This mode has been extensively tested in both 1.8 and 1.9 modes. Please report correctness bugs if your Ruby code yields incorrect output in IR mode. Note that some incorrect uses of break that ought to yield LocalJumpErrors dont always raise these errors (hence the air-quotes earlier) -- something that will be fixed in the coming months. As far as performance goes, the IR-based interpreter is slower than the AST-based interpreter, from 30% - 3X, depending on the Ruby code.
• -Xir.debug=true enables debug mode which enables verbose output of IR instructions. In addition, just to get clean debug output, you will also want to enable -Xlogger.class=org.jruby.util.log.StandardErrorLogger
• If you are going to be running RubySpec tests, please remember to pass in -Xlaunch.inproc=true so that the tests run with -X-CIR mode in the same process. If not, new spawned processes will run with default JRuby mode (-X+C) which doesn't use IR yet. Alternatively, you can set env var JRUBY_OPTS=-X-CIR, and that should work as well.
• -X+CIR enables JIT-ting in IR mode. This is not yet functionally complete. Charlie is working on this as you are reading this. But, for starters, we are going to target Java 7 (using invokedynamic functionality). While it is not hard to add support for Java 6, we'll only add that when it becomes necessary. As this work moves along, we expect -X+CIR will perform better relative to -X+C than -X-CIR does relative to -X-C, since the IR is designed to help the JIT generate better code. But don't hold your breath -- this is all work in progress. But, I would be surprised if -X+CIR doesn’t match up with and eventually do better than -X+C.

Besides these basic flags, you can also play around with a few IR passes from the command-line. The following passes are currently operational:

• OptimizeTempVarsPass: As part of generating IR, a lot of temporary variables are generated by the IR builder (%v_0, %v_1, %v_2, %v_3 in the IR example above). This pass eliminates unused temporaries, does simple constant and copy propagation of temporaries, and heavily reuses temporaries to minimize the number of temporary variables used. This compaction reduces memory allocation in the interpreter and local variables in the JIT.
• LocalOptimizationPass: Does simple copy/constant propagation within a basic block. Nothing fancy.
• DeadCodeElimination: As the name says, it gets rid of code that doesn’t need to be executed. This pass is fairly conservative right now -- caused by the use of blocks-as-binding which means all writes to local variables would have to be preserved for the worst-case scenario. Even so, it does yield 10% or more speedup in certain cases. In future, we'll explore ways to relax these constraints. This code has been debugged and tested for correctness but this needs additional testing. Please report bugs where you run into them.
• AddLocalVarLoadStoreInstructions: In Ruby, local variables are not like locals in languages like C/Java because of closures. They reside in a heap structure (binding). The current JRuby runtime is smart enough to detect when a method doesn’t require a binding, but cannot do much more than that. This pass lets JRuby try to be smart about when to store the variable to the binding and load from it. This is experimental and not yet well-tested for performance (will be once JIT is in place). This is primarily targeted at the JIT, but in certain scenarios, benefits the interpreter as well (up to 15% speedup in certain cases). More details in a future post.

You can enable passes on the commandline via -Xir.passes flag. A few valid options:
-Xir.passes=LinearizeCFG
-Xir.passes=OptimizeTempVarsPass,LocalOptimizationsPass,LinearizeCFG
-Xir.passes=DeadCodeElimination,LinearizeCFG
-Xir.passes=DeadCodeElimination,AddLocalVarLoadStoreInstructions,LinearizeCFG

This CLI is still quite basic at this time, and a lot of functionality is missing, but it lets you mix and match passes which is especially good for testing the IR code in different ways. But, given the preliminary state of this CLI flag, you need to always add LinearizeCFG as the last pass whenever you provide an explicit passes flag, and the ordering of passes on the CLI matters as well. As long as you preserve the relative order of passes as above, you should be good.

We also have preliminary profiling and inlining (methods and blocks) in place, but it is not fully enabled on master (requires a non-committed patch at this time). In the interest of keeping length under control, I am skipping the details here since you cannot really play with it right now.

Now that we got the preliminaries out of the way, let us look at where we are at right now, and what lies in the future.

Current status

As should have been clear from the previous sections, a lot of the work so far has been to try out different things with the IR (interpret in 1.8 and 1.9 modes, compile to bytecode, run analyses and code transformations, profile, inline) with the goal to stabilize the IR and the code around it so we are increasingly confident that it accurately captures Ruby semantics in a clean way, and that we can use it for code optimizations. While we have dabbled in some performance experiments, most of these have been preliminary stabs. Once the JIT is complete, we will have a solid interpreter and JIT that are functionally correct, and we can more confidently experiment with code transformations to improve performance.

Overall, it is best to think of this project as moving towards a newer JRuby-level VM that is based on this new IR. As it stands now, we have a high degree of confidence that the IR, as it exists now, accurately captures semantics of Ruby, and that the IR-based interpreter, in both 1.8 and 1.9 modes, is as functionally correct as the default JRuby implementation. Based on this confidence, over the past few months, Tom has proceeded to do some much needed code cleanup and refactoring.

To summarize, here is where things are with this project and the pieces that are already in place:

• IR builder to transform AST to a sequence of IR instructions.
• Local peephole optimizations (post-IR construction).
  - Minimizes use of temporary variables.
  - Simple copy and constant propagation within a basic block.
• A generic abstract-interpretation based dataflow analysis framework.
• Dead code elimination (based on the dataflow framework).
• Preliminary profiling support.
  - Implements scope hotness profiling by embedding profiling on thread poll instructions placed on scope entries and loop back-edges. Thread-poll counts act as proxies for a hot scope.
• Method and block inlining transformation (with some known bugs and gaps).
• Local variable access opts (based on the dataflow framework).
• IR -> bytecode JIT is coming along.
  - For now, this targets JDK 7.
• Beginnings of work to optimize the IR stream for interpretation.
  - There is some tension between what the interpreter wants from the IR and what opt passes and JIT want from the IR. The latter want more explicit information and simpler instructions, whereas interpreter wants fewer complex instructions (to eliminate dispatching and interpretation costs). So, one experiment we started is to transform instruction chains in a basic block into an expression tree which eliminates dispatch costs, and temporary variable read/write overheads from the interpreter loop.

Future work

The first thing we want to accomplish is get the IR -> bytecode JIT (-X+CIR) functionally complete and correct.

Going forward, there are several things we want to address:

• Additional IR cleanup: There are still some oddities left in the instruction set (GetClassVarContainerModuleInstr, InstanceofInstr) that we want to get rid of. That just reflects a gap in my understanding of Ruby semantics as I worked on this.
• Improving interpreter performance: The 3X slowdown that I reported up there was to dampen your expectations :). The 3X slowdown is on micro-benchmarks like fib_recursive. “Real” programs perform much better. But, this also highlights another issue with performance testing: micro-benchmarks aside, is there a good set of benchmarks on which we can evaluate performance that is meaningful? That apart, we still have some gap to close with the AST interpreter before it can become the default interpreter.
• Expose more JRuby state in the IR so we can more explicitly analyze and manipulate it - where is a piece of state really required, and where can it be removed. For example, on a typical call, JRuby passes a block, sets up a call frame with many fields, sets up a "binding" for local variables, all of which adds to the cost of Ruby calls. Not all of this is necessary at all call sites. We have brainstormed several ideas about how to tackle this and this is something that we'll be working on. No immediate results here since some of this state winds its way through the existing runtime, and we have to carefully unwind it.
• Additional analyses/optimizations (non-exhaustive):
  - Optimize allocation of bindings: Right now, bindings/dyn-scopes are always allocated -- we will import some heuristics from the AST JIT, but more work to be done here.
  - Migrate call setup code out of JRuby runtime into the IR (split between the call site and callee entry/exit).
  - Method and block inlining: As discussed earlier, beginnings of this are in place which you can play with via the -Xir.profile=true option. This still needs work to iron out bugs and figuring out good inlining strategies which is different from the transformation (which is needed when the strategy decides scope A should be inlined into scope B).
  - Unboxing of primitives: Rather than create first-class Ruby/Java objects for fixnums, booleans, can we get away using JVM primitives instead?
  - Profiling: What is the kind of profiling we need and how do we do it effectively with low overheads?
  - VM manager: We need a manager process that orchestrates this whole thing. How much do you optimize before interpretation? When do you profile and how long? When do you JIT? When do you inline and how much do you inline? How much work should JRuby do and how much should we let the JVM deal with? Our strategy will be one where we always let the JVM do its magic and find areas where we can help it along. But, to me, it is clear that we cannot expect the JVM to optimize everything that is thrown at it
  which is where this VM will play a role.

Some of these tasks will take months and others will happen over a few years. So, lots to be done.

In Case You Missed It: February’s JRuby Meetup »

Created at: 06.03.2012 19:58, source: Engine Yard Blog, tagged: Open Source ruby

A couple of weeks ago, we combined forces with New Relic and put on another great JRuby Meetup. This time, we were on New Relic's turf. Nick Sieger, JRuby core committer and creator of JRuby-Rack spoke about what has been happening in JRuby Land in the past year and covered everything from VisualVM to Method Handles and Benchmarks.

Once again, thanks so much to our good friends at Marakana and especially to Max Walker for recording and editing the talk. We love you guys!

CFP for JRuby Conf 2012 is Now Open! »

Created at: 15.02.2012 01:11, source: Engine Yard Blog, tagged: community Open Source ruby

We’re very pleased to announce that the JRuby Conf call for papers is officially open. While we’re certainly looking for great stories about how you’re winning using JRuby, we also won’t mind a couple of proposals that fall outside the JRuby-sphere. If you feel passionate and want to speak about a project or topic that you think suits the mix of people that will be at JRubyConf, let us hear about it! We’re looking forward to a great event and can’t wait to see your proposals.

If you haven’t gotten your ticket yet, go ahead and get registered. If we accept your proposal and you already bought a ticket, let us know and we’ll refund the ticket right away.

My Summer of (Open) Source »

Created at: 05.01.2012 22:25, source: Engine Yard Blog, tagged: Open Source ruby Technology

The last few months have been an great experience for me. I’m a graduate student from Potsdam, Germany. However, as some of you might already know, I’m also rather active in the Ruby community. This past year, I had an amazing opportunity.

Engine Yard sponsors a couple of Open Source developers to work full time on their projects. When I asked Dr. Nic Williams whether they would sponsor me spending three months in Portland, working together with Brian Ford on Rubinius, I expected nothing but a no. Turns out, Engine Yard was at least as thrilled about this idea as I was. A few days ago, I finally got back to Germany, and I wanted to give you a quick overview of what I’ve been working on during my time overseas.Like many others, I started contributing to Rubinius a while ago. However, I never really dared to play with the internals. So, my first stop was the Rubinius compiler. To make sure I really understood it and that it’s as flexible as it claims to be, I wrote a Smalltalk implementation using the Rubinius compiler infrastructure and looked into improving its API.

It’s a fun thing to do, as the Rubinius compiler is written entirely in Ruby. And, since Rubinius is bootstrapped, it also runs on other Ruby implementations. That is how you usually install Rubinius: You load the compiler from CRuby, it then compiles the compiler to Rubinius bytecode. If you want to look into this, there is some excellent documentation available on the Rubinius website.

This bytecode can then be executed by the Virtual Machine, which was my next stop. It took me a while to fully understand how things work within the VM. It is actually the only major part of Rubinius not written in Ruby, and the main reason for it’s blazing performance and excellent memory footprint. I am planning ton writing another blog post, or possibly even a series of blog posts about these internals.

Apart from bug fixes and API improvements, I used the gained knowledge to fix, for instance, one of Ruby’s least known and most confusing feature: the implemented flip-flops.

The last thing I worked on was Puma, a new web server for Rails/Rack/Sinatra applications. Rubinius 2.0 is about to be released, fully able to make the best use of all your CPUs. However, most web servers used for deploying Ruby applications are actually single-threaded. Since there is no real threaded option that is still maintained and not JRuby specific, Evan Phoenix and I started working on a new server.

Like many other servers, it uses the rapid HTTP parser that comes with Mongrel. It also uses a dynamically sized thread-pool for processing requests in parallel. With Puma, you now have a go to choice when it comes to deploying web applications on Rubinius. And since it does not contain any Rubinius specific code, it also works quite well on JRuby or CRuby.

To make sure we are heading in the right direction, I started working on a tool for benchmarking web applications under realistic load. The main issue with just using ab, the standard solution for measuring HTTP performance, is that it results in unrealistic numbers both on JRuby and Rubinius. When using ab, you just send the same request over and over again, causing the JIT and code inliner to highly optimize for exactly that request. This usually doesn’t reflect the actual production behavior, though. I therefore wrote code simulating a real browser session and, of course, running multiple of these sessions in parallel.

You think that’s all? Far from it! The Engine Yard OSS Community Grant Program enabled me to speak at six different conferences all over America. At Rocky Mountain Ruby, RubyConf Brazil and RubyConf Uruguay, I gave a talk on “Real Time Rack”. In San Francisco, at GoGaRuCo, I gave a presentation about “Smalltalk On Rubinius - or How To Implement Your Own Programming Language”. At this past year’s RubyConf in New Orleans, I spoke about “Message in a Bottle” and last but not least I gave a presentation titled “Beyond Ruby” at RubyConf Argentina in Buenos Aires.