bgror

Engine Yard Expands IaaS Offerings with HP Cloud Services »

Created at: 10.05.2012 15:01, source: Engine Yard Blog, tagged: Engine Yard Cloud News Partners

We’re excited to announce that we’re expanding the infrastructure options available to our customers with support for HP Cloud Services. We empower developers by providing a rock-solid platform with choices of infrastructure and components that make their job of building great applications as easy as possible. Engine Yard is one of the first PaaS providers to add support for HP’s public cloud, which is based on their world-class hardware and software, using key elements of the HP Converged Infrastructure combined with OpenStack technology.

For the past six years, our customers have relied on the Engine Yard platform to enable them to innovate faster, with higher reliability and while maintaining control of their environment. We continue to invest deeply in our open source PaaS to provide value for our customers. We want to ensure developers can rapidly build and iterate their applications while using Engine Yard Cloud to provide the on-demand scalability and reliability they need as their businesses grow and succeed.

By combining our leading commercial-grade open PaaS with HP's public cloud infrastructure offerings, we’re providing development teams a powerful new solution to rapidly deploy both large and small applications in the cloud. Engine Yard has deep roots in open source, and we continue to champion open computing by adding new IaaS options like HP’s public cloud.

We’ll be announcing more details about availability. To get updates, click here.

more »

Building Structured API Clients with API Smith »

Created at: 28.06.2011 21:31, source: Engine Yard Blog, tagged: Partners Technology Tips & Tricks apismith

Whilst prototyping the early stages of a new app with Filter Squad, we found ourselves prototyping a lot of API clients for new versions of APIs that were lacking up to date clients or in the case of other APIs, were missing functionality we required. As our starting tool set, we started with a fairly standard / typical API toolset - HTTParty for the HTTP portion and Hashie for the basic rapid prototyping data structures. As we prototyped more and more APIs, we started to see a clear pattern in how these two portions were interfacing and we started to see ourselves reinventing the same portions. To this end, Steve Webb and I wrote a library called 'API Smith' - a simple tool set combining HTTParty and Hashie in a manner that we found was incredibly fast and powerful to prototype with. By combining several simple concepts together, API Smith aims to fulfill three basic requirements:

1. Converting API responses into Ruby objects should be a straight forward and simple process.
2. Extracting parts of a response shouldn't be complex.
3. HTTP is your friend and should be simple.

Based on these three things, today we're going to briefly show how to build part of an API client for Clicky, a fairly robust analytics suite. For this cut down example, we're going to model the tallies methods as noted in the Clicky API docs. Before we do anything, you'll want to first get api_smith. This is a simple matter of running:

gem install api_smith

Step 1. Declaring your data types

First, to get a better feel for the data we're working with, we want to model the output response. From the Clicky docs, under the "Responses" section we see how the Clicky API will return the data to us In the API Smith side portion, we're lucky because API Smith's mapping objects - APISmith::Smash (for "Smarter Hash") are fairly easy to map to the incoming data. In this example, we can simply subclass the API Smith class and then use the property class method to declare what our possible fields are:

require 'rubygems'
require 'api_smith'

SMART_NUMBER_TRANSFORMER = lambda { |v| v =~ /^\d+$/ ? Integer(v) : v }

class TallyStat < APISmith::Smash
  property :title
  property :value,         :transformer => SMART_NUMBER_TRANSFORMER
  property :value_percent, :transformer => :to_f
  property :url
  property :clicky_url
  # Goal Information
  property :incompleted, :transformer => :to_i
  property :conversion,  :transformer => :to_f
  property :revenue
  property :cost
end

If you've worked with Hashie::Dash before, this will look reasonably familiar. The primary change that API Smith introduces that underpins most of the application design is that of 'transformers' - Namely, in this context a transformer is simply an object that responds to the call method and returns 'transformed' data. In the example code above, you can see that we provide the :transformer option on the property method. In the :value field, we pass it a lambda (the basic form of a call-able object) and in the next three we pass it a symbol representing a method name. Under the hood, API Smith will convert the symbol in to a proc and will apply it only when a value is given for said property. To test it out, fire up IRB, paste in the code and then run:

p TallyStat.new(:value => '1000', :clicky_url => 'http://example.com'

And you should see the output of our object. Likewise, as a short cut out APISmith::Smash subclasses also define call as a class method, making it possible to pass a subclass (e.g. Our TallyStat class) to another object as a smart transformer, e.g.:

property :tallies, :transformer => TallyStat

Along with this, APISmith::Smash is also smart enough to convert arrays and the like when called as a transformer.

Step 2. Writing the HTTP Client

The next step is to write a HTTP client by using the APISmith::Client mixin. Much like the normal HTTParty mixin, this gives us handy methods on the class like get and post, but in the case of API Smith we're also given the option to configure endpoints (separate from the base_uri), a fairly simple hierarchical way of declaring query and body parameters as well as specifying request options for HTTParty. Secondly, it also adds tools to unpack and transform data. Similar to the :transformer method on the property class method for APISmith::Smash subclasses, it also has a :transform option (With similar rules - Essentially, anything that has a #call method will used to take the raw response and convert it into a usable object. In addition to this, it provides a :response_container method that accepts an array of keys and will extract the data from the response before passing it to the transformer. To get started, add the following below your code from the original code:

class ClickyClient
  include APISmith::Client

  class TallyStatCollection < APISmith::Smash
    property :type
    property :date
    property :dates, :transformer => lambda { |c| c.map { |v| TallyStat.call(v['items']) }.flatten }
  end

  TALLY_METHODS = %w(visitors visitors-unique actions actions-average time-average time-average-pretty bounce-rate visitors-online feedburner-statistics)

  class Error < StandardError; end

  base_uri 'http://api.getclicky.com/'
  endpoint 'api/stats/4'

  attr_reader :site_id, :site_key

  def initialize(site_id, site_key)
    @site_key = site_key
    @site_id  = site_id
    add_query_options! :site_id => site_id, :sitekey => site_key
  end

  TALLY_METHODS.each do |m|
    define_method m.tr('-', '_') do |*args|
      api_tally_call m, *args
    end
  end

  private

  def check_response_errors(response)
    if response.first.is_a?(Hash) and (error = response.first['error'])
      raise Error.new(error)
    end
  end

  def base_query_options
    {:output => 'json'}
  end

  def api_tally_call(type, options = {})
    get '/', :extra_query => {:type => type}.merge(options),
      :response_container => [0], :transform => TallyStatCollection
  end

end

This will declare a client that not only accepts a site id and key as parameters (e.g. client = ClickyClient.new(1234, 'your-site-key')) but that will also:

• Correctly unpack errors (e.g. if an invalid api response is returned, it will raise a ClickyClent::Error exception).
• It will pass through any parameters in the query string
• It will automatically return a collection of tally stats, grouped by a date.

Whilst it's a pretty basic example, if you try it out with some real credentials (hint: see the clicky api docks for an example site id and site key), you'll see it works as expected:

c = ClickyClient.new('siteid', 'sitekey')
p c.visitors

Which should print out something similar to:

<#ClickyClient::TallyStatCollection dates=[<#TallyStat value=129>] type="visitors">

Conclusion

As you've seen in this rapid fire tutorial, we've managed to built a fairly simple client for a portion of the Clicky API. More importantly, even though it's still reasonably brittle and missing support for many options in the Clicky API, in a relatively short time we were able to not only pull back data from API calls but transform them into structured objects we can further manipulate. If you wish to take it further, there are a few more things you can implement to extend it into a more complete Clicky API client:

• Support for converting option types (e.g. date ranges)
• Better support for extracting data (e.g. getting daily data over a period of X days versus just the current day)
• Support for more API methods
• Actual proper tests

I've put a very rough version I wrote on GitHub here so you can take a look, but I suggest having a play personally - maybe there is a small app you use that has an API for which you could build a client using API Smith.

more »

Engine Yard Cloud Out Loud S01E06: New Relic RPM Bronze is FREE!! »

Created at: 21.01.2011 19:37, source: Engine Yard Blog, tagged: News Partners cloud out loud podcast

We at Engine Yard could not be more stoked about our partnership with our good friends at New Relic. If you have a Ruby application then you should already be using New Relic RPM. Now that New Relic RPM Bronze is offered for free to all Engine Yard AppCloud customers, you really don't have an excuse for not using it. Tune in for our interview with New Relic Founder and CEO Lew Cirne—his insights on production visibility and the entrepreneurial spirit are not to be missed. Fun fact: Lew Cirne. New Relic. Anagram.

more »

Ruby Performance Testing Tips and Tricks »

Created at: 13.01.2011 01:31, source: Engine Yard Blog, tagged: Partners Technology tips

Use em_proxy to load-test code changes

by Jason Morrison Running a high-traffic site with little room for downtime could make deploying large changes daunting. What if your new code has a large impact on performance? We’ve been in this situation several times with Hoptoad as we make large changes and improvements to the codebase. Things like changing databases, upgrading major versions of Rails, and adding queueing can have unpredictable effects on performance. Little things can have a big effect, too, when most of your traffic is focused in one write-heavy API endpoint. We’ve boosted our confidence in rolling out these changes by performance testing ahead of time. We’ve tried a variety of synthetic load testing approaches, but the most realistic way to do this is by using real traffic. We use Ilya Grigorik’s em-proxy to fork traffic in real time from our production environment to a load testing environment which is identical save for the new code. (See the duplex.rb example in em-proxy). We use Engine Yard’s “Clone Environment” feature to make a reasonably up-to-date copy of production, and we use a Chef recipe to toggle em-proxy on and off. We use New Relic RPM to keep an eye on performance, and make sure that the load testing environment performance is acceptable. Once we’ve run a few hours to a day’s worth of traffic, we’ll shut down the load testing environment, and deploy the change live.

Automatic cucumber step generation with factory girl

by Dan Croak Did you know that Factory Girl includes some steps for your integration testing pleasure? They are currently available but remain relatively unknown. Let's assume you've defined your factories normally in test/factories.rb or spec/factories.rb:

Factory.define :user do |user|
  user.email                 { Factory.next(:email) }
  user.password              { "password"   }
  user.password_confirmation { "password"   }
end

Factory.define :author, :parent => :user do |author|
  author.after_create { |a| Factory(:article, :author => a) }
end

Factory.define :recruiter, :parent => :user do |recruiter|
  recruiter.is_recruiter { true }
end

Once those are in place, and assuming you've otherwise loaded factory girl correctly, add this to features/support/env.rb:

require 'factory_girl/step_definitions'

Then, write Cucumber features using the simple "create record" step:

Given a user exists

...or the "create record & set one attribute" step:

Given an author exists with an email of "author@example.com"

...or the "create record & set multiple attributes" step:

Given the following recruiter exists:
| email            | phone number | employer name |
| bill@example.com | 1234567890   | thoughtbot    |

These steps will be available for all your factories, so stop writing boilerplate steps and shake what Factory Girl gave you.

(Testbot * Fog) + Hudson = Faster Tests!

by Nick Quaranto Test suites can get slow, even when running on one high powered machine. So why not spin up EC2 instances and distribute the tests? This is made really easy with testbot, a distributed test runner. It works like so:

1. A requester kicks off the process, asking for tests to be run
2. The server determines how many tests to run and how they will be distributed
3. Runners on your army of EC2 instances execute the tests given to them
4. THE KICK! (back up to the server)
5. BAWWWWWWWW (results get returned to the requester)

Visually, it looks like: (thanks to the testbot readme)

Requester -- (files to run) --> Server -- (files to run) --> (many-)Runner(s)
^                               |    ^                                      |
|-------------------------------|    |--------------------------------------|
             (results)                            (results)

In our situation, the requester is Hudson (CI server project), which gets kicked off when we commit to GitHub. Testbot not only supports multiple machines but also multiple cores, so our tests run on 16 medium instances in 32 different processes. We start up the boxes on EC2 using a fork of cloud_bot, which uses fog to quickly spin up instances and runs a simple bash script to install what we need to run the app's tests. The result: Instead of a 60+ minute test suite, the tests run in around 10 minutes. If your tests need a major speed boost, definitely look into testbot.

Hoptoad deploy tracking resolves errors

by Matt Jankowski This has been around for quite some time, but we still get a lot of questions about it. People who use hoptoad, our web-based error tracking application, often will ask how they “resolve all errors” in their account. Typically, they’ve been running production code for a while and have accumulated a lot of “old” errors that they want to stop paying attention to. They’d like to focus instead on errors that are happening NOW and not be distracted by things they might have fixed. There are two ways to resolve all, and both are connected to deployment. If you have an account that supports this, and you use capistrano, then this feature “just works”. When you do a deployment using capistrano, you’ll see output towards the end of the deploy indicating a request to hoptoad has been made which tells it about the deploy. The next time you access your hoptoad account, you’ll see that all previous errors have been marked as resolved, and will only be reopened if they keep coming in. If you don't use capistrano, you can get the same effect by running a rake task. Running “rake hoptoad:deploy TO=production” (where production is the name of whatever environment you’d like to resolve all errors for) will have the same effect. We've found that on hosting platforms where you can't use capistrano, you can usually find some sort of post deploy hook to help automate this step as well. We’ve resisted adding this functionality to the web interface, because in our opinion there is never a scenario where someone should want to resolve old errors that doesn’t also coincide with a deployment.

more »

Leveraging iPads for Data Collection »

Created at: 13.01.2011 01:15, source: Engine Yard Blog, tagged: Partners Technology

Imagine this scenario

You’ve just won a contract to build a form heavy app for a medical practice with reams of paper forms to convert.  There’s a patient evaluation form, a treatment form, an insurer form and a dizzying array of other information they need to gather. They know the iPad will give them instant credibility with their patients.  And they are expecting you to be the magic working developer who can make it happen. Getting started is easy but if this was you, you’d hit a wall really fast. You’d be faced with a sticky ‘trilemma’:

• You could hand code each form - but that will take forever
• You could create the forms in Interface Builder – but that will make changes really difficult and it will still take forever
• You could implement a form layout framework for iOS – but that won’t be portable and will take forever too

Any time something “takes forever”, it’s bad for the client’s budget and that’s bad for you. Instead of thinking “if only I could use HTML forms” and figuring out how to get out of your newly won deal, you should consider a fourth idea that might not be as obvious.

A new idea

Mobile and web - two technologies that were made for each other like peanut butter and chocolate.  They both have strengths but there’s an underlying weakness in each too:

• iOS is great at enabling simple and usable apps. Try doing an app heavy on data entry, though, and you’ll face some steep technical hurdles.
• Web apps are extremely flexible, portable and make form building a cinch. That said, getting the kind of flair and snappiness a client wants can be a challenge.

What’s a magic working developer to do? In this post, I’ll discuss a technique I’m using at Terralien to construct a hybrid application – one built to make the iPad an excellent data entry device without losing its sleek, simple interface. While I’ll be focusing on iOS, the techniques used here should be portable across other smartphone platforms – that’s actually a big part of the appeal with this approach.

General Approach

The approach I use is fairly simple: use a web app for forms, and a combination of the native app and a RESTfulAPI for the rest. The iOS app is effectively partitioned along the two UI lines of implementation, native UI interaction and form rendering and interaction.

Structure of the iOS App

There are three primary components involved in getting this working in an iOS app: ObjectiveResource, a special UIWebView category I call PlatformIntegration, and some corresponding Javascript on the web site to enable the platform integration. Of course, there are the actual HTML forms, and native UI for the iOS app, but those are covered many other places and we won’t go in to them here.

ObjectiveResource

The role ObjectiveResource plays is to provide the native app with access to the relevant application data. How ObjectiveResource works is outside the scope of this post, but I’d encourage you to read up on it yourself: ObjectiveResource.

UIWebView+PlatformIntegration

I’ll explain more in a bit, but the purpose of this piece is to provide interaction with the content in the UIWebView, e.g. calling code on the page, receiving events, and the like. For code, see: UIWebView+PlatformIntegration snippet This is the content/server side of the equation. It provides some simple hooks to observe form changes, submit the form, etc. For code, see: platform_integration.js snippet

Interacting with the Forms

Since the native app + ObjectiveResource/REST API should be fairly straightforward, I’ll focus on explaining the form interactions. In all my travails, I’ve only needed two things: to be able to execute code on the webpage, and to be able to receive events from the webpage.

Executing Code on the Page

This is fairly simple. In fact, UIWebView already provides stringByEvaluatingJavascriptString: and Android’s WebView can be beaten into submission by calling loadData with a ‘javascript:’ URI. Most of the work, then, is involved in making the code into a usable API. For example, it’s much nicer to say, for instance, [webView hideElementWithId:@"ye_form"]; than to say [webView stringByEvaluatingJavascriptString:@"$('ye_form').hide()"]; Making these simpler facade methods gets me most of the way there, but for code that is more involved, I use the platform_integration.js code I mentioned. It’s not much more than a namespaced set of functions that simplify the client side work.

Receiving Events

My client code needs to know when the form is changed, so I can add a helpful ‘*’ next to the title, and so I can prompt for confirmation should the user choose to close the form. This is a little more tricky, perhaps even kludgey, but the fundamental technique  is reliable and commonly used. The core of the technique centers around the fact that a UIWebViewDelegate is given the responsibility of determining whether a given URL should be processed as is (see UIWebViewDelegate’s webView:shouldStartLoadWithRequest:navigationType:). Our web page can emit URIs of various kinds, some ‘http:’ and some ‘platformintegration:’. If our code sees a ‘platformintegration:’ URI, it processes it itself, and leaves all other URIs to load normally. Let’s look at some details. If you take a look at the platform_integration.js snippet, you’ll notice the following line: window.location = 'platformintegration:formchanged'; This is our event. When the form change event happens, we execute that line, our UIWebView hackery receives the event, and processes it accordingly. In my case, I call: if ([delegate respondsToSelector:@selector(webViewFormChanged:)]) [delegate webViewFormChanged:self];

Closing, Limitations

You may’ve noticed that all this depends on being online the whole time. For an iPad user in an office, this is a reasonable expectation, but for a field iPad user (our friend the DA ‘enforcer’), the implementation needs to handle sketchy or nonexistent data connections. I’ve not solved this problem, since I haven’t needed to, but I imagine something combining HTML5 offline storage and a simple data caching layer may be able to do the trick. This, though, is a problem for another day. Aside from that, it provides a relatively simple, sane, and extensible bridge between your native app and the forms on your web app. Each platform does what it is best at, and you, the developer, are left with a comfortable feeling that you’re not wasting your client’s valuable time or money.

more »