bgror

Edge Rails: PATCH is the new primary HTTP method for updates »

Created at: 26.02.2012 03:54, source: Riding Rails - home, tagged: Edge Edge http patch REST

What is PATCH?

The HTTP method PUT means resource creation or replacement at some given URL.

Think files, for example. If you upload a file to S3 at some URL, you want either to create the file at that URL or replace an existing file if there's one. That is PUT.

Now let's say a web application has an Invoice model with a paid flag that indicates whether the invoice has been paid. How do you set that flag in a RESTful way? Submitting paid=1 via PUT to /invoices/:id does not conform to the semantics, because such request would not be sending a complete representation of the invoice for replacement.

With the constraints of the methods GET, POST, PUT, DELETE, the traditional answer is to define the paid flag of a given invoice to be a resource by itself. So, you define a route to be able to PUT paid=1 to /invoices/:id/paid. You have to do that because PUT does not allow partial updates to a resource.

Now let's think about ordinary edit forms in typical Ruby on Rails applications. How many times are we sending a complete representation for replacement? Not always, perhaps we could say that it is even rare in practice that you do so. For example, the conventional created_at and updated_at timestamps normally can't be set by end-users, though they are often considered to belong to the representation of resources that map to records.

PUT in addition is an idempotent method. You should be able to replay a request as many times as you want and get the same resource, something that sometimes is violated by conventional idioms for creating children resources using nested attributes while updating a parent resource.

There's nothing theoretical preventing PUT from doing partial updates, but when HTTP was being standarized the replacement semantics were already deployed.

Because of that, the PATCH method was defined in 1995 and standarized later. PATCH is a method that is not safe, nor idempotent, and allows full and partial updates and side-effects on other resources.

In practice, as you see, PATCH suits everyday web programming way better than PUT for updating resources. In Ruby on Rails it corresponds naturally to the way we use update_attributes for updating records.

Thus, PATCH is going to be the primary method for updates in Rails 4.0.

Routing

This is an important change, but we plan to do it in a way that is backwards compatible.

When a resource is declared in config/routes.rb, for example,

resources :users

the action in UsersController to update a user is still update in Rails 4.0.

PUT requests to /users/:id in Rails 4.0 get routed to update as they are today. So, if you have an API that gets real PUT requests it is going to work.

In Rails 4.0, though, the router also routes PATCH requests to /users/:id to the update action.

So, in Rails 4.0 both PUT and PATCH are routed to update.

Forms

Forms of persisted resources:

form_for @user

get "patch" in the hidden field "_method". The RFC is deliberately vague about the way to represent changes in a PATCH request. Submitting a form is perfectly valid, client and server must simply agree on the accepted ways to update a resource.

Let me emphasize that the "_method" hack is a workaround for the limitations in web browsers. As you probably know Rails routes real HTTP methods. That is, actual PUT, DELETE, and now, PATCH requests are routed to their respective actions.

General availability

PATCH requests are available in all places where the rest of the methods are available today. There is a patch macro for the routes DSL, :via understands the symbol :patch. Tests can issue PATCH requests, request objects respond to patch?, etc. Please see the original commit for details (with an important followup here).

Will my web server understand PATCH?

Yes, it should. I have personally tried Apache, nginx, Phusion Passenger, Unicorn, Thin, and WEBrick. They all understood PATCH requests out of the box.

Also, HTTP clients should be in general able to issue PATCH requests. For example in curl(1) you'd execute:

curl -d'user[name]=wadus' -X PATCH http://localhost:3000/users/1

Credits

We would like to thank David Lee for this contribution and endless patience to keep interested in this even after months of discussion.

Also I would like to highlight the quality of the patch itself. It is excellent: code, tests, all docs revised, comments in code revised. Everything carefully and thoroughly updated. An exemplar patch.

Web-VPN: Secure Proxies with SPDY & Chrome »

Created at: 01.12.2011 19:33, source: igvita.com, tagged: Architecture http spdy ssl vpn

Amazon's recent launch of their new Kindle Fire device delivered an interesting surprise: their Silk browser routes all requests via an Amazon powered SPDY proxy! In principle, this is not new; Opera, Blackberry and many other mobile providers have been using similar approaches to "optimize" the browsing experience for some time. However, Silk's announcement immediately generated a slew of security outcries: doesn't this mean that Amazon is now effectively a man in the middle (MITM) for all of our sessions, including SSL? Is this all just a big data mining ploy, or worse?

Turns out, the answer is: none of the above. Amazon does not route SSL traffic through their SPDY proxies, instead all HTTPS requests are routed directly to the destination. However, this incident did surface an interesting underlying question: is it, in fact, possible for a web browser to securely route SSL sessions via a web-proxy?

Tunneling SSL over HTTP

First, it is worth recalling that HTTP does allow us to tunnel SSL connections, end-to-end, via an HTTP proxy. This is a little known protocol feature, but an important one for our purposes:

To proxy an HTTPS session the client connects to the proxy by sending a special "CONNECT" HTTP method, which also includes just the host and the port of the destination. The proxy authenticates the request, completes the TCP handshake with the specified destination and returns a "200 Connection Established" response, after which, it simply becomes a dumb, two-way router. At this point, the client and the server negotiate the SSL session and start to exchange encrypted data directly through the proxy. In other words, the proxy only sees encrypted data, and the only thing it knows are the hostnames and the ports of both parties.

This is great, but there is still one problem: the original CONNECT handshake (in green), alongside the host and the port of the destination is sent in clear text. Even worse, the proxy authentication (if any) is also running over the same insecure channel. Unfortunately, browsers today do not support secure HTTP proxies! Hence the reason why we've historically resorted to heavyweight SSL VPN's, SOCKS tunnels, and the like.

Chrome, SPDY and Secure Web Proxies

Except, as it turns out, what we said above is not entirely true: earlier this year, Google Chrome added support for secure web proxies! Since all SPDY sessions run over SSL, a SPDY proxy, by definition, would need to be SSL capable. Hence, the Chrome team went ahead and added secure web proxy support - nice.

Browsers today do not support secure proxies. Although proxies can tunnel SSL, connectivity to the proxy itself is only over HTTP. To support SPDY, we need to modify Chromium to support a SSL-based proxy.

Web-VPN & Chrome

Enabling SSL for the first hop to the proxy seems like a minor change - and it did go largely unnoticed - but it opens up some really interesting use cases! For one, the authentication can now be handled at a certificate level, which can be provisioned, revoked, and verified securely. In other words, your Chrome browser is now also a "Web-VPN client" - except without the annoying installation, configuration, and maintenance!

How exactly would Web-VPN work? Simply deploy a secure HTTP proxy as a gateway to any private network, point your Chrome browser at it, and voila, you will be able to resolve any internal hostname or service just as if you were running a VPN client. The first hop to the proxy runs over SSL, and all subsequent tunnels as well.

SPDY: 2012 is the year!

One of the primary motivations for SPDY is to reduce the overall page loading time. However, while speed is important, the security aspects are arguably even more critical - support for secure web proxies is one of many great examples.

Interestingly, Chrome is on track to take the #2 browser spot away from Firefox before the end of the year, and at the same time, Firefox 11 will arrive on December 20th. So what? Well, one of the less often mentioned features of Firefox 11 is the built-in SPDY support! Combined, between Chrome and Firefox, this means that over 50% of all internet sessions will be SPDY capable. Mark 2012 as the beginning of the end for HTTP.

Optimizing HTTP: Keep-alive and Pipelining »

Created at: 04.10.2011 22:16, source: igvita.com, tagged: Architecture http keepalive pipelining

The last major update to the HTTP spec dates back to 1999, at which time RFC 2616 standardized HTTP 1.1 and introduced the much needed keep-alive and pipelining support. Whereas HTTP 1.0 required strict "single request per connection" model, HTTP 1.1 reversed this behavior: by default, an HTTP 1.1 client and server keep the connection open, unless the client indicates otherwise (via Connection: close header).

Why bother? Setting up a TCP connection is very expensive! Even in an optimized case, a full one-way route between the client and server can take 10-50ms. Now multiply that three times to complete the TCP handshake, and we're already looking at a 150ms ceiling! Keep-alive allows us to reuse the same connection between different requests and amortize this cost.

The only problem is, more often than not, as developers we tend to forget this. Take a look at your own code, how often do you reuse an HTTP connection? Same problem is found in most API wrappers, and even standard HTTP libraries of most languages, which disable keepalive by default.

HTTP Pipelining

The good news is, keep-alive is supported by all modern browsers and mostly works out of the box. Unfortunately, support for pipelining is in a much worse off shape: no browsers support it officially, and few developers ever think about it. Which is unfortunate, because it can yield significant performance benefits!

While keep-alive helps us amortize the cost of creating a TCP connection, pipelining allows us to break the strict "send a request, wait for response" model. Instead, we can dispatch multiple requests, in parallel, over the same connection, without waiting for a response in serial fashion. This may seem like a minor optimization at first, but let's consider the following scenario: request 1 and request 2 are pipelined, request 1 takes 1.5s to render on the server, whereas request 2 takes 1s. What is the total runtime?

Of course, the answer depends on the amount of data sent back, but the lower bound is actually 1.5s! Because the requests are pipelined, both request 1 and request 2 can be processed by the server in parallel. Hence, request 2 completes before request 1, but is sent immediately after request 1 is complete. Fewer connections, faster response times - makes you wonder why nobody advertises that their API supports HTTP pipelining?

HTTP Keep-alive & Pipelining in Ruby

Unfortunately, many standard HTTP libraries revert to HTTP 1.0: one connection, one request. Ruby's own net/http uses a little known behavior where by default an "Connection: close" header is appended to each request, except when you're using the block form:

require 'net/http'
 
start = Time.now
Net::HTTP.start('127.0.0.1', 9000) do |http|
  r1 = http.get "/?delay=1.5"
  r2 = http.get "/?delay=1.0"
 
  p Time.now - start # => 2.5 - doh! keepalive, but no pipelining
end
 

With the example above we get the benefits of HTTP keep-alive, but unfortunately net/http offers no support for pipelining. To enable that, you'll have to use a net-http-pipeline, which is a standalone library:

require 'net/http/pipeline'
 
start = Time.now
Net::HTTP.start 'localhost', 9000 do |http|
  http.pipelining = true
 
  reqs = []
  reqs << Net::HTTP::Get.new('/?delay=1.5')
  reqs << Net::HTTP::Get.new('/?delay=1.0')
 
  http.pipeline reqs do |res|
    puts res.code
    puts res.body[0..60].inspect
  end
 
  p Time.now - start # => 1.5 - keep-alive + pipelining!
end
 

EM-HTTP & Goliath: Keep-alive + Pipelining

While pipelining is disabled in most browsers, due to many issues related to proxies and caches, it is nonetheless a useful optimization for your own, or for talking to your partner API's. The good news is, Apache, Nginx, HAproxy and others support it, but the problem is that most app servers, even the ones which claim to be "HTTP 1.1", usually don't.

True keep-alive and pipelining support is one of the reasons we built both em-http-request and Goliath for our stack at PostRank. A simple example in action:

require 'goliath'
 
class Echo < Goliath::API
  use Goliath::Rack::Params
  use Goliath::Rack::Validation::RequiredParam, {:key => 'delay'}
 
  def response(env)
    EM::Synchrony.sleep params['delay']
    [200, {}, params['delay']]
  end
end
 

require 'em-http-request'
 
EM.run do
  conn = EM::HttpRequest.new('http://localhost:9000/')
  start = Time.now
 
  r1 = conn.get :query => {delay: 1.5}, :keepalive => true
  r2 = conn.get :query => {delay: 1.0}
 
  r2.callback do
    p Time.now - start # =>  1.5 - keep-alive + pipelining
    EM.stop
  end
end
 

Total runtime is, you guessed it, 1.5s. If your public or private API's are built on top of HTTP, then keep-alive and pipelining are features you should be leveraging wherever you can.

Optimizing HTTP: Interrogate your code!

While we love to spend time optimizing our algorithms, or making the databases faster, we often forget the basics: setting up TCP connections is expensive, and pipeling can lead to big wins. Do use reuse HTTP connections in your code? Does your app server support pipelining? The answers are usually "no, and I'm not sure", which is something we need to change!

Life beyond HTTP 1.1: Google’s SPDY »

Created at: 07.04.2011 19:52, source: igvita.com, tagged: Architecture ruby http latency spdy

Tim Berners-Lee rightfully deserves all the credit for his early work around URI, HTTP and HTML. His efforts ultimately led to the official HTTP 1.0 (RFC 1945) specification in 1996, the HTTP 1.1 proposal (RFC 2068) in 1997 and consequently the official HTTP 1.1 spec (RFC 2616) in 1999. The web as we know it wouldn't exist without these protocols.

However, more then a decade has passed, we now have billions of users online, our web apps have grown by orders of magnitude in both size and complexity, and the HTTP specification is starting to show its age. Google's Chrome browser dropped the "http://" prefix in their location bar about a year ago, and as it turns out, not just for aesthetic reasons: if you are using Chrome, and you are using Google web services today, chances are, you are not running over HTTP! Let that sink in for a minute. More likely, your browser is using SPDY - let's dig in.

Evolution HTTP and Latency

As we focus on making our web-apps being first class citizens in our everyday workflow, latency immediately surfaces as the primarily hurdle. Unfortunately, this is also the weak spot for HTTP. The original HTTP 1.0 spec mandated that the connection to the server should be closed after every single request/response cycle: no connection reuse, and an implicit TCP setup and teardown between each request - an expensive pattern.

Some of the major changes in HTTP 1.1 were to address exactly this issue: RFC 2616 defaulted to "keep alive" connections and introduced "pipelining". Keep-alive allows us to reuse the same underlying TCP connection between requests, and pipelining promised us to remove the strict "one at a time" requirement. Both sensible changes, but unfortunately, while keep-alive has found its way into most clients and servers, pipelining is disabled in all but one browser (Opera) to this day.

However, even if we manage to deploy pipelining support, there is still an unresolved issue: HTTP forces strict FIFO semantics for all the requests. Have a slow dynamic request at the front of the queue? Everyone else sharing that TCP channel will have to wait until it completes.

SPDY: Life beyond HTTP 1.1

SPDY (SPeeDY) is a Google research project and an application-level protocol for transporting web content. The primary goal of SPDY is to reduce latency, which is a promise on which it delivers: head to head tests against HTTP show SPDY with up to 64% reduction in page load times! Same pages, same TCP pipe, different protocol - nice win.

High level summary of SPDY changes and features: true request pipelining without FIFO restrictions, message framing mechanism to simplify client and server development, mandatory compression (including headers), priority scheduling, and even bi-directional communication! Let's dig into these a bit more.

Unlike HTTP, each request in SPDY is assigned a "stream ID", which allows us to use a single TCP channel to send data between the client and the server in parallel: we can serve multiple resources at the same time through the same channel by simply identifying which stream the data belongs to. To support this, SPDY defines a simple to parse binary protocol with two types of "frames": control, and data.

Turns out, while HTTP payloads can be compressed, the HTTP headers always currently travel as plain-text over the wire. SPDY compresses all header meta data with a predefined dictionary. Likewise, since SPDY now supports true pipelining, it also allows us to assign request priorities to each resource (ex: HTML first, JS second). Finally, why have the browser initiate all the requests? After all, if the server knows which page you are fetching, then it can "hint" to the client which images it may need before the client even parses that HTML - server push!

SPDY in the wild

Turns out, while the original work and tests around SPDY were done in the Chromium project, since then, the official Google Chrome client has shipped with built-in SPDY support, and not surprisingly, Google's servers are also SPDY enabled. In other words, if you use Chrome, and you're using Google services, then many of those pages are not arriving to you over HTTP - you are actually running over SPDY!

Curious to try SPDY in your own stack? Check out the available Apache plugin (mod-spdy), the Python server/client implementation, or the spdy ruby gem. The Ruby parser is a work in progress, but it does provide a way to parse the incoming data, and will also help you craft a server response for your SPDY client (see the hello world example):

require 'spdy'
 
s = SPDY::Parser.new
 
s.on_headers_complete { |stream_id, associated_stream, priority, headers| ... }
s.on_body             { |stream_id, data| ... }
s.on_message_complete { |stream_id| ... }
 
s << recieved_data
 

spdy.git (Ruby SPDY protocol parser)

Downloads: 506 File Size: 0.0 KB

The road to HTTP 2.0, or beyond!

Given all the benefits that SPDY offers, it is surprising how little attention it has received since its release. Google has also been fairly quiet about it since the initial announcement - low latency web services are a big competitive advantage, perhaps they have a reason to be quiet? Regardless, it is an open protocol, and one worth investigating.

HTTParty goes foreign »

Created at: 16.03.2009 09:05, source: Robby on Rails, tagged: Ruby on Rails ruby programming google translation api httparty http REST language

Just a quick post to get share something I was tinkering with this evening.

I came across this post by Gerald Bauer, which shows you how to use the Google Translation API with Ruby via Net::HTTP. I thought I’d play with the service with HTTParty.

class GoogleApi
  include HTTParty
  base_uri 'ajax.googleapis.com'

  def self.translate(string="", to="", from="en")
    get("/ajax/services/language/translate", :query => {:langpair => "#{from}|#{to}", :q => string, :v => 1.0})
  end
end

A few examples from playing with it.

>> GoogleApi.translate('bonjour', 'en', 'fr')
=> "{\"responseData\": {\"translatedText\":\"hello\"}, \"responseDetails\": null, \"responseStatus\": 200}"

>> GoogleApi.translate('Red wine', 'fr')
=> "{\"responseData\": {\"translatedText\":\"Vin rouge\",\"detectedSourceLanguage\":\"en\"}, \"responseDetails\": null, \"responseStatus\": 200}"

>> GoogleApi.translate('Where is the bathroom?', 'es')
=> "{\"responseData\": {\"translatedText\":\"\302\277D\303\263nde est\303\241 el ba\303\261o?\",\"detectedSourceLanguage\":\"en\"}, \"responseDetails\": null, \"responseStatus\": 200}"

>> GoogleApi.translate('Good morning', 'it')
=> "{\"responseData\": {\"translatedText\":\"Buon giorno\",\"detectedSourceLanguage\":\"en\"}, \"responseDetails\": null, \"responseStatus\": 200}"

What a party!

>> GoogleApi.translate('party', 'it')
=> "{\"responseData\": {\"translatedText\":\"festa\",\"detectedSourceLanguage\":\"en\"}, \"responseDetails\": null, \"responseStatus\": 200}"
>> GoogleApi.translate('party', 'es')
=> "{\"responseData\": {\"translatedText\":\"fiesta\",\"detectedSourceLanguage\":\"en\"}, \"responseDetails\": null, \"responseStatus\": 200}"

Look how easy that was. :-)

For a previous post on using this gem, read The HTTParty has just begun.