bgror

fog 1.0 is here! »

Created at: 30.09.2011 17:00, source: Engine Yard Blog, tagged: News Open Source Technology fog

Many good things in life take longer than expected but in the end are well worth the wait. With the release of fog 1.0 we have a shining example. Begun in mid-2009 and with more than 2 years in production usage at Engine Yard, fog has seen 240,000 downloads. This is definitely a rock-solid 1.0 that has seen some real battle testing!

fog is a Ruby gem that allows you to control a number of cloud infrastructure services through a common API. fog provides the basis for building applications and cloud platforms that are infrastructure-independent, abstracting such functions as spinning up servers, connecting storage, managing DNS, etc. Internally, fog uses models to represent cloud structures analogously to the way ActiveRecord represents relational structures. Using fog makes your application or platform much more easily portable across infrastructures such as Amazon, Rackspace, and many others.

In addition to Engine Yard, fog is in use at other platform providers as well as in many core community projects such as Chef, Puppet, CarrierWave, and Paperclip. fog now supports 18 compute, 5 storage, 9 DNS, 2 CDN, and a variety of other infrastructure services. With 13 core committers, 121 contributors, and nearly 1,500 watchers, we're seeing really nice growth in participation and attention.

We believe that this release of fog is a really important step in making the cloud an easier and more powerful environment for developers to run great apps. We hope you'll give it a try, share your feedback, spread the word, and (most importantly) enjoy some great success with fog.

Get fog with a simple gem install fog. Read docs at fog.io, or grab source off GitHub at geemus/fog. Happy fogging!

more »

A Gentle Introduction to CarrierWave »

Created at: 18.08.2011 22:53, source: Engine Yard Blog, tagged: Open Source Technology carrierwave fog

CarrierWave is self-described as a "classier solution for file uploads for Rails, Sinatra and other Ruby web frameworks." Although I've head it referred to as "a new kid on the block" it's actually quite an old gem. The initial checkin is from August 2008 and the first release was in March 2009. The original name was Merb::Upload and it started without support for Rails.

The fact that CarrierWave began its life as a Merb plugin may help explain its modularity, flexibility, and extensibility.

Thanks to fog, it has support for Amazon S3, Rackspace Cloud Files, and Google Storage for Developers. It also supports plain old file storage and MongoDB's GridFS store. There's ORM support available for ActiveRecord, Mongoid, DataMapper, Sequel, Mongo Mapper, CouchDB, and more. Image processors are available for RMagick, ImageScience, MiniMagick.

Let's see this modularity first hand by building up a CarrierWave uploader from scratch.

To begin with, we'll install CarrierWave:

  gem install carrierwave

Then, we can make the world's shortest uploader:

  require 'carrierwave'

  class MyUploader < CarrierWave::Uploader::Base
    storage :file
  end

Even at this point, we can start saving files:

  file = File.open('example.jpg')
  uploader = MyUploader.new
  uploader.store!(file)

The uploader houses the logic for uploading files in self-contained classes, so this is all we need to get started.

It's dead easy to store files on Amazon S3 with fog. First, install the fog gem:

  gem install fog

Then configure the fog_credentials and set the uploader's storage to fog:

  CarrierWave.configure do |config|
    config.fog_credentials = {
      :provider               => 'AWS',
      :aws_access_key_id      => 'xxx',
      :aws_secret_access_key  => 'yyy'
    }
    config.fog_directory  = 'zzz'
  end

  class MyUploader < CarrierWave::Uploader::Base
    storage :fog
  end

Configuring fog to use Rackspace Cloud Files or Google Storage for Developers is so easy I hesitate to even mention it. This is all we'd need to change to use Rackspace:

  CarrierWave.configure do |config|
    config.fog_credentials = {
      :provider           => 'Rackspace',
      :rackspace_username => 'xxx',
      :rackspace_api_key  => 'yyy'
    }
    config.fog_directory = 'zzz'
  end

Suppose we want to have image thumbnails. First, we need to install RMagick:

  gem install rmagick

Then, we just add a version block to our uploader:

  require 'carrierwave'
  require 'rmagick'

  class MyUploader < CarrierWave::Uploader::Base
    include CarrierWave::RMagick
    version :thumb do
      process :resize_to_fill => [200,200]
    end

    storage :file
  end

This has all been so easy, though. Let's give ourselves a challenge and create an app that uses CarrierWave, Sinatra, and Sequel with SQLite.

To begin, we'll need to install these new gems:

  gem install sinatra sqlite3 sequel carrierwave-sequel

Then, we'll create the app. It's so small (55 lines) that I'll just include the whole thing here:

  require 'carrierwave'
  require 'sinatra'
  require 'sqlite3'
  require 'sequel'
  require 'carrierwave/sequel'

  # database setup

  DB = Sequel.sqlite

  DB.create_table :uploads do
    String :file
  end

  # uploader

  class MyUploader < CarrierWave::Uploader::Base
    storage :file
  end

  # model

  class Upload < Sequel::Model
    mount_uploader :file, MyUploader
  end

  # sinatra app

  get '/' do
    @uploads = Upload.all
    erb :index
  end

  post '/' do
    upload = Upload.new
    upload.file = params[:image]
    upload.save
    redirect to('/')
  end

  __END__

  @@ index
  <!DOCTYPE html>
  <html>
    <body>
      <form action="/" method="post" enctype="multipart/form-data"></div>
        <p><input type="file" name="image" /></p>
        <p><input type="submit" name="submit" value="Upload" /></p>
      </form>
      <% @uploads.each do |upload| %>
        <img src="<%= upload.file.url %>" />
      <% end %>
    </body>
  </html>

The app can be run like so:

  ruby example.rb

There's a lot more I could have covered in this post, but my hope is that this brief illustration of the modularity, flexibility, and extensibility of CarrierWave will appeal to you as it did to me.

If you're ready for more, check out:

• the readme
• the wiki
• the Mocking fog when using it with CarrierWave post from Mike Gehard
• and the How to set up CarrierWave for local storage with AppCloud documentation for Engine Yard AppCloud users.

I'm sure you'll enjoy using CarrierWave as much as I have!

more »

fog 2011 Halftime Show »

Created at: 04.08.2011 01:58, source: Engine Yard Blog, tagged: Open Source Technology fog

Everything has been going stunningly for fog this year. Now that we have reached the halfway mark (and 0.10.0), it seems we are overdue for an update.

Since Last Episode

[caption id="attachment_9987" align="alignright" width="300" caption="Photo courtesy of flickr user nplove"][/caption]

Since the last update in mid-March a lot has changed. A new, stronger focus on documentation has helped clarify things alongside many, many fixes and improvements. In addition to fixing the old stuff we have also added support for a number of new services:

 
• AWS Autoscaling was added (tests)
• AWS CloudWatch was added (tests)
• AWS Simple Notification Service was added (tests)
• AWS Simple Queue Service was added (tests)
• DNSMadeEasy DNS service was added (examples)
• Linode Compute was expanded with models (tests)
• Ninefold Compute (tests)
• Ninefold Storage (tests)
• OpenStack Compute Support (utilizing Rackspace Implementation)
• OpenStack Rackspace Support (utilizing Rackspace Implementation)
• vCloud 1.0 (tests)

Stats

In addition to the codebase, growth and stability have also dominated the userbase. Since the beginning of the year there have been huge gains across the board. We started the year at 50,000 downloads and have now reached 180,000 downloads. Our 860 followers have increased to 1360, 100 forks to 225, and 40 contributors to 105.

The most important change may be the number of commiters. At the start of the year I was the sole committer. Since then I have kept a careful eye out for dedicated contributors. There are now 8 of us. These awesome commiters are: Brian Hartsock, Christopher Oliver, Dylan Egan, Henry Addison, Aaron Suggs, Lincoln Stoll, Luqman Amjad, Nick Ricketts, Nick Osborn, Paul Thornthwaite. I thank them all and encourage you to do the same (HINT: high fives are awesome).

fog 2011 World Tour

I just returned from OSCON, but there are still plenty of chances to catch up this year. I'll be speaking at Lone Star Ruby Conf V August 11-13th, Golden Gate Ruby Conference September 16-17th, and Strange Loop September 18-19th. Finally for the cherry on top of a busy conference season, I'll be attending RubyConf September 29-October 1st. So, be sure to find me to help answer fog questions, or to offer high fives.

NEXT!

The hits keep coming and I expect we will continue to see new providers appear as new competitors emerge and people become interested in trying the latest and greatest. Beyond that there is still plenty of work to be done to improve the consistency between some of the providers and consolidating the shared feature set. You can expect more great stuff that you have grown accustomed to, but what else might await?

A lot of that is in your hands. What other services would you like to see abstracted? Should there be a CLI? How about RESTFUL web service? Should we convert the shindo tests to minitest? These are just a few of the many questions I am focused on answering as we continue to move the project forward. This is a great time to get involved and cast your vote. You can find all kinds of information about getting involved on the ever expanding fog.io. Let me know what you think!

more »

Spinning Up Cloud Compute Instances »

Created at: 28.06.2011 00:33, source: Engine Yard Blog, tagged: Open Source Tips & Tricks compute fog

[caption id="" align="alignright" width="240" caption="Next-Generation Supercomputer Delivered to NERSC by Lawrence Berkeley National Laboratory, on Flickr"][/caption] Computation is the lifeblood of the cloud, providing the raw resources for innovation. Unfortunately, with great power comes great responsibility complexity. The actual process of refining these resources into services is serious business, further complicated by the huge variance between offerings. Thankfully flexibility and power no longer have to be out of reach. fog simplifies the process of utilizing these resources and smooths the differences between providers so that you can focus on creating the next revolutionary cloud service.

Installing fog

fog is distributed as a RubyGem:

gem install fog

Or for bundler users, you can add it in your Gemfile:

gem "fog"

fog => Rackspace Cloud Servers

We can start our exploration of cloud computing with Rackspace's Cloud Servers. You can sign up here and copy down your api key and username from here. We are about to get into the code samples, so be sure to fill in anything in ALL_CAPS with your own values!

# create a connection
connection = Fog::Compute.new({
  :provider           => 'Rackspace',
  :rackspace_username => RACKSPACE_USERNAME,
  :rackspace_api_key  => RACKSPACE_API_KEY
})

Servers the Rackspace way

Creating a server on Rackspace is very easy if you are willing to accept the defaults (the smallest server size, using Ubuntu 10.04 LTS).

server = connection.servers.create

You can then list your servers to see that it now appears.

connection.servers

Or you can fetch the latest data for your server in particular.

servers.get(server.identity)

As you might imagine, this is a pretty common use case, so fog simplifies it by providing the `reload` method to refresh the state of a model to the freshest available.

server.reload

But this too can get tedious quickly, especially when servers can take several minutes to boot. Fog uses `wait_for` in cases like this to periodically reload a model until either the block returns true or a timeout occurs (by default the timeout is 600 seconds). We can combine wait_for with `ready?` to check when a server has finished booting without needing to know the specifics of each service.

server.wait_for { ready? }

Once we are done with that we can shut it down.

server.destroy

Bootstrap: Servers the fog Way

Cycling servers is great, but in order to actually ssh to a server on Rackspace you need to place ssh keys (and ideally disable password authentication for root). Rather than worrying about the nitty gritty, we can utilize `bootstrap`.

server = connection.servers.bootstrap({
  :private_key_path => '~/.ssh/id_rsa',
  :public_key_path => '~/.ssh/id_rsa.pub'
})

Bootstrap will create the server, but it will also make sure that port 22 is open for traffic and has ssh keys setup. In order to get all the pieces put together the server will have to be running, so we can skip checking `ready?` since it should already be true. Now we can send commands directly to the server.

server.ssh('pwd')
server.ssh(['pwd', 'whoami'])

These return an array of results, where each has stdout, stderr and status values so you can check out what your commands accomplished. Now just shut it down to make sure you don't continue getting charged.

server.destroy

Using Amazon EC2 and fog

Sign up for an account here and copy down your secret access key and access key id from here. First, create a connection with your new account:

require 'rubygems'
require 'fog'

# create a connection
connection = Fog::Compute.new({
  :provider                 => 'AWS',
  :aws_secret_access_key    => YOUR_SECRET_ACCESS_KEY,
  :aws_access_key_id        => YOUR_SECRET_ACCESS_KEY_ID
})

With that in hand we are ready to start making EC2 calls! We should be able to reuse all our old code, except for one small exception. fog uses the official Canonical Ubuntu image as a default on AWS (official Canonical Releases). This image uses 'ubuntu' as the username, rather than 'root'. So the bootstrap call will look slightly different.

server = connection.servers.bootstrap({
  :private_key_path => '~/.ssh/id_rsa',
  :public_key_path => '~/.ssh/id_rsa.pub',
  :username => 'ubuntu'
})

Just like on Rackspace, this will boot a server and place our keys so that we can ssh in, run commands and eventually shut down the server with destroy.

Mocking out Compute

You can also start any of these scripts with

Fog.mock!

or start the fog interactive tool from the command line with

$ FOG_MOCK=true fog

to run in mock mode. In this mode commands are run as local simulation, so no cloud resources are ever consumed and things operate much faster. Not everything has mocks written for it, but if you run up against these edges errors will be raised to quickly alert you that you are entering not-yet-mocked territory. The functionality that has been mocked gets exercised by the same test suite as the real code, so you should feel confident that its behavior should be consistent.

Cleaning up

To cover your tracks it is a good idea to check for running servers and shut them down, here is one way you might do that.

connection.servers.select {|server| server.ready? && server.destroy}

Other Providers

Rackspace and Amazon are just the tip of the iceberg. fog also supports compute offerings from BlueBox, Brightbox, Terremark, GoGrid, Linode, Slicehost, Storm on Demand and Voxel. Once you wrap your head around that you can also check out fog's many supported storage, DNS and other services. With familiar commands throughout it is easy to pick up and try new services. Quickly your toolbox runneth over and you should have all the power you need to innovate.

Summary

Compute can be tricky, but the abstractions in fog make it much easier to get started. With your servers up and running you can then focus on the task at hand and get some work done. Congratulations on adding a new tool to your arsenal. Let us know what we can do better.

more »

fog 0.7.0, 0.6.0 and 0.5.0 »

Created at: 18.03.2011 18:37, source: Engine Yard Blog, tagged: Open Source fog fog 0.7.0 magicruby MountainWest RubyConf red dirt rubyconf ruby nation silicon valley ruby on rails group

I've been so busy with all the additions to fog and new contributors that I forget to update everybody on how things are going (they are great!). The last post I pushed out coincided with 0.4.0 back around the new year and now I'm back to let you know about all the new goodies that have culminated in 0.7.0.

Adoption

Adoption continues at an amazing pace. We went from 40 contributors two short months ago up to 60, crossed 80,000 downloads from rubygems.org and 1,000 followers on github [ed: Wes was very giddy on that day]. All the extra attention has helped to weed out even more bugs and grow the available services to meet ever more diverse use cases.

New Providers and Services

Once again, several new services have joined the fog family, as well as a few notable additions to the existing services:

• AWS Cloudformation basics allow for configuring whole clusters at once (tests)
• AWS IAM has been fleshed out considerably from it's humble origins in 0.4.0 (tests)
• AWS RDS basics add the ability to manage hosted database servers (tests)
• AWS S3 gained static website support (tests)
• AWS SES basics support for cloud email delivery (example)
• DNSimple DNS service was added (examples)
• VirtualBox basics support managing local virtual machines (example)
• Voxel Compute service was added (tests)

Contribution Overhaul

I am always looking for ways to make it easier for people to get involved, so let me know if you have suggestions. First off I made the standing t-shirt offer more explicit, but that didn't have too much impact. After that I decided I ought to better groom the issues so that they would be more accessible. The clearest thing is that must issues are now labeled as either easy, medium or hard. Rather than just plucking these values out of thin air I consider each issue and add notes on what needs to be done to accommodate them before adding the label. So if you would like to help out you can pick a difficulty matching the time you have available and familiarity with the code and feel confident that everything you need to complete the task will be available. You can also feel free to skip the non-categorized tickets, since I either haven't figured out how to do them or they will require my attention directly (i.e. accepting pull requests).

Integrations

In addition to working inside fog I have been working to help others better use it. Both carrierwave and paperclip have received patches with fog support. Both of these libraries provide ways to handle files for users in your web apps and by adding fog support they no longer have to worry about supporting numerous different providers and they can focus on their specialties. We will have more details about these integrations and how you can take advantage of them in future posts.

On the Road

I have also been privileged to be accepted to speak at a number of conferences and meetups. I spoke in January at Silicon Valley Ruby on Rails Group (recorded talk video here) and in February at Magic Ruby. Yesterday I presented at MountainWest RubyConf, and next month I'll be at Ruby Nation April 1-2, and Red Dirt RubyConf April 21-22. If you are out and about at these conferences come say hello and let me know how you use fog.

Toward 1.0

With all these improvements and bug fixes I can't help but feel like we must be nearing 1.0. It will be great to reach that point but I still have a couple things I want to sort out along the way. Although DNS and storage provide fairly universal access across providers, compute still has a few rough edges. One key focus will be smoothing differences between the compute providers. Another goal is to better sort out testing. There is a working set of tests now, but it is split between rspec and shindo and there are some discrepancies between style and usage throughout. I'd like to solidify how these will work, as well as making it easier for contributors to run them without have credentials for everything. Last, but certainly not least is a stronger focus on documentation. There is a bit here and a bit there currently and I'd like to consolidate things on fog.io and make them easier to explore, as well as reworking the generated in-code documentation to be more accessible. Over all I see 1.0 outside of that as being more of the same great stuff, from bug fixes to new services as things are released and the needs of users change, and beyond 1.0 will be no different. It is just a stepping stone for us to acknowledge the great level of support and stability we have already reached.

more »