bgror

Introducing the Engine Yard Data Team »

Created at: 02.09.2011 21:08, source: Engine Yard Blog, tagged: News Technology engine yard data team mongodb mysql PostgreSQL

The Engine Yard Data Team’s mission is to formulate and iterate the Engine Yard data storage strategy, to help customers understand and choose their data solutions, to support OSS projects that match our vision, and to strengthen our leadership in the Ruby on Rails community. Our team is composed of Ines Sombra and our amazing DBAs: Erik Jones, Tyler Poland, and John Dalton.

We want to ensure that when you choose Engine Yard as your platform provider you have access to interesting data stores, features, tools, and information to build robust and scalable solutions.

Our Goals

• Simplify and hide the complexities associated with data repositories.
• Choose the best-in-class technologies and publish our own best practices.
• Work closely with our customers and developers to understand their use cases and usage patterns.
• Routinely assess how closely our data products and support match our customers needs.
• Ensure Engine Yard’s data repositories are up-to-date with the rapidly evolving storage technology landscape.
• Establish partnerships with key data providers to offer our customers additional services.
• Design and support multi-site disaster recovery and business continuity plans for each type of data store by working with our customers to understand their scaling needs.

What we are currently working on

Upgrading our MySQL implementation

The team is currently working at a rapid pace to upgrade MySQL to version 5.1 and 5.5. We’ll also be improving MySQL replication, monitoring, backup, and fail-over tools.

Expanding the DB stack

We are working hard on expanding our stack to offer interesting NoSQL data stores, caches, and full-text search solutions. The products we are working on at the moment are:

PostgreSQL 9 PostgreSQL 9 is now in Alpha (click here for access). We are currently working towards a public Beta for all our customers to try. Postgres extensions will be available as part of the Beta release.

MongoDB We are working towards an Alpha release of MongoDB. We have been collaborating with customers that are interested in this technology to further develop our tools and finalize our supported architecture. We have established critical partnerships with hosted MongoDB providers (MongoHQ and MongoLab), and are working with 10gen to provide additional support options for our customers.

Redefining environments and data stores relationships

We are changing the way environments interact with data stores to provide greater flexibility and configuration options. We aim to support zero to multiple data stores in the same environment to facilitate the creation of polyglot systems in our platform.

Improving our data documentation

We are collaborating with the PANDAs and our Documentation team to restructure and augment our data documentation. Watch for changes in docs.engineyard.com and let us know if we are not covering something that you are interested in.

We want to hear from you!

We are actively evaluating customer requested features and we will keep you informed as our work grows from ideas to projects. Our intention is to create a strong collaborative process with our customer community, so let us know what you need.

We’d love to hear your feedback! Please drop us a note at: docs.engineyard.com/data-feedback.html

Want to work with us?

Are you passionate about data? We are hiring!

Caching With Mongo »

Created at: 15.07.2010 15:00, source: RailsTips - Home, tagged: mongodb gems

For those of you that do not follow me on Twitter or Github, a while back I released Bin, an ActiveSupport MongoDB cache store. Since I have been quiet here, I thought I would talk a bit about it to help get back in the swing of things.

Using Bin is just like using any other AS cache store.

connection = Mongo::Connection.new
db = connection['bin_cache']

Rails::Initializer.run do |config|
  config.cache_store = Bin::Store.new(db)
end

Once you have set things up, you can use all the typical Rails.cache methods.

Rails.cache.write('foo', 'bar')
Rails.cache.read('foo') # 'bar'

Rails.cache.fetch('foo') do
  # some expensive thing
end

The list goes on, but in the interest of brevity, I will just link you to the specs. The cool thing about bin is that it supports both ActiveSupport 2 and 3 along with Ruby 1.8.7 and 1.9.1. Oh, and it supports expires with the same API as the memcache store.

That pretty much covers the basics, feel free to go kick the tires or hang around here a bit to learn how I made Bin work with AS2 and AS3.

Supporting AS2/3

In both ActiveSupport 2 and 3, you inhert from ActiveSupport::Cache::Store to create a new store. The difference between the version is quite subtle though. In AS3, you override the methods read, write, etc. as needed and use super with a block to get the inherited functionality. In AS2, you do the same thing, but super does not accept a block. Being that as a community we are now mugwumps (mug on Rails 2 and wumps on Rails 3), I thought it would be nice to support both.

In order to make this happen, I knew all I need to do was shim compatibility for Rails 2. So what I did is create a compatibility class that inherits from ActiveSupport::Cache::Store for AS3 and then if active support’s version is less than 3, I reopen the class and add in the compatibility stuff to make it work like 3. Here is the code in its entirety:

# encoding: UTF-8
module Bin
  class Compatibility < ActiveSupport::Cache::Store
    def increment(key, amount=1)
      yield
    end

    def decrement(key, amount=1)
      yield
    end
  end

  if ActiveSupport::VERSION::STRING < '3'
    class Compatibility
      def write(key, value, options=nil, &block)
        super(key, value, options)
        yield
      end

      def read(key, options=nil, &block)
        super
        yield
      end

      def delete(key, options=nil, &block)
        super
        yield
      end

      def delete_matched(matcher, options=nil, &block)
        super
        yield
      end

      def exist?(key, options=nil, &block)
        super
        yield
      end
    end
  end
end

So then Bin::Store just inherits from Compatibility:

module Bin
  class Store < Compatibility
    # ... stuff
  end
end

I cringe using a specific version string comparison like above, but it was simple and worked so I went with it. The last piece of the puzzle was setting up rake tasks to run the specs against different active support versions.

namespace :spec do
  Spec::Rake::SpecTask.new(:all) do |t|
    t.ruby_opts << '-rubygems'
    t.verbose = true
  end

  task :as2 do
    sh 'ACTIVE_SUPPORT_VERSION="<= 2.3.8" rake spec:all'
  end

  task :as3 do
    sh 'ACTIVE_SUPPORT_VERSION=">= 3.0.0.beta3" rake spec:all'
  end
end

desc 'Runs all specs against Active Support 2 and 3'
task :spec do
  Rake::Task['spec:as2'].invoke
  Rake::Task['spec:as3'].invoke
end

Note that I make an all task to run the specs then two distinct tasks to run against AS2 and AS3. All those tasks do is set an environment variable that I use in the test to force a particular version.

gem 'activesupport', ENV['ACTIVE_SUPPORT_VERSION']

Now when I run rake, it runs the tests against a 2.3 and a 3.0+ version of ActiveSupport so I know when something goes wrong with either. No flipping gem sets or other shenanigans. As always, if you have improvements or other way so doing stuff like this, please let me know. I am here to learn people.

Using Bin on the last project I worked on to cache large fragments of the layout significantly reduced response times. Always fun to see numbers like that drop after a deploy!

Mongo Scout Plugins »

Created at: 14.07.2010 05:00, source: RailsTips - Home, tagged: mongodb scout

As I have stated before, Harmony is happily hosted with the fine folks at Railsmachine. They use and recommend Scout. While I have been quite happily using several scout plugins to keep track of Harmony, including the MongoDB Slow Query Plugin, I felt like a few things were missing. Last night and tonight, I did a bit of hacking and created a few more plugins to help round out our view into Mongo.

Mongo Stats

First, in order to learn how scout plugins work, I started with something simple: db stats. You may or may not know that you can grab stats for your Mongo databases quite easily from the ruby driver. Using this along with scout’s plugin API, I wrote a Mongo Stats plugin that reports a few things about your database:

• The number of objects
• The size of your data
• The storage size of your data
• The number of indexes
• The size of your indexes

Nothing fancy, but a good start and since keeping your indexes in RAM is important, having the index size reported continually will be quite handy in the future.

Mongo Server Status

Next, I did some digging on mongodb.org and found the server status command. I remembered that Kyle had written about how to call any Mongo command from the Ruby driver and a few minutes later, I had even more mongo information to peruse. The server status plugin keeps track of:

• btree accesses, hits, misses, resets, and miss ratio
• global lock total time, lock time, and lock ratio
• background flushes total, total ms, average ms and last ms
• memory bits, resident, virtual, and mapped (if your Mongo version is up to date)

Most of this information is not super helpful now, but down the road I know I will be happy I was logging it, especially when things go wrong (they always do).

If you are curious as to how you can call the server status command in ruby, here is the money shot:

db = Mongo::Connection.new.db('testing')
db.command('serverStatus' => 1)

It returns a hash with all kinds of information. Obviously, you would have to setup your connection correctly to do this.

Mongo Ops

But wait! There is more! The server status command also brings back operation counts. You can get totals for commands, deletes, get mores, inserts, queries, and updates. With this information and scout’s handy remember/memory, I can keep track of the numbers for these values from the last run and compare them with the current run to get up to date values for the mongo ops plugin. If you aren’t giggling right now, well, there is nothing I can do for you.

Harmony’s numbers are not going to impress anyone right now, but I will post a few screenshots so you can get some idea of what the plugins look like before running off to install them.

With the addition of these three plugins, in cooperation with the slow query plugin, and entirely thanks to Scout, I now have lots of numbers and graphs that allow me to watch growth. Fun stuff!

Note that scout is not the only way you can get this information. MongoMachine actually already shows this kind of stuff on their dashboard. Check it out if you aren’t into running Mongo yourself.

What are you using to keep track of Mongo?

MongoTips: All Mongo, All The Time »

Created at: 17.02.2010 22:00, source: RailsTips - Home, tagged: mongodb site news

So last night I realized that I really love MongoDB. I love it so much that I want everyone to be using it whenever it makes sense (which is quite often). I thought about what I could do to help get the word out more and starting a new site made the most sense (instead of flooding RailsTips with Mongo stuff).

MongoTips is a new blog dedicated to MongoDB news and information. I’ll be posting things there that we have learned building Harmony and anything I can find from the community.

If you are interested in Mongo, be sure to subscribe to the feed or follow on twitter.

The Process

What I love about the process of making this site is how smooth it was with Harmony. I registered the domain, setup the DNS, added the site in Harmony, duplicated the RailsTips theme and a few minutes later the site was ready to go. All that was needed was a few posts and it was ready to launch.

That is really what I want Harmony to bring to the table. I want it to take away all the pain so all you are left with is authoring your site. Lovely.

Live From Harmony »

Created at: 06.01.2010 08:00, source: RailsTips - Home, tagged: mongodb harmony

After more than a year and a half of self-funded work on and off, Harmony is almost ready. It is so close in fact, that we (Steve and I) are beginning to eat our own dog food by moving our own sites into it. Rails Tips is the first to go live (if you can see this post, you are a witness) and I’m not going to lie, it feels good!

As I mentioned previously, we recently switched Harmony to MongoDB. A few have asked what issues we ran into and so far it has only been mental (free your mind matrix-style stuff). Now we will start getting into the nitty gritty. Expect some good posts soon.

Rails Tips is all moved and our company site, OrderedList.com, is in the final stages of the redesign process and should be going live in Harmony later this week.

Redirects

Harmony has redirect support (including regex’s and back references) so I’ve put a few in place to fix the URLs, but there are bound to be some broken things around here. If you run into anything that isn’t working, just let me know with a comment or something.

Technical Details To Come

I am really excited about the future of Harmony and I’ll be posting more on the technical details soon, but for now this lame post will have to suffice. :)