bgror

Caching With Mongo »

Created at: 15.07.2010 15:00, source: RailsTips - Home, tagged: mongodb gems

For those of you that do not follow me on Twitter or Github, a while back I released Bin, an ActiveSupport MongoDB cache store. Since I have been quiet here, I thought I would talk a bit about it to help get back in the swing of things.

Using Bin is just like using any other AS cache store.

connection = Mongo::Connection.new
db = connection['bin_cache']

Rails::Initializer.run do |config|
  config.cache_store = Bin::Store.new(db)
end

Once you have set things up, you can use all the typical Rails.cache methods.

Rails.cache.write('foo', 'bar')
Rails.cache.read('foo') # 'bar'

Rails.cache.fetch('foo') do
  # some expensive thing
end

The list goes on, but in the interest of brevity, I will just link you to the specs. The cool thing about bin is that it supports both ActiveSupport 2 and 3 along with Ruby 1.8.7 and 1.9.1. Oh, and it supports expires with the same API as the memcache store.

That pretty much covers the basics, feel free to go kick the tires or hang around here a bit to learn how I made Bin work with AS2 and AS3.

Supporting AS2/3

In both ActiveSupport 2 and 3, you inhert from ActiveSupport::Cache::Store to create a new store. The difference between the version is quite subtle though. In AS3, you override the methods read, write, etc. as needed and use super with a block to get the inherited functionality. In AS2, you do the same thing, but super does not accept a block. Being that as a community we are now mugwumps (mug on Rails 2 and wumps on Rails 3), I thought it would be nice to support both.

In order to make this happen, I knew all I need to do was shim compatibility for Rails 2. So what I did is create a compatibility class that inherits from ActiveSupport::Cache::Store for AS3 and then if active support’s version is less than 3, I reopen the class and add in the compatibility stuff to make it work like 3. Here is the code in its entirety:

# encoding: UTF-8
module Bin
  class Compatibility < ActiveSupport::Cache::Store
    def increment(key, amount=1)
      yield
    end

    def decrement(key, amount=1)
      yield
    end
  end

  if ActiveSupport::VERSION::STRING < '3'
    class Compatibility
      def write(key, value, options=nil, &block)
        super(key, value, options)
        yield
      end

      def read(key, options=nil, &block)
        super
        yield
      end

      def delete(key, options=nil, &block)
        super
        yield
      end

      def delete_matched(matcher, options=nil, &block)
        super
        yield
      end

      def exist?(key, options=nil, &block)
        super
        yield
      end
    end
  end
end

So then Bin::Store just inherits from Compatibility:

module Bin
  class Store < Compatibility
    # ... stuff
  end
end

I cringe using a specific version string comparison like above, but it was simple and worked so I went with it. The last piece of the puzzle was setting up rake tasks to run the specs against different active support versions.

namespace :spec do
  Spec::Rake::SpecTask.new(:all) do |t|
    t.ruby_opts << '-rubygems'
    t.verbose = true
  end

  task :as2 do
    sh 'ACTIVE_SUPPORT_VERSION="<= 2.3.8" rake spec:all'
  end

  task :as3 do
    sh 'ACTIVE_SUPPORT_VERSION=">= 3.0.0.beta3" rake spec:all'
  end
end

desc 'Runs all specs against Active Support 2 and 3'
task :spec do
  Rake::Task['spec:as2'].invoke
  Rake::Task['spec:as3'].invoke
end

Note that I make an all task to run the specs then two distinct tasks to run against AS2 and AS3. All those tasks do is set an environment variable that I use in the test to force a particular version.

gem 'activesupport', ENV['ACTIVE_SUPPORT_VERSION']

Now when I run rake, it runs the tests against a 2.3 and a 3.0+ version of ActiveSupport so I know when something goes wrong with either. No flipping gem sets or other shenanigans. As always, if you have improvements or other way so doing stuff like this, please let me know. I am here to learn people.

Using Bin on the last project I worked on to cache large fragments of the layout significantly reduced response times. Always fun to see numbers like that drop after a deploy!

MongoMapper 0.8: Goodies Galore »

Created at: 16.06.2010 22:00, source: RailsTips - Home, tagged: mongomapper gems

Let me tell you, this release has been a tough one. It is made up of 43 commits to Plucky and 92 commits to MongoMapper. Features added include a sexy query language, scopes, attr_accessible, a fancy cache key helper, a :typecast option for array/set keys, and a bajillion little improvements. Let’s run through each of them just for fun.

Sexy Query Language

This right here is all thanks to plucky. The goal for plucky is a fancy query language on top of MongoDB. It has been created in a such a way that other MongoDB projects (Mongoid, Candy, MongoDoc, etc.) can benefit from it if they wish. It still has a long way to go in covering edge cases and deeply nested queries, but the majority of queries one will do are covered quite nicely.

User.where(:age.gt => 27).sort(:age).all
User.where(:age.gt => 27).sort(:age.desc).all
User.where(:age.gt => 27).sort(:age).limit(1).all
User.where(:age.gt => 27).sort(:age).skip(1).limit(1).all

All of the above are supported out of the box. Each query method (limit, reverse, update, skip, fields, sort, where) returns a Plucky::Query object so they can be changed together until a kicker is hit, such as all, first, last, paginate, count, size, each, etc. It is fashioned in a similar form as ARel in this manner, but more simple as ARel has to handle a lot more than just simple queries (joins, etc).

Scopes

The main thing I was waiting for to do scopes was to get plucky to a point where scopes would be just a sprinkling of code to merge plucky queries. Thankfully that day has finally arrived and with this latest release, you can now scope away. The code is so compact, that I figured I would drop it in here for those that are curious:

module MongoMapper
  module Plugins
    module Scopes
      module ClassMethods
        def scope(name, scope_options={})
          scopes[name] = lambda do |*args|
            result = scope_options.is_a?(Proc) ? scope_options.call(*args) : scope_options
            result = self.query(result) if result.is_a?(Hash)
            self.query.merge(result)
          end
          singleton_class.send :define_method, name, &scopes[name]
        end

        def scopes
          read_inheritable_attribute(:scopes) || write_inheritable_attribute(:scopes, {})
        end
      end
    end
  end
end

Yep, that is it. With that bit of code, you can now do stuff like this:

class User
  include MongoMapper::Document

  # plain old vanilla scopes with fancy queries
  scope :johns,   where(:name => 'John')

  # plain old vanilla scopes with hashes
  scope :bills, :name => 'Bill'

  # dynamic scopes with parameters
  scope :by_name,  lambda { |name| where(:name => name) }
  scope :by_ages,  lambda { |low, high| where(:age.gte => low, :age.lte => high) }

  # Yep, even plain old methods work as long as they return a query
  def self.by_tag(tag)
    where(:tags => tag)
  end

  # You can even make a method that returns a scope
  def self.twenties; by_ages(20, 29) end

  key :name, String
  key :tags, Array
end

# simple scopes
pp User.johns.first
pp User.bills.first

# scope with arg
pp User.by_name('Frank').first

# scope with two args
pp User.by_ages(20, 29).all

# chaining class methods on scopes
pp User.by_ages(20, 40).by_tag('ruby').all

# scope made using method that returns scope
pp User.twenties.all

I am sure there are some edge cases, but I cannot wait to start swapping some of the code I have out for scopes. This is definitely one of the features I missed most from ActiveRecord.

attr_accessible

Previously, MongoMapper only supported attr_protected. The main reason was that I am lazy and someone from the community contributed the beginnings of the code. I spent some time today adding attr_accessible, so now you can really lock down your models if you want to.

class User
  include MongoMapper::Document

  attr_accessible :first_name, :last_name, :email

  key :first_name, String
  key :last_name, String
  key :email, String
  key :admin, Boolean, :default => false
end

Based on the example above, only first_name, last_name and email can be assigned when using mass assignment, such as in .new or #update_attributes.

Cache Key

On a recent MongoMapper project, I had to some caching. This led me to create bin, a MongoDB ActiveSupport cache store. The first thing you notice when you start to cache stuff is that you need a key to name the cached object or fragment. I dug around in AR and discovered the cache_key method. MongoMapper’s cache_key works the same with a little twist. You can pass arguments to it and they will become suffixes on the cache key. Lets look at an example:

class User
  include MongoMapper::Document
end

User.new.cache_key # => "User/new"
User.create.cache_key # => "User/:id"
User.create.cache_key(:foo, :bar) # => "User/:id/foo/bar"

It should also be noted that if the User model has an updated_at key, that will be appended after the id like so User/:id-:timestamp. This addition is definitely going to clean up some code on a project of mine.

Typecasting Array/Set values

A common idiom in MongoDB modeling is to use Array keys for many to many type relationships. You have a User model and a Site model. Sites can have many Users and Users can have many Sites. Typically, I make a key :user_ids, Array and store the ids of each user that has access to the site.

When this is done from web forms, everything comes in as a string, so you have to typecast those strings to object ids. The new :typecast option wraps this up in a single key/value.

class Site
  include MongoMapper::Document
  key :user_ids, Array, :typecast => 'ObjectId'
end

Now, whenever user_ids is assigned, each value gets typecast to an ObjectId. This will work with any class that defines the to_mongo class method, which means you can use it with custom types as well.

Conclusion

I learned more about Ruby while working on this release of MongoMapper than probably any other period in my brief history. I really feel like this release brings MongoMapper to the forefront of MongoDB/Ruby object mappers.

All the typical dressings are now in place and with a few more tweaks, I can smell 1.0. Hope you all find this stuff useful and as always, if you don’t, that is ok because I am enjoying the heck out of working on this stuff. :)

Oh, and if all of this above did not excite you, know that the new MongoMapper site, including full documentation, is well underway and should be ready for consumption soon. Hooray!

Because Gem Names Are Like Domains in the 90's »

Created at: 30.03.2010 18:00, source: RailsTips - Home, tagged: gems testing

One of my favorite parts of every new gem is naming it. The other day, when I was trying to name joint, it occurred to me that I should always check if a gem name is available before I create my project. I did a quick search on RubyGems and discovered it was available.

Last night, I decided I should whip together a tiny gem that allows you to issue a whois command to see if a gem name is taken. Why leave the command line, eh?

Installation

gem install gemwhois

This adds the whois command to gem. Which means usage is pretty fun.

Usage

$ gem whois httparty

   gem name: httparty
     owners: John Nunemaker, Sandro Turriate
       info: Makes http fun! Also, makes consuming restful web services dead easy.
    version: 0.5.2
  downloads: 40714
  
$ gem whois somenonexistantgem

  Gem not found. It will be mine. Oh yes. It will be mine. *sinister laugh*

If the gem is found, you will see some details about the project (maybe you can convince them to hand over rights if they are squatting). If the gem is not found, you will receive a creepy message in the same vein as the RubyGems 404 page.

The Fun Parts

The fun part of this gem was recently I noticed that other gems have been adding commands to the gem command. I thought that was interesting so I did a bit of research. I knew that both gemedit and gemcutter added commands so I downloaded both from Github and began to peruse the source. Turns out it is quite easy.

First, you have to have a rubygems_plugin.rb file in your gems lib directory. This is mostly ripped from gemcutter:

if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.3.6')
  require File.join(File.dirname(__FILE__), 'gemwhois')
end

Next, you have to create a command. At the time of this post, here is the entirety of the whois command:

require 'rubygems/gemcutter_utilities'

class Gem::Commands::WhoisCommand < Gem::Command
  include Gem::GemcutterUtilities

  def description
    'Perform a whois lookup based on a gem name so you can see if it is available or not'
  end

  def arguments
    "GEM       name of gem"
  end

  def usage
    "#{program_name} GEM"
  end

  def initialize
    super 'whois', description
  end

  def execute
    whois get_one_gem_name
  end

  def whois(gem_name)
    response = rubygems_api_request(:get, "api/v1/gems/#{gem_name}.json") do |request|
      request.set_form_data("gem_name" => gem_name)
    end

    with_response(response) do |resp|
      json = Crack::JSON.parse(resp.body)
      puts <<-STR.unindent

        gem name: #{json['name']}
          owners: #{json['authors']}
            info: #{json['info']}
         version: #{json['version']}
       downloads: #{json['downloads']}

      STR
    end
  end

  def with_response(resp)
    case resp
    when Net::HTTPSuccess
      block_given? ? yield(resp) : say(resp.body)
    else
      if resp.body == 'This rubygem could not be found.'
        puts '','Gem not found. It will be mine. Oh yes. It will be mine. *sinister laugh*',''
      else
        say resp.body
      end
    end
  end
end

The important part is inheriting from Gem::Command. Be sure to require 'rubygems/command_manager' at some point as well. Once you have the rubygems_plugin file and a command created, you simple register the command:

Gem::CommandManager.instance.register_command(:whois)

The comments and code in RubyGems itself is pretty helpful if you are curious about what you can do.

Testing

The trickier part was testing the command. Obviously, building the gem from gemspec and installing over and over does not a happy tester make. I did a bit of research and found the following testing output helpers and the unindent gem:

module Helpers
  module Output
    def assert_output(expected, &block)
      keep_stdout do |stdout|
        block.call
        if expected.is_a?(Regexp)
          assert_match expected, stdout.string
        else
          assert_equal expected.to_s, stdout.string
        end
      end
    end

    def keep_stdout(&block)
      begin
        orig_stream, $stdout = $stdout, StringIO.new
        block.call($stdout)
      ensure
        s, $stdout = $stdout.string, orig_stream
        s
      end
    end
  end
end

With these little helpers, it was quite easy to setup the command and run it in an automated way:

require 'helper'

class TestGemwhois < Test::Unit::TestCase
  context 'Whois for found gem' do
    setup do
      @gem = 'httparty'
      stub_gem(@gem)
      @command = Gem::Commands::WhoisCommand.new
      @command.handle_options([@gem])
    end

    should "work" do
      output = <<-STR.unindent

        gem name: httparty
          owners: John Nunemaker, Sandro Turriate
            info: Makes http fun! Also, makes consuming restful web services dead easy.
         version: 0.5.2
       downloads: 40707

      STR
      assert_output(output) { @command.execute }
    end
  end
  
  context "Whois for missing gem" do
    setup do
      @gem = 'missing'
      stub_gem(@gem, :status => ["404", "Not Found"])
      @command = Gem::Commands::WhoisCommand.new
      @command.handle_options([@gem])
    end

    should "work" do
      output = <<-STR.unindent

        Gem not found. It will be mine. Oh yes. It will be mine. *sinister laugh*

      STR
      assert_output(output) { @command.execute }
    end
  end
end

The only other piece of the puzzle was using FakeWeb to stub the http responses for the found and missing gems. You can see more on that in the test helper file.

Conclusion

At any rate, the gem is pretty tiny and possibly useless to others, but it was fun. Gave me a chance to play around with testing STDOUT and creating RubyGem commands. Plus, now I know if the gem name I want is available in just a few keystrokes.

A Nunemaker Joint »

Created at: 27.03.2010 05:00, source: RailsTips - Home, tagged: joint mongomapper gems gridfs

Since The Changelog has already scooped me (darn those guys are fast), I figured I should post something here as well. Last December I posted Getting a Grip on GridFS. Basically, I liked what Grip was doing, but made a few tweaks. Grip’s development has continued, but it is headed down a bit different path, so I thought it might be confusing to keep my fork named the same.

Today, I did some more work on the project and now that it is unrecognizable when compared to Grip, I renamed it to Joint. Yes, I realize that I now have gems named crack and joint. Joint is a tiny piece of code that joins MongoMapper and the new Ruby GridFS API.

Usage

What I love about joint is its simplicity. Simply declare the attachment and you are good to go.

class Asset
  include MongoMapper::Document
  plugin Joint # add the plugin

  attachment :file # declare an attachment named image
end

With that simple declaration, you get #file and #file= instance methods and several keys (file_id, file_name, file_type, and file_size). The #file instance method returns a simple proxy to make the API a bit prettier and sends all other calls on the proxy to the GridIO instance.

asset = Asset.create(:file => params[:file])
asset.file.id   # GridFS Object Id
asset.file.name # file name
asset.file.type # mime type as determined by wand gem
asset.file.size # size in bytes

There is no limit to the number of attachments, but each attachment uses 4 keys so I would not add more than one or two (more is a sign you are doing something wrong). As mentioned in the comment above, it uses my wand project to determine the mime type.

For those that are not familiar with wand (as I have not posted here about it), it first attempts to determine the mime type using the mime-types gem. If that fails to returning anything, it drops down to the unix file command.

What It Does Not Do

Anything else. All joint handles is assigning a file and storing it in GridFS. It doesn’t do versions, resizing, etc. For this type of stuff, I would recommend imagery with some HTTP caching (varnish, et el.) sitting in front of it.

I can certainly see having some triggers (callbacks) at some point such as after jointed or something for when you do want to do post processing. I’ll leave that for a rainy day though.

Canable: The Flesh Eating Permission System »

Created at: 27.02.2010 22:00, source: RailsTips - Home, tagged: gems

A while back I wrote about how to add simple permissions to your apps. Since then, I have worked on a few applications (Harmony among them) where I have taken that concept and expanded it. Yesterday, I decided that I had repeated myself enough times (3) and that I should abstract the shared functionality of those apps into a gem. Thus, Canable, the flesh eating permission system, was born.

Can

Canable does not actually implement any permissions for you (or actually eat flesh). Instead, it provides you with all the helpers and then (gasp) you have to do the work. The idea centers around running all permissions through current_user. Anytime you check if a user can do something you use a can method:

user.can_create?(article)
user.can_update?(article)

Able

Instead of having a big case statement in those can methods for each different type of object, I use the strategy pattern to just ask the object if the user has permission to do the action. This is done by having a matching “able” method to the “can” method, thus canable.

class Item
  def updatable_by?(user)
    creator == user
  end
end

The above code, for example, makes it so that only the creator of an item can update it. Obviously, you can get more in depth from there. By default, I add the following can and able methods:

:view => :viewable
:create => :creatable
:update => :updatable
:destroy => :destroyable

Custom Actions

If you need permissions for actions other than the defaults, you can add your own quite easily:

Canable.add(:publish, :publishable)

The readme over on Github has far more details, but I figured I would at least cover it here a bit. It might seem a bit weird at first, but once you start rolling with it, it makes for a pretty easy to implement and understand permission system.

The really funny part is that it is only like 80 lines of code, as most of the methods are dynamically generated. I am perfectly fine if I am the only one who uses this and finds it helpful, but you never know, so feel free to install it as a gem or fork it on github.

Note: No permissions were harmed in the making of this gem.