Avoid using metaprogramming (seriously!) »

Created at: 04.05.2008 11:29, source: Rails on the Run - Home, tagged: benchmark merb metaprogramming speed

Ruby is sexy, Ruby is cool and its metaprogramming potential offers some really cook features. However you might not realize that your cleverness is slowing down your code.

Today I was working on cleaning up merb_helper a Merb plugin that brings a lot of the stuff Rails developers are used to. In Merb we aim for speed and try to avoid magic.

merb_plugin didn't receive a lot of love from the main contributors but few features were added by different contributors and the code became hard to maintain.

Looking at the code I quickly found this bad boy:

(Old Merb Time DSL using metaprogramming)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21


   
  module MetaTimeDSL

    {:second => 1, 
     :minute => 60, 
     :hour => 3600, 
     :day => [24,:hours], 
     :week => [7,:days], 
     :month => [30,:days], 
     :year => [364.25, :days]}.each do |meth, amount|
      define_method "m_#{meth}" do
        amount = amount.is_a?(Array) ? amount[0].send(amount[1]) : amount
        self * amount
      end
      alias_method "m_#{meth}s".intern, "m_#{meth}"
    end

  end
  Numeric.send :include, MetaTimeDSL

The above code looks awful to me and I decided to rewrite it a way I thought would be more efficient:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


  module TimeDSL

    def second
      self * 1
    end
    alias_method :seconds, :second

    def minute
      self * 60
    end
    alias_method :minutes, :minute

    def hour
      self * 3600
    end
    alias_method :hours, :hour

    def day
      self * 86400
    end
    alias_method :days, :day

    def week
      self * 604800
    end
    alias_method :weeks, :week

    def month
      self * 2592000
    end
    alias_method :months, :month

    def year
      self * 31471200
    end
    alias_method :years, :year

  end
  Numeric.send :include, TimeDSL

To make sure I was right, I run the following benchmarks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127


require 'benchmark'
TIMES = (ARGV[0] || 100_000).to_i

Benchmark.bmbm do |x|
  
  x.report("metaprogramming 360.seconds") do
    TIMES.times do    
      360.m_seconds
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.seconds
    end
  end
  
  x.report("metaprogramming 360.minutes") do
    TIMES.times do    
      360.m_minutes
    end
  end
  
  x.report("no metaprogramming 360.minutes") do
    TIMES.times do
      360.minutes
    end
  end
  
  x.report("metaprogramming 360.hours") do
    TIMES.times do    
      360.m_hours
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.hours
    end
  end
  
  x.report("metaprogramming 360.days") do
    TIMES.times do    
      360.m_days
    end
  end
  
  x.report("no metaprogramming 360.days") do
    TIMES.times do
      360.days
    end
  end
  
  x.report("metaprogramming 360.weeks") do
    TIMES.times do    
      360.m_weeks
    end
  end
  
  x.report("no metaprogramming 360.weeks") do
    TIMES.times do
      360.weeks
    end
  end

  x.report("metaprogramming 18.months") do
    TIMES.times do    
      18.m_months
    end
  end
  
  x.report("no metaprogramming 18.months") do
    TIMES.times do
      18.months
    end
  end
  
  x.report("metaprogramming 7.years") do
    TIMES.times do    
      7.m_years
    end
  end
  
  x.report("no metaprogramming 7.years") do
    TIMES.times do
      7.years
    end
  end
  
end


 Rehearsal ------------------------------------------------------------------
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.133164)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042655)
metaprogramming 360.minutes      0.130000   0.000000   0.130000 (  0.133327)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.042401)
metaprogramming 360.hours        0.140000   0.000000   0.140000 (  0.134312)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.043125)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.134949)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043745)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.135581)
no metaprogramming 360.weeks     0.050000   0.000000   0.050000 (  0.043544)
metaprogramming 18.months        0.130000   0.000000   0.130000 (  0.135234)
no metaprogramming 18.months     0.050000   0.000000   0.050000 (  0.044354)
metaprogramming 7.years          0.140000   0.000000   0.140000 (  0.144062)
no metaprogramming 7.years       0.050000   0.000000   0.050000 (  0.044392)
--------------------------------------------------------- total: 1.260000sec

                                     user     system      total        real
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.132567)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.042777)
metaprogramming 360.minutes      0.140000   0.000000   0.140000 (  0.132554)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.043193)
metaprogramming 360.hours        0.130000   0.000000   0.130000 (  0.133027)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042613)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.138637)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043213)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.134049)
no metaprogramming 360.weeks     0.040000   0.000000   0.040000 (  0.043713)
metaprogramming 18.months        0.140000   0.000000   0.140000 (  0.134941)
no metaprogramming 18.months     0.040000   0.000000   0.040000 (  0.043980)
metaprogramming 7.years          0.150000   0.000000   0.150000 (  0.143389)
no metaprogramming 7.years       0.040000   0.000000   0.040000 (  0.044585)
 0.136591)

The metaprogramming version of the same implementation is almost 3 times slower!

Moral of the story: only use metaprogramming if you really have to or if you don't care about speed of execution.


more »

Avoid using metaprogramming (seriously!) »

Created at: 04.05.2008 11:29, source: Rails on the Run - Home, tagged: benchmark merb metaprogramming speed

Ruby is sexy, Ruby is cool and its metaprogramming potential offers some really cook features. However you might not realize that your cleverness is slowing down your code.

Today I was working on cleaning up merb_helper a Merb plugin that brings a lot of the stuff Rails developers are used to. In Merb we aim for speed and try to avoid magic.

merb_plugin didn't receive a lot of love from the main contributors but few features were added by different contributors and the code became hard to maintain.

Looking at the code I quickly found this bad boy:

(Old Merb Time DSL using metaprogramming)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21


   
  module MetaTimeDSL

    {:second => 1, 
     :minute => 60, 
     :hour => 3600, 
     :day => [24,:hours], 
     :week => [7,:days], 
     :month => [30,:days], 
     :year => [364.25, :days]}.each do |meth, amount|
      define_method "m_#{meth}" do
        amount = amount.is_a?(Array) ? amount[0].send(amount[1]) : amount
        self * amount
      end
      alias_method "m_#{meth}s".intern, "m_#{meth}"
    end

  end
  Numeric.send :include, MetaTimeDSL

The above code looks awful to me and I decided to rewrite it a way I thought would be more efficient:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


  module TimeDSL

    def second
      self * 1
    end
    alias_method :seconds, :second

    def minute
      self * 60
    end
    alias_method :minutes, :minute

    def hour
      self * 3600
    end
    alias_method :hours, :hour

    def day
      self * 86400
    end
    alias_method :days, :day

    def week
      self * 604800
    end
    alias_method :weeks, :week

    def month
      self * 2592000
    end
    alias_method :months, :month

    def year
      self * 31471200
    end
    alias_method :years, :year

  end
  Numeric.send :include, TimeDSL

To make sure I was right, I run the following benchmarks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127


require 'benchmark'
TIMES = (ARGV[0] || 100_000).to_i

Benchmark.bmbm do |x|
  
  x.report("metaprogramming 360.seconds") do
    TIMES.times do    
      360.m_seconds
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.seconds
    end
  end
  
  x.report("metaprogramming 360.minutes") do
    TIMES.times do    
      360.m_minutes
    end
  end
  
  x.report("no metaprogramming 360.minutes") do
    TIMES.times do
      360.minutes
    end
  end
  
  x.report("metaprogramming 360.hours") do
    TIMES.times do    
      360.m_hours
    end
  end
  
  x.report("no metaprogramming 360.hours") do
    TIMES.times do
      360.hours
    end
  end
  
  x.report("metaprogramming 360.days") do
    TIMES.times do    
      360.m_days
    end
  end
  
  x.report("no metaprogramming 360.days") do
    TIMES.times do
      360.days
    end
  end
  
  x.report("metaprogramming 360.weeks") do
    TIMES.times do    
      360.m_weeks
    end
  end
  
  x.report("no metaprogramming 360.weeks") do
    TIMES.times do
      360.weeks
    end
  end

  x.report("metaprogramming 18.months") do
    TIMES.times do    
      18.m_months
    end
  end
  
  x.report("no metaprogramming 18.months") do
    TIMES.times do
      18.months
    end
  end
  
  x.report("metaprogramming 7.years") do
    TIMES.times do    
      7.m_years
    end
  end
  
  x.report("no metaprogramming 7.years") do
    TIMES.times do
      7.years
    end
  end
  
end


 Rehearsal ------------------------------------------------------------------
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.133164)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042655)
metaprogramming 360.minutes      0.130000   0.000000   0.130000 (  0.133327)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.042401)
metaprogramming 360.hours        0.140000   0.000000   0.140000 (  0.134312)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.043125)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.134949)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043745)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.135581)
no metaprogramming 360.weeks     0.050000   0.000000   0.050000 (  0.043544)
metaprogramming 18.months        0.130000   0.000000   0.130000 (  0.135234)
no metaprogramming 18.months     0.050000   0.000000   0.050000 (  0.044354)
metaprogramming 7.years          0.140000   0.000000   0.140000 (  0.144062)
no metaprogramming 7.years       0.050000   0.000000   0.050000 (  0.044392)
--------------------------------------------------------- total: 1.260000sec

                                     user     system      total        real
metaprogramming 360.seconds      0.130000   0.000000   0.130000 (  0.132567)
no metaprogramming 360.hours     0.040000   0.000000   0.040000 (  0.042777)
metaprogramming 360.minutes      0.140000   0.000000   0.140000 (  0.132554)
no metaprogramming 360.minutes   0.040000   0.000000   0.040000 (  0.043193)
metaprogramming 360.hours        0.130000   0.000000   0.130000 (  0.133027)
no metaprogramming 360.hours     0.050000   0.000000   0.050000 (  0.042613)
metaprogramming 360.days         0.130000   0.000000   0.130000 (  0.138637)
no metaprogramming 360.days      0.050000   0.000000   0.050000 (  0.043213)
metaprogramming 360.weeks        0.130000   0.000000   0.130000 (  0.134049)
no metaprogramming 360.weeks     0.040000   0.000000   0.040000 (  0.043713)
metaprogramming 18.months        0.140000   0.000000   0.140000 (  0.134941)
no metaprogramming 18.months     0.040000   0.000000   0.040000 (  0.043980)
metaprogramming 7.years          0.150000   0.000000   0.150000 (  0.143389)
no metaprogramming 7.years       0.040000   0.000000   0.040000 (  0.044585)
 0.136591)

The metaprogramming version of the same implementation is almost 3 times slower!

Moral of the story: only use metaprogramming if you really have to or if you don't care about speed of execution.


more »

Rails or Merb, what's best for you? »

Created at: 10.04.2008 11:10, source: Rails on the Run - Home, tagged: benchmark merb rails

If you follow my blog, you already know what Merb is.

I love Rails and I truly believe it has changed web development. At least it has changed the way I do web development.

But Merb looks slick, apparently is way faster than Rails, and has less "fluff" and less magic.

Now that we are getting really close to a Merb 1.0 (scheduled for Rails Conf '08) it's time to evaluate if Merb is the good choice for some of my clients' projects.

However, according to Merb's author, Ezra, at MountainWest RubyConf 2008, Rails will get you there faster. In a client's case, they don't need to build a huge app but need a lot of speed and the ability to easily handle a heavy load right away without using caching. Also most of the traffic will go through an API so we won't have to manage too many views.

Let's see how fast Merb really is.

To test Merb's speed, I built the very same prototype using Merb 0.9.2 and Rails Edge (pre 2.1). Both apps use ActiveRecord and are connected to a UTF8 MySQL database, both apps have exactly the same views. (Note that Merb would run way faster using DataMapper, but I don't feel that DM 0.9x is production ready yet, also, using a rack handler would certainly be way faster but my goal was really to compare ActionPack vs Merb.)

Both apps use the same ActiveRecord class, their controllers are a bit different but basically do the same thing.

Here is what was tested:

  • The Merb/Rails app should receive a GET request with a JSON object in the query.

  • The Merb/Rails app should route the request to a controller and pass the JSON object to an AR class.

  • The AR class should parse the JSON object (which contains an array of objects), extract each object, and try to find them in the database using one of the attributes. If the object isn't found, it should be created, otherwise it should return the AR object. The amount of hits should be incremented by 1 and the object should be saved back to the database.

  • A simple HTML view should be rendered

Quick Merb benchmark

merb

I setup Merb to run locally on my MacBook 2.16Ghz Core Duo 2, 2Gb Ram. To test the raw performance, Merb is started in production mode.

I then used httperf to make 10000 connections to the server at a rate of 500 (--rate=500 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1)

Here are the results:

Maximum connect burst length: 29
Total: connections 4377 requests 4221 replies 2932 test-duration 41.629 s
Connection rate: 105.1 conn/s (9.5 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 41.0 avg 1920.4 max 35390.8 median 898.5 stddev 4887.3
Connection time [ms]: connect 2118.1
Connection length [replies/conn]: 1.000

Request rate: 101.4 req/s (9.9 ms/req)
Request size [B]: 321.0

*Reply rate [replies/s]: min 0.0 avg 73.3 max 143.0 stddev 65.8 (8 samples)*
Reply time [ms]: response 809.0 transfer 18.1
Reply size [B]: header 121.0 content 557.0 footer 0.0 (total 678.0)
*Reply status: 1xx=0 2xx=2932 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.35 system 36.54 (user 0.8% system 87.8% total 88.6%)
Net I/O: 78.4 KB/s (0.6*10^6 bps)

Errors: total 7068 client-timo 0 socket-timo 0 connrefused 0 connreset 1445
Errors: fd-unavail 5623 addrunavail 0 ftab-full 0 other 0

What we care about is the reply rate/s. We have an average of 73.3 requests per second with a standard deviation of 65.8 using 8 samples.

We also make sure that all the replies were successful. (status == 2xx)

I also checked the database, made sure my AR object was created and that the hits were increased. AR object hits: 2932, which matches the amount of replies reported by httperf.

We don't care so much about the rest of the httperf. Let's move on to the Rails benchmark.

Quick Rails benchmark

rails

Rails is set the same way, running locally in production mode, same httperf settings.

Here are the results:

Maximum connect burst length: 44

Total: connections 2923 requests 2825 replies 1672 test-duration 37.418 s

Connection rate: 78.1 conn/s (12.8 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 382.7 avg 5635.4 max 36384.5 median 1887.5 stddev 10103.1
Connection time [ms]: connect 3631.2
Connection length [replies/conn]: 1.000

Request rate: 75.5 req/s (13.2 ms/req)
Request size [B]: 319.0

*Reply rate [replies/s]: min 0.0 avg 43.4 max 75.2 stddev 30.8 (7 samples)*
Reply time [ms]: response 1568.1 transfer 36.7
Reply size [B]: header 471.0 content 581.0 footer 0.0 (total 1052.0)
*Reply status: 1xx=0 2xx=1672 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.25 system 31.31 (user 0.7% system 83.7% total 84.4%)
Net I/O: 69.4 KB/s (0.6*10^6 bps)

DB hits: 1672

First thing, the database object was created properly and the hits incremented to 1672 which matches the amount of replies reported by httperf.

Then, we notice that on this test, we only got 7 samples, that's more than enough though. The standard deviation is 30.8 which is better than Merb's 65.8. That means that in our benchmarks, the reply speed difference in Merb's requests was bigger than Rails'. Not a big deal, this is not a scientific test but it's good to acknowledge it.

What we really care about is the average reply rate: 43.4

Let's also note that all the replies had a 2xx status, so everything went well.

Results

Based on this really basic benchmark, my Merb app had an average reply rate of 73.3 requests per second against Rails' 43.4 requests per second.

That means that in this very specific case,

Merb is 69% faster than Rails! Sexy!

In other words, my Merb prototype could handle 69% more requests than the Rails prototype in the same amount of time.

I heard people reporting than Merb was 3 to 5 times faster than Rails. Honestly, it really depends on what you do. By using ActiveRecord on both prototypes, I limited the speed difference since AR is not multithread and therefore Merb can't run as fast as it would using Sequel or DataMapper. By actually hitting the database on every single request, I also made sure to really compare ActionPack vs Merb.

Conclusion

The conclusion is simple, I recommended that my client go with Merb. Merb 1.0 is almost ready, the public API has been frozen. My client needs speed and simplicity. Using Merb I get exactly what I need and nothing more. Actually, we'll probably increase the performance by writing a rack handler and bypassing the entire framework for API calls (that should be wicked fast!). Also, as soon as DataMapper becomes production ready, we'll switch to DM and should get way better performance!

Am I suggesting to give up Rails and switch to Merb? Absolutely not! First off, Merb is a "lower level" framework. It requires a deeper understanding of Web Development in general and being more than just 'acquainted' with the Ruby language. So, unless you are an advanced developer or have time to learn, I would suggest to keep on using Rails (start using Merb on personal projects, it's a perfect way of learning). If you have a lot of views and/or use loads of AJAX, RJS, built-in helpers, you probably want to stick to Rails and start looking at how you can do all of that from scratch. By default Rails uses nasty helpers that create inline javascript, and is something you really want to avoid. RJS is fun, but it goes against Merb's philosophy, so you need to make sure you can live without it (note that you can reproduce the same behavior in Merb rendering JS, it just requries more work). If you rely a lot of Rails plugins, you might want to delay your switch, Merb is pretty new and doesn't have a mass-load of plugins yet.

Finally, Merb doesn't have a lot of documentation and changed a lot when 0.9 got released. To understand how Merb works, you will need to go through the source code, specs, Google, and ask on the Merb IRC channel.

It turns out that in our case we have experienced developers, a great need for speed, not too many views and are following Merb's development really closely . I honestly think it's the best choice for my client and I'm excited they accepted to use Merb.

Merb is addressing different issues than Rails and doing it well. I think there is a bright future for Merb. And don't even think that Rails is going away, that won't happen anytime soon!

Recently, Sony Playstation even posted a job post looking for a Rails/Merb developer. This is very promising for the Merb community!


more »

Rails or Merb, what's best for you? »

Created at: 10.04.2008 11:10, source: Rails on the Run - Home, tagged: benchmark merb rails

If you follow my blog, you already know what Merb is.

I love Rails and I truly believe it has changed web development. At least it has changed the way I do web development.

But Merb looks slick, apparently is way faster than Rails, and has less "fluff" and less magic.

Now that we are getting really close to a Merb 1.0 (scheduled for Rails Conf '08) it's time to evaluate if Merb is the good choice for some of my clients' projects.

However, according to Merb's author, Ezra, at MountainWest RubyConf 2008, Rails will get you there faster. In a client's case, they don't need to build a huge app but need a lot of speed and the ability to easily handle a heavy load right away without using caching. Also most of the traffic will go through an API so we won't have to manage too many views.

Let's see how fast Merb really is.

To test Merb's speed, I built the very same prototype using Merb 0.9.2 and Rails Edge (pre 2.1). Both apps use ActiveRecord and are connected to a UTF8 MySQL database, both apps have exactly the same views. (Note that Merb would run way faster using DataMapper, but I don't feel that DM 0.9x is production ready yet, also, using a rack handler would certainly be way faster but my goal was really to compare ActionPack vs Merb.)

Both apps use the same ActiveRecord class, their controllers are a bit different but basically do the same thing.

Here is what was tested:

  • The Merb/Rails app should receive a GET request with a JSON object in the query.

  • The Merb/Rails app should route the request to a controller and pass the JSON object to an AR class.

  • The AR class should parse the JSON object (which contains an array of objects), extract each object, and try to find them in the database using one of the attributes. If the object isn't found, it should be created, otherwise it should return the AR object. The amount of hits should be incremented by 1 and the object should be saved back to the database.

  • A simple HTML view should be rendered

Quick Merb benchmark

merb

I setup Merb to run locally on my MacBook 2.16Ghz Core Duo 2, 2Gb Ram. To test the raw performance, Merb is started in production mode.

I then used httperf to make 10000 connections to the server at a rate of 500 (--rate=500 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1)

Here are the results:

Maximum connect burst length: 29
Total: connections 4377 requests 4221 replies 2932 test-duration 41.629 s
Connection rate: 105.1 conn/s (9.5 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 41.0 avg 1920.4 max 35390.8 median 898.5 stddev 4887.3
Connection time [ms]: connect 2118.1
Connection length [replies/conn]: 1.000

Request rate: 101.4 req/s (9.9 ms/req)
Request size [B]: 321.0

*Reply rate [replies/s]: min 0.0 avg 73.3 max 143.0 stddev 65.8 (8 samples)*
Reply time [ms]: response 809.0 transfer 18.1
Reply size [B]: header 121.0 content 557.0 footer 0.0 (total 678.0)
*Reply status: 1xx=0 2xx=2932 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.35 system 36.54 (user 0.8% system 87.8% total 88.6%)
Net I/O: 78.4 KB/s (0.6*10^6 bps)

Errors: total 7068 client-timo 0 socket-timo 0 connrefused 0 connreset 1445
Errors: fd-unavail 5623 addrunavail 0 ftab-full 0 other 0

What we care about is the reply rate/s. We have an average of 73.3 requests per second with a standard deviation of 65.8 using 8 samples.

We also make sure that all the replies were successful. (status == 2xx)

I also checked the database, made sure my AR object was created and that the hits were increased. AR object hits: 2932, which matches the amount of replies reported by httperf.

We don't care so much about the rest of the httperf. Let's move on to the Rails benchmark.

Quick Rails benchmark

rails

Rails is set the same way, running locally in production mode, same httperf settings.

Here are the results:

Maximum connect burst length: 44

Total: connections 2923 requests 2825 replies 1672 test-duration 37.418 s

Connection rate: 78.1 conn/s (12.8 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 382.7 avg 5635.4 max 36384.5 median 1887.5 stddev 10103.1
Connection time [ms]: connect 3631.2
Connection length [replies/conn]: 1.000

Request rate: 75.5 req/s (13.2 ms/req)
Request size [B]: 319.0

*Reply rate [replies/s]: min 0.0 avg 43.4 max 75.2 stddev 30.8 (7 samples)*
Reply time [ms]: response 1568.1 transfer 36.7
Reply size [B]: header 471.0 content 581.0 footer 0.0 (total 1052.0)
*Reply status: 1xx=0 2xx=1672 3xx=0 4xx=0 5xx=0*

CPU time [s]: user 0.25 system 31.31 (user 0.7% system 83.7% total 84.4%)
Net I/O: 69.4 KB/s (0.6*10^6 bps)

DB hits: 1672

First thing, the database object was created properly and the hits incremented to 1672 which matches the amount of replies reported by httperf.

Then, we notice that on this test, we only got 7 samples, that's more than enough though. The standard deviation is 30.8 which is better than Merb's 65.8. That means that in our benchmarks, the reply speed difference in Merb's requests was bigger than Rails'. Not a big deal, this is not a scientific test but it's good to acknowledge it.

What we really care about is the average reply rate: 43.4

Let's also note that all the replies had a 2xx status, so everything went well.

Results

Based on this really basic benchmark, my Merb app had an average reply rate of 73.3 requests per second against Rails' 43.4 requests per second.

That means that in this very specific case,

Merb is 69% faster than Rails! Sexy!

In other words, my Merb prototype could handle 69% more requests than the Rails prototype in the same amount of time.

I heard people reporting than Merb was 3 to 5 times faster than Rails. Honestly, it really depends on what you do. By using ActiveRecord on both prototypes, I limited the speed difference since AR is not multithread and therefore Merb can't run as fast as it would using Sequel or DataMapper. By actually hitting the database on every single request, I also made sure to really compare ActionPack vs Merb.

Conclusion

The conclusion is simple, I recommended that my client go with Merb. Merb 1.0 is almost ready, the public API has been frozen. My client needs speed and simplicity. Using Merb I get exactly what I need and nothing more. Actually, we'll probably increase the performance by writing a rack handler and bypassing the entire framework for API calls (that should be wicked fast!). Also, as soon as DataMapper becomes production ready, we'll switch to DM and should get way better performance!

Am I suggesting to give up Rails and switch to Merb? Absolutely not! First off, Merb is a "lower level" framework. It requires a deeper understanding of Web Development in general and being more than just 'acquainted' with the Ruby language. So, unless you are an advanced developer or have time to learn, I would suggest to keep on using Rails (start using Merb on personal projects, it's a perfect way of learning). If you have a lot of views and/or use loads of AJAX, RJS, built-in helpers, you probably want to stick to Rails and start looking at how you can do all of that from scratch. By default Rails uses nasty helpers that create inline javascript, and is something you really want to avoid. RJS is fun, but it goes against Merb's philosophy, so you need to make sure you can live without it (note that you can reproduce the same behavior in Merb rendering JS, it just requries more work). If you rely a lot of Rails plugins, you might want to delay your switch, Merb is pretty new and doesn't have a mass-load of plugins yet.

Finally, Merb doesn't have a lot of documentation and changed a lot when 0.9 got released. To understand how Merb works, you will need to go through the source code, specs, Google, and ask on the Merb IRC channel.

It turns out that in our case we have experienced developers, a great need for speed, not too many views and are following Merb's development really closely . I honestly think it's the best choice for my client and I'm excited they accepted to use Merb.

Merb is addressing different issues than Rails and doing it well. I think there is a bright future for Merb. And don't even think that Rails is going away, that won't happen anytime soon!

Recently, Sony Playstation even posted a job post looking for a Rails/Merb developer. This is very promising for the Merb community!


more »

bgurl
ипотпал
аквичоп


Tag cloud

Categories

Links

Blogs: 43
Posts: 2516