bgror

The Road to Passenger 3: Technology Preview 2 – Stability, robustness, availability, self-healing »

Created at: 18.06.2010 16:13, source: Phusion Corporate Blog, tagged: Phusion Passenger

In Technology Preview 1 we’ve shown that Phusion Passenger 3 can be up to 55% faster on OS X. Performance is good and all, but it won’t do you any good unless the software keeps running. When Phusion Passenger goes down it’s an annoyance at best, but in the worst case any amount of down time can cost your organization real money. Any HTTP request that’s dropped can mean a lost transaction.

Although stability, robustness and availability aren’t as hot and fashionable as performance, for Phusion Passenger 3 we have not neglected these areas. In fact we’ve been working on hard on implementing additional safeguards, as well as refactoring our designs to make things more stable, robust and available.

Self-healing

In Phusion Passenger 2.2′s architecture, there are a number of processes that work together. At the very front there is the web server, which could consist of multiple processes. If you’ve ever typed passenger-memory-stats then you’ve seen the web server processes at work. Apache typically has a dozen of processes (prefork MPM) and Nginx typically has 3.

For Phusion Passenger to work, there must be some kind of global state, shared by all web server processes. In this global state information is stored such as which Ruby app processes exist, which ones are currently handling requests, etc. This allows Phusion Passenger to make decisions such as which Ruby app process to route a request to, whether there is any need to spawn another process, etc. This global state exists in a separate process which all web server processes communicate with. On Apache this is the ApplicationPoolServerExecutable, on Nginx this is the HelperServer. For simplicity’s sake, let’s call both of them HelperServer (in Passenger 3 they’ve both been renamed to PassengerHelperAgent). The HelperServer is written in C++ and is extremely fast and lightweight, consuming only about 500 KB of real memory.

As you can see the HelperServer is essentially the core of Phusion Passenger. The problem with 2.2 is that if the HelperServer goes down, Phusion Passenger goes down with it entirely. Phusion Passenger will stay down until the web server is restarted. For various architectural reasons in Apache and Nginx, it is not easily possible to restart the HelperServer upon a crash in a reliable way.

Now, why would the HelperServer ever crash?

1. Bugs. We are humans too and we can make mistakes, so it’s possible that there are crasher bugs in the HelperServer. In the past 2 years we’ve spent a lot of effort into making the HelperServer stable. For example we check all system calls for error results, and we’ve spent a lot of effort into making sure that uncaught exceptions are properly logged and handled. However one can never prove that a system is entirely bug free. We aren’t aware of any crasher bugs at this time but they might still exist.
2. Limited system resources. For example if the system is very low on memory, the kernel will invoke the Out-Of-Memory Killer (OOM Killer). Properly selecting a process to kill in low-memory conditions is actually a pretty hard problem, and more often than not the OOM Killer selects the wrong process, e.g. our HelperServer.
3. System administrator mistakes. Passing the wrong PID to the kill command and things like that.
4. System configuration problems, hardware problems (faulty RAM and stuff) and operating system bugs.

There are some people who have reported problems with the HelperServer. Their HelperServer crashes tend to happen sporadically and they usually cannot reproduce the problem reliably themselves. For many people (2) is often the cause of HelperServer crashes, and increasing the amount of swap is reported to help, but for other people the problems lie elsewhere.

The crashes aren’t always our fault (i.e. not bugs), but they are always our problem. It saddens us to say that we’ve been unable to help these people so far because we simply cannot reproduce their problems even when we mimic their system configuration.

But this is going to change.

Enter Phusion Passenger 3 with self-healing architecture

Phusion Passenger 3 now introduces a lightweight watchdog process into the architecture. It monitors both the web server and the HelperServer. If the HelperServer crashes, then the watchdog restarts the HelperServer immediately.

Of course if the watchdog is killed then it’s still game over, but we’ve taken extra care in terms of code to try to make this extremely unlikely to happen. The watchdog for starters is extremely lightweight, even more so than the HelperServer. It is written in C++ and uses about 150-200 KB of memory. Its only job is to start the HelperServer and other Phusion Passenger helper processes and to monitor them. The codebase extensively uses C++ idioms that promote code stability, such as smart pointers and RAII. By employing heavy testing as well, we’re expecting to have brought the possibility that the watchdog contains crashing bugs to a minimum. The small footprint and the fact that it does nothing most of the time minimizes the chances that it’s killed by the OOM Killer. In fact, if the watchdog is running on Linux and has root access, it will register itself as not OOM-killable.

No longer will HelperServer crashes take down Phusion Passenger, even if the crash isn’t our fault.

Restarts are fast

It only takes a few hundred miliseconds to restart the HelperServer.

Crashing signals are logged

If the HelperServer crashes then the watchdog will tell you whether it crashed because of a signal, e.g. SIGSEGV. This makes it much easier for system administrators to see why a component crashed so that they might fix the underlying cause. In Phusion Passenger 2.2 this was not possible.

Guaranteed cleanup

Upon shutting down or restarting the web server, Phusion Passenger 2.2 gracefully notifies application processes to shut down. It does not force them to. This would pose a problem for broken web applications that don’t shut down properly, e.g. web applications that are stuck in an infinite loop, stuck in a database call, etc.

Phusion Passenger 3 guarantees that all application processes are properly shut down when you shutdown/restart the web server. It gives application processes a deadline of 30 seconds to shutdown gracefully; if any of them fail to do that, they’ll be terminated with SIGKILL.

This mechanism works so well that it even extends to background processes that have been spawned off by the web application processes. All of those processes belong to the same process group. Phusion Passenger sends SIGKILL to the entire process group and terminates everything. No longer will you have to manually clean up processes; you can be confident that everything is gone if you shutdown/restart the web server.

Zero-downtime web server restart

In Phusion Passenger 2.2, whenever you restart the web server, HTTP requests that are currently in progress are dropped and the clients receive ugly “Connection reset by server” or similar error messages. This can be a major problem for large websites, because during the 1 second that Phusion Passenger is restarting hundreds of people could be getting errors. If your visitor happens to be clicking on that “Buy” button, well, tough luck.

In Phusion Passenger 3 we’ve implemented zero-downtime web server restart. Phusion Passenger and the web server are restarted in the background, and while this is happening, the old web server instance (with the old Phusion Passenger instance) will continue to process requests.

The architecture is actually a little bit more complicated than what’s shown in the diagram because behind the web server there are a bunch of Phusion Passenger processes, but you get the gist of it.

When the new web server (along with the new Phusion Passenger) has been started, it will immediately begin accepting new requests. Old requests that aren’t finished yet will continue to be processed by the old web server and Phusion Passenger instance. The old instance will shut down 5 seconds after all requests have been finished, to counter the possibility that the kernel still has leftover requests in the socket backlog that hit the old instance after its done processing everything already in its queue.

This works so well that we can restart the web server while running an ‘ab’ benchmark with 100 concurrent users without dropping a single request!

Zero-downtime application shutdown

Suppose that a Ruby application process has gone rogue and you want to shut it down. The most obvious way to do that is by sending it a SIGTERM or SIGKILL signal. However this would also abort whatever request it is currently processing.

In Phusion Passenger 2.2, you could also send SIGUSR1 to the process, causing it to shut down gracefully after it has processed all requests in its socket backlog. However this introduces two problems:

• If the website is very busy then the process’s socket backlog will never be empty, and so the process will never exit.
• Exiting after the process has detected an empty backlog can introduce a race condition. Suppose that, right after the process has determined that its backlog is empty but before it has shutdown completely, Phusion Passenger tries to send another request to the process. This request would be lost.

In Phusion Passenger 3, SIGUSR1 will now cause the application process to first unregister itself so that Phusion Passenger won’t route any new requests to it anymore. It will then proceed with exiting 5 seconds after its socket backlog has become empty. This way you can gracefully shutdown a process without losing a single request.

Conclusion

Although Phusion Passenger has been powering many high-traffic Ruby websites for a while now, some people still have some doubts about whether Phusion Passenger is fit for production. Instead of using words convince them, we would rather convince them with real results. Phusion Passenger 3 raises the bar in the areas of performance, stability, robustness and availability yet higher, but it doesn’t stop here. Please stay tuned for the next Technology Preview in which will unveil even more of Phusion Passenger 3.

The Road to Passenger 3: Technology Preview 1 – Performance »

Created at: 10.06.2010 16:11, source: Phusion Corporate Blog, tagged: Phusion Passenger

It has already been two years since we’ve first released Phusion Passenger. Time sure flies and we’ve come a long way since then. We were the first to implement a working Ruby web app deployment solution that integrates seamlessly in the web server, and all the features that we’ve developed over time – smart spawning and memory reduction, upload buffering, Nginx support, etc – have served us for a long time. Nevertheless, it is time to say goodbye to the old Phusion Passenger 2.2 codebase. In the past we had focused primarily on three things:

• Ease of use.
• Stability.
• Robustness.

Notice that “performance” is not on the above list. We strived to make Phusion Passenger “fast enough”, e.g. not ridiculously slower than the alternatives. Lately it would appear that competitors are once again focusing on performance. We can of course not afford to stay behind. We’ve been working on Phusion Passenger 3 for a while now. Today we will begin unveiling the technology behind this new major Phusion Passenger version. This blog post is the first of the multiple technology previews to come.

The performance test

It’s not very useful to benchmark Phusion Passenger performance using a Rails application because most of the time is spent in Rails and the application itself. Therefore we’ll benchmark with a simple Rack application. Consider the following hello world Rack application:

app = proc do |env|
  [200, { "Content-Type" => "text/html" }, ["hello world\n"]]
end

run app

How fast does this run on Phusion Passenger 2.2?

• Operating system: OS X Snow Leopard
• ab -c 25 -n 10000 http://rack.test/, pool size 25
• Apache: 1628 req/sec
• Nginx: 1843 req/sec

Now let’s look at Phusion Passenger 3:

• Operating system: OS X Snow Leopard
• Apache: 2225 req/sec; 36% faster
• Nginx: 2864 req/sec; 55% faster

That’s right, the Nginx version is over 50% faster than 2.2 on OS X!

A graph is worth more than a thousand words

Suffice it to say, even though Phusion Passenger was already pretty fast, we believe we’ve created some pretty significant improvements in terms of performance and it will be interesting to see how the final version of Phusion Passenger 3 will stack up against the competition. Needless to say, we’ve performed our own benchmarks already and have concluded that “the self-proclaimed fastest deployment solution” really isn’t the fastest deployment solution compared to Phusion Passenger 3. That said, benchmarks are lies, lies, lies, damn lies of course and your mileage may definitely vary so we will encourage you to perform any kind of benchmark you’d like when we release 3. For us, the most important issue still lies in the trade off of how much time you have to spend actually maintaining your setup, but as the graphs indicate, we’ve made some pretty monstrous improvements to performance as well.

How did we do it?

When it comes to optimizing software, there’s the saying that 20% of the code is responsible for 80% of the time. Not so with Phusion Passenger: we’ve found that there were no obvious performance bottlenecks. Even profilers turned out to be totally useless because all the times are so small and so close to each other. Phusion Passenger was already pretty fast.

Instead, we optimized the hard way: with lots and lots of micro-optimizations. 2% here, 3% there, etc etc. In other words, blood, sweat, tears and lots of sleepless nights. The optimizations can be summed up as follows:

Reducing system calls

System calls are pretty expensive compared to userspace computation. They require a context switch to the kernel. For example, all I/O operations (read(), write()) are system calls. We’ve performed an extensive code inspection and removed and coalesced a lot of redundant system calls.

The beginning of a zero-copy I/O architecture

The CPU is very fast nowadays. In fact it is so fast that RAM speed cannot keep up with the CPU. This makes memory access very expensive. In case of I/O intensive applications such as web servers, one would benefit from copying I/O data as little as possible. In order to optimize memory access, we’ve implemented the beginning of a zero-copy I/O architecture. This architecture covers both the C++ and the Ruby parts of Phusion Passenger.

Less Ruby garbage production

The garbage collector in Ruby can be a significant bottleneck. We’ve heavily optimized the Ruby part of the request handler and reduced creation of Ruby objects to a minimum. This made the request handler significantly faster in our tests.

Optimizing algorithms and optimizing Ruby code in C

Some algorithms have been optimized, e.g. some O(N) algorithms have been replaced by O(log N) or O(1) algorithms. Some key Ruby code has been replaced by C code. The former didn’t give us a lot of performance because all the O(N) algorithms weren’t doing a lot of work in the first place, but the latter gave us a much more noticeable boost.

Reducing context switches

Phusion Passenger is heavily multithreaded and consists of multiple threads and service processes. However, some communication between threads and processes required round trips, which caused more context switches than necessary. We’ve optimized our internal protocols and reduced context switching to a minimum.

The future

As stated in this blog post, this is just a glimpse of what we’ve got in store for you and as you’ve come to expect from us, we want to make sure that our findings will hold up in real life scenarios as well. With close to two years of field testing with Phusion Passenger 2, witnessing some of the most high demanding environments in web hosting of our clients, we’ve been working for the last few months now on forging this experience back into Phusion Passenger 3. Through beta testing in these high-demand Rails environments, we hope to ensure that they will give you the best experience both in an enterprise environment as well as for your personal use. Performance has been touched upon in this blog post, and in the coming period leading up to the release of Phusion Passenger 3, we’ll start to unveil bit by bit what we’ve been tinkering on for the last few months. In particular, we’re looking forward how the zero-copy I/O architecture will unfold in a real life scenario as well as the optimizations we’ve performed over the last months. Even though we’re not done yet in terms of optimizing, we will likely hit a ceiling at some point where optimizations will get harder and harder and this in particular is true if you want to retain features such as ease of use that define Phusion Passenger. One thing is for sure, we want this release to be nothing less than stunning so we encourage you to submit your wish list to us as well. We’ve likely implemented a lot of them already, but we just want to make sure that we’re not missing anything.

The Road to Passenger 3: Technology Preview 1 – Performance »

Created at: 09.06.2010 00:31, source: Phusion Corporate Blog, tagged: Phusion Passenger

This post has been moved: http://blog.phusion.nl/2010/06/10/the-road-to-passenger-3-technology-preview-1-performance-2/

Phusion Passenger 2.2.14 released »

Created at: 04.06.2010 13:36, source: Phusion Corporate Blog, tagged: Phusion Passenger

Just hours after releasing 2.2.12 some changes have been made that would warrant a new release. And so we uploaded the 2.2.13 gem, but before we could post the announcement some people contributed patches that would warrant another release. So we’ve decided to skip the 2.2.13 announcement altogether and jump straight to 2.2.14.

Changes since 2.2.12

Fixed some Rails 3 compatibility issues that were recently introduced.

About a week ago the Rails team committed a change which broke our Rails loader. This has now been fixed. Rails 3 remains to be a moving target but we’ll keep moving along with it.

[Nginx] Fix a localtime() crash on FreeBSD

This was caused by insufficient stack space for threads. Issue #499.

Added support for Rubinius

Patch contributed by Evan Phoenix.

Fixed a mistake in the SIGQUIT backtrace message.

Patch contributed by Christoffer Sawicki.

Fixed a typo that causes config/setup_load_paths.rb not to be loaded correctly.

This is related to the new Bundler support.

Phusion Passenger 2.2.12 released »

Created at: 31.05.2010 22:36, source: Phusion Corporate Blog, tagged: Phusion Passenger

Phusion Passenger is an Apache and Nginx module for deploying Ruby on Rails web applications, and is mainly focused on ease of use and stability.

Recent changes

Phusion Passenger is under constant maintenance and development. We are pleased to announce Phusion Passenger version 2.2.11. This is a bug fix release.

Improved Bundler support.

Previous versions might not be able to correctly load gems bundled by Bundler. We’ve also documented how our Bundler support works and how to override our support if you need special behavior. Please refer to the Phusion Passenger Users Guide, section "Bundler support".

Worked around some user account handling bugs in Ruby.

Issue #192.

[Apache] Suppress bogus mod_xsendfile-related error messages.

When mod_xsendfile is being used, Phusion Passenger might print bogus error messages like "EPIPE" or "Apache stopped forwarding the backend’s response" to the log file. These messages are normal, are harmless and can be safely ignored, but they pollute the log file. So in this release we’ve added code to suppress these messages when mod_xsendfile is being used. Issue #474.

[Nginx] Fixed “passenger_user_switching off” permission problems

If Nginx is running as root and passenger_user_switching is turned off, then Phusion Passenger would fail to initialize because of a permission problem. This has been fixed. Issue #458.

[Nginx] Nginx >= 0.8.38 is now supported.

Thanks to Sergey A. Osokin for reporting the problem.

[Nginx] passenger-install-nginx-module upgraded

It now defaults to installing Nginx 0.7.65 instead of 0.7.64.

Others

• Fixed some Ruby 1.9 tempfile.rb compatibility problems.
• Fixed some compilation problems on some ARM Linux platforms.

How do I upgrade to 2.2.12?

Via a gem

Please install it with the following command:

gem install passenger

Next, run:

passenger-install-apache2-module

Or, if you’re an Nginx user:

passenger-install-nginx-module

Please don’t forget to copy & paste the Apache/Nginx config snippet that the installer gives you.

Via a native Linux package

John Leach from Brightbox has kindly provided an Ubuntu 8.04 package for Phusion Passenger. The package is available from the Brightbox repository which you can find at:

http://apt.brightbox.net

Add the following line to the Third Party Software Sources:

deb http://apt.brightbox.net hardy main

(The simplest way to do that is to create a file in /etc/apt/sources.list.d/ containing the deb instruction, and then run ‘apt-get update’).

Once you’ve done this then you can install Phusion Passenger by running:

sudo apt-get install libapache2-mod-passenger

-or-

sudo apt-get install nginx-brightbox

(Note that John is currently packaging 2.2.12, so it might take a while before this release shows up in the apt repository.)

Final

Phusion Passenger is provided to the community for free. If you like Phusion Passenger, please consider sending us a donation. Thank you!