bgror

fog 1.0 is here! »

Created at: 30.09.2011 17:00, source: Engine Yard Blog, tagged: News Open Source Technology fog

Many good things in life take longer than expected but in the end are well worth the wait. With the release of fog 1.0 we have a shining example. Begun in mid-2009 and with more than 2 years in production usage at Engine Yard, fog has seen 240,000 downloads. This is definitely a rock-solid 1.0 that has seen some real battle testing!

fog is a Ruby gem that allows you to control a number of cloud infrastructure services through a common API. fog provides the basis for building applications and cloud platforms that are infrastructure-independent, abstracting such functions as spinning up servers, connecting storage, managing DNS, etc. Internally, fog uses models to represent cloud structures analogously to the way ActiveRecord represents relational structures. Using fog makes your application or platform much more easily portable across infrastructures such as Amazon, Rackspace, and many others.

In addition to Engine Yard, fog is in use at other platform providers as well as in many core community projects such as Chef, Puppet, CarrierWave, and Paperclip. fog now supports 18 compute, 5 storage, 9 DNS, 2 CDN, and a variety of other infrastructure services. With 13 core committers, 121 contributors, and nearly 1,500 watchers, we're seeing really nice growth in participation and attention.

We believe that this release of fog is a really important step in making the cloud an easier and more powerful environment for developers to run great apps. We hope you'll give it a try, share your feedback, spread the word, and (most importantly) enjoy some great success with fog.

Get fog with a simple gem install fog. Read docs at fog.io, or grab source off GitHub at geemus/fog. Happy fogging!

JRuby GA on an Engine Yard Cloud Near You »

Created at: 28.09.2011 16:13, source: Engine Yard Blog, tagged: News Product Technology jruby jruby on appcloud jruby on engine yard cloud trinidad

We are very excited to announce the general availability of JRuby and Trinidad on Engine Yard Cloud!

Everyone who runs production Ruby web applications should be interested in JRuby. Want a list of life-changing reasons? Reasons you cannot get with any other Ruby implementation nor any other Platform as a Service?

• Major efficiency and performance boost from real concurrent native threads
• Fast in-process caching with ehcache and many other Java libraries
• Use your own existing Java libraries (put the jar files in lib/jars)
• Your development stack is the same as our production stack with the Trinidad project

Want to try Trinidad/JRuby on your laptop right this very instant? Do this. Do it now:

rvm install jruby
rvm use jruby
git clone git://github.com/engineyard/todo.git
cd todo
bundle
trinidad

Your application is now running in development mode on your laptop!

Want to try Trinidad/JRuby with your own Ruby web app this very instant? Here is a simple tutorial to get started!

Want to learn more about the awesomeness in JRuby and Trinidad (amongst other topics)? In New Orleans this week for RubyConf? Then on Saturday come and see “High Performance Ruby: Evented vs Threaded”.

We are so excited to be able to support JRuby and to finally be offering it and supporting it for big businesses and small on our Engine Yard Cloud and Engine Yard Managed platforms.

For the last two years, Engine Yard has been funding the development of JRuby and we are excited to provide fully supported JRuby for your application development. This is a major milestone for the JRuby project led by Charlie Nutter, Thomas Enebo, Nick Seiger, Hiro Asari, and David Calavera. Charlie has written a great post about the history and next steps of the project here.

Engine Yard’s Plans for Ruby 1.8.6 »

Created at: 20.09.2011 21:05, source: Engine Yard Blog, tagged: News Open Source Technology Ruby 1.8.6

Back in May of 2009, Engine Yard’s Kirk Haines took over legacy maintenance duties for Ruby 1.8.6. Since then, the core Ruby team led by Kirk has released patches, security updates, and performance tweaks for Ruby 1.8.6 benefiting Engine Yard’s customers who wished to remain on Ruby 1.8.6. However, as is often the case, all good things must come to an end. 

At RubyKaigi this year, Kirk announced that he along with the rest of the core Ruby team would no longer be supporting Ruby 1.8.6. This means that customers using Ruby 1.8.6 will no longer benefit from the latest security patches and performance improvements being developed. While your applications will continue to operate, we believe our customers would benefit most from using the latest technology components.

No immediate action is currently required but now would be a good time to start evaluating a migration to 1.8.7 or 1.9.2 for everyone currently running 1.8.6. Engine Yard will be contacting customers who currently have environments running 1.8.6 in order to discuss options and timelines. At a high level, the three different options for customers are:

• Perform the migration yourself by following these high level steps
• Contact Engine Yard Professional Services in order to have the migration performed on your behalf.
• Take no action, acknowledging that no new MRI patches or updates will be available to your environment.

When considering your migration strategy, it will also be important to keep in mind that Ruby 1.8.7 will be downgraded to support-only maintenance in the summer of 2012 and will reach end-of-life during the following summer in 2013. We recently set Ruby 1.9.2 as our default run-time and strongly recommend you consider it for your migration.

5 subtle ways you’re using MySQL as a queue, and why it’ll bite you »

Created at: 15.09.2011 03:14, source: Engine Yard Blog, tagged: Contests events Technology

I work for Percona, a MySQL consulting company. To augment my memory, I keep a quick-reference text file with notes on interesting issues that customers ask us to solve. One of the categories of frequent problems is attempts to build a job queue in MySQL. I have so many URLs under this bullet point that I stopped keeping track anymore. Customers have endless problems with job queues in their databases. By "job queue" I simply mean some list of things they've inserted, which usually need to be processed and marked as completed. I've seen scores -- maybe hundreds -- of cases like this.

Many people realize the difficulties in building a good job queue or batch processing system, and try not to create one inside MySQL. Although the job queue is a great design pattern from the developer's point of view, they know it's often hard to implement well in a relational database. However, experience shows me that job queues sneak up in unexpected ways, even if you're a seasoned developer.

Here are some of the most common ways I've seen the job-queue design pattern creep into an application's database. Are you using MySQL for any of the following?

1. Storing a list of messages to send: whether it's emails, SMS messages, or friend requests, if you're storing a list of messages in a table and then looking through the list for messages that need to be sent, you've created a job queue.
2. Moderation, token claims, or approval: do you have a list of pending articles, comments, posts, email validations, or users? If so, you have a job queue.
3. Order processing: If your order-processing system looks for newly submitted orders, processes them, and updates their status, it's a job queue.
4. Updating a remote service: does your ad-management software compute bid changes for ads, and then store them for some other process to communicate with the advertising service? That's a job queue.
5. Incremental refresh or synchronization: if you store a list of items that has changed and needs some background processing, such as files to sync for your new file-sharing service, well, by now you know what that is.

As you can see, queues are sly; they slip into your design without you realizing it. Frankly, many of them aren't really a problem in reality. But the potential is always there, and I've observed that it's hard to predict which things will become problems. This is usually because it depends on behavior that you don't know in advance, such as which parts of your application will get the most load, or what your users will promote to their friends.

Let's dive a little bit into why job queues can cause trouble, and then I'll show you some ways to help reduce the chance it'll happen to you. The problem is usually very simple: performance. As time passes, the job queue table starts to either perform poorly, or cause other things to perform badly through collateral damage. There are three primary reasons for this:

1. Polling. Many of the job queue systems I see have one or more worker processes checking for something to do. This starts to become a problem pretty quickly in a heavily loaded application, for reasons I'm about to explain.
2. Locking. The specific implementation of the polling often looks like this: run a SELECT FOR UPDATE to see if there are items to process; if so, UPDATE them in some way to mark them as in-process; then process them and mark them as complete. There are variations on this, not necessarily involving SELECT FOR UPDATE, but often something with similar effects. The problem with SELECT FOR UPDATE is that it usually creates a single synchronization point for all of the worker processes, and you see a lot of processes waiting for the locks to be released with COMMIT. Bad implementations of this (not committing until the workers have processed the items, for example) are really horrible, but even "good" implementations can cause serious pile-ups.
3. Data growth. I can't tell you how many times I've seen email list management applications that have a single huge emails table. New emails go into the table with a "new" status, and then they get updated to mark them as sent. As time passes, these email tables can grow into millions or even billions of rows. Even though there might only be hundreds to thousands of new messages to send, that big bloated table makes all the queries really, really slow. If you combine this with polling and/or locking and lots of load on the server, you have a recipe for epic disaster.

The solutions to these problems are actually rather simple: 1) avoid polling; 2) avoid locking; and 3) avoid storing your queue in the same table as other data. Implementing these solutions can take a bit of creativity, however.

First, let's look at how to avoid polling. I wish that MySQL had listen/notify functionality, the way that PostgreSQL and Microsoft SQL Server do (just to mention two). Alas, MySQL doesn't, but you can simulate it. Here are three ideas: use GET_LOCK() and RELEASE_LOCK(), or write a plugin to communicate through Spread, or make the consumers run a SLEEP(100000) query, and then kill these queries to "signal" to the worker that there's something to do. These can work quite well, although it'd be nice to have a more straightforward solution.

Locking is actually quite easy to avoid. Instead of SELECT FOR UPDATE followed by UPDATE, just UPDATE with a LIMIT, and then see if any rows were affected. The client protocol tells you that; there's no need for another query to the database to check. Make sure autocommit is enabled for this UPDATE, so that you don't hold the resultant locks open for longer than the duration of the statement. If you don't have autocommit enabled, the application must follow up with a COMMIT to release any locks, and that is really no different from SELECT FOR UPDATE. (The rest of the work can be done with autocommit disabled; you need to enable it for only this statement.) While I'm wishing for things, I wish that SELECT FOR UPDATE had never been invented. I haven't seen a case yet where it can't be done a better way, nor have I seen a case where it has failed to cause problems

Finally, it's also really easy to avoid the one-big-table syndrome. Just create a separate table for new emails, and when you're done processing them, INSERT them into long-term storage and then DELETE them from the queue table. The table of new emails will typically stay very small and operations on it will be fast. And if you do the INSERT before the DELETE, and use INSERT IGNORE or REPLACE, you don't even need to worry about using a transaction across the two tables, in case your app crashes between. That further reduces locking and other overhead. If you fail to execute the DELETE, you can just have a regular cleanup task retry and purge the orphaned row. (Hmm, sounds like another job queue, no?) You can do much the same thing for any type of queue. For example, articles or comments that are pending approval can go into a separate table. This is really required on a large scale, although you shouldn't worry that your Wordpress blog doesn't do things this way (unless you've been hired to rewrite CNN.com using Wordpress as a backend).

Finally, and I've saved perhaps the most obvious solution for last, don't use the database at all! Use a real queueing system, such as Resque, ActiveMQ, RabbitMQ, or Gearman. Be careful, however, that you don't enable persistence to a database and choose to use MySQL for that. Depending on the queue system, that can just reintroduce the problem in a generic way that's even less optimal. Some queue systems use all of the database worst practices I enumerated above.

I hope this article has given you some insight into the variety of ways that job queues inside of MySQL can sneak up on you and bite you in the tendermost parts. And I hope you can learn to recognize and avoid this design pattern yourself, or at least implement it in a way that won't hurt you. It really is such a common problem that it's become one of the classic questions I see. Now, I'm off to check my list of pending consulting requests and see what I should work on next.

Submit a patch for Rails on Github using “fork and edit button” »

Created at: 11.09.2011 10:49, source: Ruby Rockers, tagged: Solutions Technology contributions github rails

Hi Folks,

I have recently written about my Rails Contributions experience here.

I see that now days contribution is Rails is increased. People love to contribute in Open Source projects. And the way should be easy not painful.

I found that some people are struggling in submitting Pull Requests in Rails on Github. So i thought of writing the same to help them.

Some people are new to git and they don’t know much about git… or sometime patch is very small and they want to use Github’s “fork and edit this file” feature to submit a patch.

And some people who are good in Rails will do it in more easy manner. And easy to open multiple pull request at a time from the same branch.

I am writing some steps to use Github’s fork and edit feature to open pull request.

Open up your project on Github open up file where you want to change

Change your desired things into the file. You can only change one file in one commit

Propese your file changes! Here you can write about your changes. Give some references about issue. Can see the file changed. Can see the commits which you have made.

Change commit is the most important part. The reason is when people are doing changes in specific branch. Let’s take a example of Rails. If you are going to submit a patch against Rails 3.0.X version then you must have to choose 3-0-stable branch instead of master in Base branch.

After updating the commit range you can submit the Pull Request and also must see the File Changed.

Use this feature when you are fixing any typos, updating any docs. For submitting any code changes you should be running tests.  

Feel free to ask me if you face any problem in doing these things. We want more contributions!

You can tweet me @arunagw or catch me on my email any time.

Cheers!