bgror

Win a Free Ticket to php|tek for Your User Group! »

Created at: 17.05.2012 00:05, source: Engine Yard Blog, tagged: community Contests events php

User group leaders, listen up! We have an extra Full Experience tickets to php|tek, and we'd like to give it to the community to use. php|tek happens May 22-25 in Chicago, Illinois, and is jam packed with PHP goodness, including an Unconference and a Hackathon. I mean, have you seen this schedule?!  Your winner is also welcome to join us at the Engine Yard JAUNT on Friday night after the conference. There is undoubtedly much awesomeness to be had next week.

If you are interested in participating, drop me a line by Thursday, May 17 and let me know. We'll randomly select a user group to receive the ticket, and you can give it out however you wish. Thumb wrestling champion? Karaoke competition? Hackathon winner? Best high fiver? It's really up to you.

As soon as you have a winner, let me know and we'll make the arrangements with the fine folks at Blue Parabola, hosts of the conference.

Orchestra Users: We Want to Hear From You! »

Created at: 27.10.2011 21:53, source: Engine Yard Blog, tagged: Contests Orchestra Platform Davey Shafik Elizabeth Naramore php

Hello! I'm Elizabeth Naramore, the new PHP Community Manager for Engine Yard/Orchestra. I started coding PHP back in 2002, and since then, I've been a pretty active member of the global PHP Community, my local PHP user group (OINK-PUG in Cincinnati, Ohio) and with PHPWomen.org, where I'm President. I also recently came from SourceForge, where I was the Community Development Manager. I'm an author, speaker, coder, and I am so happy to be back in the PHP community once again. It's something that's near and dear to my heart, and it's where I belong.

I'm also excited about getting to know the Orchestra community, which is why we're running this contest. We know you guys are doing awesome things on the Orchestra platform, and we want to hear your stories. We will pick 5 lucky winners who will not only get a write-up on our blog, they will also get a copy of Sitepoint's recently released PHP Master: Write Cutting Edge Code, co-authored by our very own Davey Shafik.

So tell us your story! What cool project are you working on? What does it do? How is the Orchestra platform helping you? What challenges have you had, and how have you overcome them? What was the inspiration for your idea?

We want to hear it all. Send your stories to me by November 15, to enaramore@engineyard.com. Looking forward to getting to know you!

Fáilte Orchestra Contest Winner: Daniel McOrmond’s Irish Orchestra.io Tweets »

Created at: 21.09.2011 00:45, source: Engine Yard Blog, tagged: community Contests events contest funconf php

It's with pleasure that I announce Daniel McOrmond as the winner of our Fáilte Orchestra competition.

Daniel's "Irish Orchestra Tweets" entry is a dead simple app that takes the tweet-stream from the @orchestra_io Twitter account, displays them against a cringe-worthy 'oirish backdrop, and translates them to what appears to be an American's interpretation of what an Irish accent sounds like.

I mentioned last week that we wanted the "smartest, wittiest or most-charming" app, and Daniel's entry was most fitting. We really felt that it was in the spirit of the competition: combining some nice cheesy Irish humour with a nice little application of technology. Nice work, Daniel! We'll see you in Ireland for Funconf!

5 subtle ways you’re using MySQL as a queue, and why it’ll bite you »

Created at: 15.09.2011 03:14, source: Engine Yard Blog, tagged: Contests events Technology

I work for Percona, a MySQL consulting company. To augment my memory, I keep a quick-reference text file with notes on interesting issues that customers ask us to solve. One of the categories of frequent problems is attempts to build a job queue in MySQL. I have so many URLs under this bullet point that I stopped keeping track anymore. Customers have endless problems with job queues in their databases. By "job queue" I simply mean some list of things they've inserted, which usually need to be processed and marked as completed. I've seen scores -- maybe hundreds -- of cases like this.

Many people realize the difficulties in building a good job queue or batch processing system, and try not to create one inside MySQL. Although the job queue is a great design pattern from the developer's point of view, they know it's often hard to implement well in a relational database. However, experience shows me that job queues sneak up in unexpected ways, even if you're a seasoned developer.

Here are some of the most common ways I've seen the job-queue design pattern creep into an application's database. Are you using MySQL for any of the following?

1. Storing a list of messages to send: whether it's emails, SMS messages, or friend requests, if you're storing a list of messages in a table and then looking through the list for messages that need to be sent, you've created a job queue.
2. Moderation, token claims, or approval: do you have a list of pending articles, comments, posts, email validations, or users? If so, you have a job queue.
3. Order processing: If your order-processing system looks for newly submitted orders, processes them, and updates their status, it's a job queue.
4. Updating a remote service: does your ad-management software compute bid changes for ads, and then store them for some other process to communicate with the advertising service? That's a job queue.
5. Incremental refresh or synchronization: if you store a list of items that has changed and needs some background processing, such as files to sync for your new file-sharing service, well, by now you know what that is.

As you can see, queues are sly; they slip into your design without you realizing it. Frankly, many of them aren't really a problem in reality. But the potential is always there, and I've observed that it's hard to predict which things will become problems. This is usually because it depends on behavior that you don't know in advance, such as which parts of your application will get the most load, or what your users will promote to their friends.

Let's dive a little bit into why job queues can cause trouble, and then I'll show you some ways to help reduce the chance it'll happen to you. The problem is usually very simple: performance. As time passes, the job queue table starts to either perform poorly, or cause other things to perform badly through collateral damage. There are three primary reasons for this:

1. Polling. Many of the job queue systems I see have one or more worker processes checking for something to do. This starts to become a problem pretty quickly in a heavily loaded application, for reasons I'm about to explain.
2. Locking. The specific implementation of the polling often looks like this: run a SELECT FOR UPDATE to see if there are items to process; if so, UPDATE them in some way to mark them as in-process; then process them and mark them as complete. There are variations on this, not necessarily involving SELECT FOR UPDATE, but often something with similar effects. The problem with SELECT FOR UPDATE is that it usually creates a single synchronization point for all of the worker processes, and you see a lot of processes waiting for the locks to be released with COMMIT. Bad implementations of this (not committing until the workers have processed the items, for example) are really horrible, but even "good" implementations can cause serious pile-ups.
3. Data growth. I can't tell you how many times I've seen email list management applications that have a single huge emails table. New emails go into the table with a "new" status, and then they get updated to mark them as sent. As time passes, these email tables can grow into millions or even billions of rows. Even though there might only be hundreds to thousands of new messages to send, that big bloated table makes all the queries really, really slow. If you combine this with polling and/or locking and lots of load on the server, you have a recipe for epic disaster.

The solutions to these problems are actually rather simple: 1) avoid polling; 2) avoid locking; and 3) avoid storing your queue in the same table as other data. Implementing these solutions can take a bit of creativity, however.

First, let's look at how to avoid polling. I wish that MySQL had listen/notify functionality, the way that PostgreSQL and Microsoft SQL Server do (just to mention two). Alas, MySQL doesn't, but you can simulate it. Here are three ideas: use GET_LOCK() and RELEASE_LOCK(), or write a plugin to communicate through Spread, or make the consumers run a SLEEP(100000) query, and then kill these queries to "signal" to the worker that there's something to do. These can work quite well, although it'd be nice to have a more straightforward solution.

Locking is actually quite easy to avoid. Instead of SELECT FOR UPDATE followed by UPDATE, just UPDATE with a LIMIT, and then see if any rows were affected. The client protocol tells you that; there's no need for another query to the database to check. Make sure autocommit is enabled for this UPDATE, so that you don't hold the resultant locks open for longer than the duration of the statement. If you don't have autocommit enabled, the application must follow up with a COMMIT to release any locks, and that is really no different from SELECT FOR UPDATE. (The rest of the work can be done with autocommit disabled; you need to enable it for only this statement.) While I'm wishing for things, I wish that SELECT FOR UPDATE had never been invented. I haven't seen a case yet where it can't be done a better way, nor have I seen a case where it has failed to cause problems

Finally, it's also really easy to avoid the one-big-table syndrome. Just create a separate table for new emails, and when you're done processing them, INSERT them into long-term storage and then DELETE them from the queue table. The table of new emails will typically stay very small and operations on it will be fast. And if you do the INSERT before the DELETE, and use INSERT IGNORE or REPLACE, you don't even need to worry about using a transaction across the two tables, in case your app crashes between. That further reduces locking and other overhead. If you fail to execute the DELETE, you can just have a regular cleanup task retry and purge the orphaned row. (Hmm, sounds like another job queue, no?) You can do much the same thing for any type of queue. For example, articles or comments that are pending approval can go into a separate table. This is really required on a large scale, although you shouldn't worry that your Wordpress blog doesn't do things this way (unless you've been hired to rewrite CNN.com using Wordpress as a backend).

Finally, and I've saved perhaps the most obvious solution for last, don't use the database at all! Use a real queueing system, such as Resque, ActiveMQ, RabbitMQ, or Gearman. Be careful, however, that you don't enable persistence to a database and choose to use MySQL for that. Depending on the queue system, that can just reintroduce the problem in a generic way that's even less optimal. Some queue systems use all of the database worst practices I enumerated above.

I hope this article has given you some insight into the variety of ways that job queues inside of MySQL can sneak up on you and bite you in the tendermost parts. And I hope you can learn to recognize and avoid this design pattern yourself, or at least implement it in a way that won't hurt you. It really is such a common problem that it's become one of the classic questions I see. Now, I'm off to check my list of pending consulting requests and see what I should work on next.

Win a Trip to Ireland on Engine Yard »

Created at: 08.09.2011 23:51, source: Engine Yard Blog, tagged: community Contests events contest funconf php

As you have heard, Engine Yard recently joined forces with Orchestra. Not only was this big news for the PHP community in general, it was huge for the Irish tech community.

What you may not know is that last year, Eamon Leonard, co-founder of Orchestra co-organised Funconf with me. Engine Yard were huge supporters of Funconf last year, and their support is even more significant this year.

I'm delighted to announce "Fáilte Orchestra" : your chance to win a once-in-a-lifetime trip to Ireland.

The prize includes:

• Return Flight to Ireland (up to $1500, depending on where you are)
• A Ticket to Funconf (includes 2 nights accommodation and all food and drink, and an amazing lineup)

To be in with a chance to win, in true Irish style, we're looking for the "smartest, wittiest or most-charming" app deployed on http://orchestra.io. Here are the rules:

• The app must be your original work but feel free to make use of frameworks and libraries (e.g. CakePHP, PEAR)
• It must be open source
• It must (obviously) be deployed on Orchestra

That's it! We want you to be creative, smart, witty, interesting. What will win this will be the app, or page, or project that makes us laugh the loudest, or drop our jaw in awe, or is just plain awesome.

To enter, send a tweet (CC: @engineyard) with the hashtag #failteorchestra with a link to your deployed Orchestra app, and a link to the Github repo.

Please get your entries in before midnight on the 15th of September. The winner will be chosen and announced here on the 16th.

You have 7 days. Let the games begin, and good luck!