bgror

Tutorial: setting up Gitlab on Debian 6 »

Created at: 21.04.2012 17:24, source: Phusion Corporate Blog, tagged: Software

We host many git repositories on our servers using SSH and file system ACLs for access control. However, having been spoiled by Github for so long, this method feels archaic and cumbersome. While Github provides private repositories, sometimes it’s just not an option because there are some things that may never leave the organization. If you still want to have a fancy web interface for your Git repositories, then there are several alternatives:

• Use Github Enterprise.
• Use Gitorious.
• Use Gitlab.

Gitlab is an excellent option. It’s not as fully featured as Github, but like Gitorious it is open source. We’ve found that it’s not only more user friendly than Gitorious but also easier to install.

This tutorial teaches you how to setup Gitlab on Debian 6, Ruby 1.9.3, Phusion Passenger and Nginx. It assumes that Gitlab and the git repositories are hosted on the same machine. Your Gitlab installation will be protected by SSL. Your users will be able to pull from and push to your repositories using the ssh:// protocol, but they won’t have actual shell access and you don’t need to have separate system accounts for each user to control access. Gitlab (or to be more correct, gitolite, which Gitlab uses) manages access without system accounts.

Step 1: Setup sudo

In this tutorial we’re going to use sudo to switch between user accounts. Sudo is not installed by default on Debian so install it if you don’t already have it:

$ su
# apt-get update
# apt-get install sudo

Now add your own account to /etc/sudoers:

# visudo

Add something like this to the end of the file:

your_username_here   ALL=(ALL) ALL

Step 2: Install the base software

Install basic dependencies:

$ sudo apt-get update
$ sudo apt-get install build-essential git-core wget curl gcc checkinstall libxml2-dev libxslt-dev sqlite3 libsqlite3-dev libcurl4-openssl-dev libreadline-dev libc6-dev libssl-dev libmysql++-dev make build-essential zlib1g-dev libicu-dev redis-server openssh-server python-dev python-pip libyaml-dev

Install Pygments, which Gitlab needs for syntax highlighting:

$ sudo pip install pygments

Gitlab needs the sendmail command in order to send emails (for things like lost password recovery). This command is provided by the exim4, postfix and sendmail packages but you can only have one of them installed. If you don’t already have one of them installed, then we recommend postfix.

First, check whether you already have the sendmail command:

$ ls /usr/sbin/sendmail

If you get a ‘file not found’ then install Postfix:

$ sudo apt-get install postfix

Step 3: Install gitolite

We need gitolite. Gitolite is a tool used by Gitlab for managing access control to git repositories. It works by providing a bunch of config files in which you can register users and their public keys. Gitolite modifies ~/.ssh/authorized_keys appropriately based on the config files’ contents. Gitolite is not supposed to run as root; instead, it runs as a single user.

Install gitolite:

$ sudo apt-get install gitolite

The Debian package will automatically create an account called gitolite. Gitlab controls gitolite by logging into gitolite’s user account through SSH, so we want to create a passwordless SSH keypair for this. This SSH keypair is what we call the gitolite admin key.

$ sudo -u gitolite ssh-keygen
...
Enter file in which to save the key (/var/lib/gitolite/.ssh/id_rsa):    <--- enter nothing here and press enter
...
Enter passphrase (empty for no passphrase):    <--- enter nothing here and press enter
Enter same passphrase again:                   <--  enter nothing here and press enter
...
Your public key has been saved in /var/lib/gitolite/.ssh/id_rsa.pub.
...

Now you need to tell gitolite that you want to use this key as the admin key. First, let’s print the content of the public key and copy it to the clipboard:

$ sudo -u gitolite cat /var/lib/gitolite/.ssh/id_rsa.pub
(now select the output in your terminal and copy it to the clipboard)

Now let’s configure gitolite:

$ sudo dpkg-reconfigure gitolite

When asked for a username, keep it at the default:

When asked for a repository path, keep it at the default:

When asked for the admin key, paste the contents of the key you copied to clipboard:

Finally, we need to set the REPO_MASK option to 0007.

$ sudo -u gitolite vi /var/lib/gitolite/.gitolite.rc

Look for:

$REPO_UMASK = 0077;         # gets you 'rwx------'

Change it to:

$REPO_UMASK = 0007;  # rwxrwx---

Tighten security

We think the default gitolite home directory security is a bit weak: everybody on the system can access the config files and the repository files.

$ ls -ld /var/lib/gitolite
drwxr-xr-x 5 gitolite gitolite 4096 Apr 21 08:40 /var/lib/gitolite

Let’s tighten it up a little bit so that only gitolite can access the files:

$ sudo -u gitolite chmod o-rx /var/lib/gitolite

Step 4: Install Ruby 1.9

Gitlab requires Ruby 1.9 and their developers recommend Ruby 1.9.2, but we don’t recommend 1.9.2 because it has a lot of critical bugs compared to 1.9.3. We’ve been running Gitlab in production on Ruby 1.9.3 for a while now and so far everything works great, so in this tutorial we’ll teach you how to install Ruby 1.9.3. But feel free to install a different version if you disagree with us.

Debian does not provide a recent enough version of Ruby through apt, so we need to install it manually.

$ wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p194.tar.gz
$ tar xzvf ruby-1.9.3-p194.tar.gz
$ cd ruby-1.9.3-p194
$ ./configure
$ make
$ sudo make install

Install Bundler:

$ sudo gem install bundler

Step 5: Install Gitlab

Download the source

$ cd /opt
$ sudo git clone git://github.com/gitlabhq/gitlabhq.git
$ sudo chown -R gitolite:gitolite gitlabhq
$ cd gitlabhq
$ sudo -u gitolite git checkout 9af14e4bda35

(You can also checkout the stable branch for the latest stable version, but this tutorial is written specifically for commit 9af14e4bda35.)

Configure

Setup the database configuration. Make sure you fill in the database details under the production section.

$ sudo -u gitolite cp config/database.yml.example config/database.yml
$ sudo -u gitolite vi config/database.yml

Setup other Gitlab configuration.

$ sudo -u gitolite cp config/gitlab.yml.example config/gitlab.yml
$ sudo -u gitolite vi config/gitlab.yml

• Set email.from to what should be filled in the ‘From’ header in emails.
• Set email.to to the domain name on which you want to host Gitlab, without the protocol scheme and without the path.
• Set email.protocol to https.
• Set git_host.admin_uri to gitolite@localhost:gitolite-admin.
• Set git_host.base_path to /var/lib/gitolite/repositories/.
• Set git_host.host to the system’s SSH domain name.
• Set git_host.git_user to gitolite.

Now tighten up security:

$ sudo -u gitolite chmod o-rwx config/*.yml

Install gems and setup database

$ sudo -u gitolite -H bundle install --without development test --deployment
$ sudo -u gitolite bundle exec rake db:setup RAILS_ENV=production
$ sudo -u gitolite bundle exec rake db:seed_fu RAILS_ENV=production

This last command will output an initial administrator account’s username and password. Write it down somewhere.

Check status

$ sudo -u gitolite bundle exec rake gitlab:app:status RAILS_ENV=production

You should get all YES:

Starting diagnostic
config/database.yml............exists
config/gitlab.yml............exists
/home/git/repositories/............exists
/home/git/repositories/ is writable?............YES
remote: Counting objects: 603, done.
remote: Compressing objects: 100% (466/466), done.
remote: Total 603 (delta 174), reused 0 (delta 0)
Receiving objects: 100% (603/603), 53.29 KiB, done.
Resolving deltas: 100% (174/174), done.
Can clone gitolite-admin?............YES
UMASK for .gitolite.rc is 0007? ............YES

Run a Resque worker

This worker daemon is for processing background tasks.

$ sudo -u gitolite bundle exec rake environment resque:work QUEUE=* RAILS_ENV=production BACKGROUND=yes

Step 6: Generate an SSL certificate

Generate a self-signed certificate. You may enter arbitrary certificate information but the Common Name must be set to the domain name on which you want to host Gitlab.

$ cd /opt/nginx/conf
$ sudo openssl req -new -x509 -nodes -days 3560 -out gitlab.crt -keyout gitlab.key
$ sudo chmod o-r gitlab.key

Installing the certificate into your browser

This self-signed certificate by itself provides encryption, but not authentication, making it vulnerable to man-in-the-middle attacks. To solve this problem, install this certificate into your browser.

Here’s how you do it on Google Chrome on OS X. First, copy the certificate to your local machine:

$ scp your-server.com:/opt/nginx/conf/gitlab.crt .

Now open Keychain Access. Under the “Keychains” list, select “System”. Then click on the “+” button and add the certificate file.

Step 7: Deploy to Phusion Passenger and Nginx

Install Nginx and Phusion Passenger:

$ sudo gem install passenger --no-rdoc --no-ri
$ sudo passenger-install-nginx-module
(Choose option 1 and install Nginx to /opt/nginx)

Now edit the Nginx config file

$ sudo vi /opt/nginx/conf/nginx.conf

and add this to the http block:

# This is a normal HTTP host which redirects all traffic to the HTTPS host.
server {
    listen 80;
    server_name gitlab.yourdomain.com;
    root /nowhere;
    rewrite ^ https://gitlab.phusion.nl$request_uri permanent;
}

# The actual Gitlab HTTPS host.
server {
    listen 443;
    server_name gitlab.yourdomain.com;
    root /opt/gitlabhq/public;
    ssl on;
    ssl_certificate gitlab.crt;
    ssl_certificate_key gitlab.key;
    add_header Strict-Transport-Security "max-age=315360000";
    location / {
        passenger_enabled on;
    }
    location /assets {
        expires max;
        add_header Cache-Control public;
        passenger_enabled on;
    }
}

Now start Nginx:

$ sudo /opt/nginx/sbin/nginx

And viola, you’re up and running! You can access Gitlab through https://gitlab.yourdomain.com/.

Already have Phusion Passenger installed and already running Ruby 1.8?

Phusion Passenger for Nginx does not yet support multiple Ruby versions (but it will in 3.2). If you already have Ruby apps deployed on your server using Phusion Passenger, and those apps require Ruby 1.8, then you’ll have to run Gitlab on Ruby 1.9 using Phusion Passenger Standalone, and connect it to Nginx through a reverse proxy setup.

$ cd /opt/gitlabhq
$ sudo -u gitolite passenger start -d -e production --max-pool-size 2 -S gitlab.socket

The HTTPS server block should then look like this:

upstream gitlab {
    server unix:/opt/gitlabhq/gitlab.socket;
}

# The actual Gitlab HTTPS host.
server {
    listen 443;
    server_name gitlab.yourdomain.com;
    root /opt/gitlabhq/public;
    ssl on;
    ssl_certificate gitlab.crt;
    ssl_certificate_key gitlab.key;
    add_header Strict-Transport-Security "max-age=315360000";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Ssl on;
    location / {
        if (!-f $request_filename) {
            proxy_pass http://gitlab;
            break;
        }
    }
    location /assets {
        expires max;
        add_header Cache-Control public;
        if (!-f $request_filename) {
            proxy_pass http://gitlab;
            break;
        }
    }
}

Donate

Gitlab started a donation campaign a few weeks ago. It is excellent, high-quality open source software so if you like it, consider giving them a donation to support their development.

Don’t be perfectionist: big code dumps suck »

Created at: 14.04.2012 00:44, source: Phusion Corporate Blog, tagged: Software

We’ve been developing Phusion Passenger 3.2 on local repositories for a while now. We didn’t want to show the changes to the world until we know that it’s at least somewhat ready for public consumption. For a long time 3.2 didn’t even fully compile. The end results speak for themselves: Phusion Passenger 3.2 introduces quite a large number of changes. And yet there’s something that does not feel quite right here. Not only was there a large time gap between 3.0 and 3.2, we also feel that we haven’t properly communicated our development progress to the public, which could possibly have resulted in the perception that development was slow.

We have been too perfectionist. Dr Nic was right in booing our development method.

From an open source perspective as well, development behind closed doors is not the way to go. We plan on publishing changes in a more gradual manner in the future.

Don’t Mistake Meetings for Process »

Created at: 08.04.2012 22:02, source: Mike Perham, tagged: Software

I’ve joined several teams over the last few years, including one or two that effectively had no development process. Even with the smartest engineers in the world, the latter always show the same symptoms:

• You have no idea what your coworkers are working on right now.
• You don’t know what you will be working on next week.

Does this sound like your team? The problem is almost always the same: lack of communication. The point of development process is simple: ensure team communication. The simplest effective development process for your team ensures that everyone is in sync about the present and near-future goals for the team.

Process gets a bad reputation because often meetings are used as a substitute for process. Since lack of communication is the problem, we should just throw everyone in a room together to talk, right? Good, high-bandwidth meetings can be very useful but just as important are tools which allow the team to communicate on their own schedule (think issue trackers, code review tools, group chat, a pinboard with index cards, etc). Your process should be a mixture of synchronous, high-bandwidth meetings and asynchronous, low-bandwidth tools. There’s no right mixture but your process must have two mandatory properties:

1. Everyone must sign off on it as “the way”.
2. Everyone must actually follow it.

I generally try for two team meetings per week: a Monday morning sync about upcoming work for this week and next week and Friday afternoon demo of work accomplished that week (the demo almost always finds bugs and issues to be fixed next week). Monday morning kicks the work week into high gear and Friday afternoon winds down the work so everyone can enjoy their weekend.

Great, which process should you use? Agile, XP, Scrum, Kanban? I treat them like web frameworks: pick one, learn it well and modify it as necessary for your own needs. The one booklet that I keep going back to year after year is this booklet on Scrum and XP practices offered by InfoQ. It’s a very practical, quick read and low tech enough that you can implement it in 2007 or 2017.

Deleting Duplicate Rows in MySQL »

Created at: 02.03.2012 18:48, source: Mike Perham, tagged: Software

You have a table with duplicate rows – somehow a unique index didn’t get created and a bug has added duplicate records to your table. A pox upon that bug!

Here’s two easy ways to clean out that table quickly.

1) Use ALTER IGNORE on MySQL 5.1+

MySQL will allow you to create a unique index on a table with duplicate records with its IGNORE SQL extension:

ALTER IGNORE TABLE 'SHIPMENTS' ADD UNIQUE INDEX (CART_ID, TRACKING_NUMBER)

Duplicates will be deleted.

2) Recreate the table with GROUP BY

    execute 'CREATE TABLE shipments_deduped like shipments;'
    execute 'INSERT shipments_deduped SELECT * FROM shipments GROUP BY cart_id, tracking_number;'
    execute 'RENAME TABLE shipments TO shipments_with_dupes;'
    execute 'RENAME TABLE shipments_deduped TO shipments;'
    add_index :shipments, [:cart_id, :tracking_number], :unique => true
    execute 'DROP TABLE shipments_with_dupes;'

Recreating the table is much, much faster than trying to delete the records in the existing table and doesn’t lock the existing table, making your application downtime minimal.

Getting iChat to automatically reconnect »

Created at: 31.12.2011 06:32, source: Mike Perham, tagged: Software

I’ve noticed a problem with iChat for the last year or two: if your network drops, you stay Disconnected until you manually tell iChat to log back in. That’s pretty lame, my Comcast cable drops several times a day so I need something a little more robust than that. I found a workaround: use cron to do the work for you. Fire up a Terminal, run crontab -e and put this in it:

*/5 * * * * osascript -e 'tell application "System Events" to if (processes whose name is "iChat") exists then tell application "iChat" to log in'

This uses AppleScript to tell iChat to log in every 5 minutes. Now if the network drops, you’ll only be disconnected a few minutes.