Building a WordPress Cloud, Cluster Setup

This how-to will explain how to build a WordPress site on 2, 3, or more Rackspace cloud servers, with full load-balancing and redundancy.

To accomplish this, you will setup multiple web-servers and one or more mySQL servers, behind two Rackspace cloud load-balancers. One load-balancer will server all you normal user internet traffic from all the web-servers.  The other load-balancer will server only your secured traffic to your admin sites (Dashboard) from a single, master, server.  You will then set WordPress to only server the admins sites threw a secure connections, this way all uploads will be saved to a single server and may be distributed from there. This also insures that you will be able to see the newly uploaded file, even before it has a chance to propagate to the other servers. The flaw with this configuration is that if the Master server goes down, no posts may be created until the issue is resolved.

The Setup

There are meny different ways we can accomplish this.  Here I am going to show a two server setup, but you can easily expand this into as many servers as you wish.

  • Server1 – Master Database Server, Master Web Server
  • Server2 – Slave Database Server, Slave Web Server or only Slave Web Server

Building Server1

Server1 is our main server, if Server2+went down, the site would still be fully up, just slower.

This How-To assumes your using Fedora hosts for these setup’s.  To start out, we need Apache, php, mySQL installed on the server.

Advertisements

Moving to the new Rackspace Load balancer

Recently I have moved this site and others to a new load balancer solution by Rackspace (the company who also hosts the servers these sites run on).

Prior to this change, I was running a single 1024mb ram/40gb server, with this change I am now running two, fully master/master mirror 512mb/20gb servers, so the threw put should be a bit faster since there is now two.

I will hold my breath and let you know how it goes.

Installing the GIT Daemon for Read Only Access to Repoistory

The idea here is to allow anyone to download your GIT repository using the native git protocol.  This is similar to the post I wrote about Creating a secure Git repository server, but the below method allows full read only public access to the repositories.

Start out by modifying the new file git-daemon.

vim /etc/xinetd.d/git-daemon

Add the below text to the file.

# description: The git daemon offers anonymous access to git repositories
service git
{
    disable        = no
    type           = UNLISTED
    port           = 9418
    socket_type    = stream
    wait           = no
    user           = git
    server         = /usr/libexec/git-core/git-daemon
    server_args    = --inetd --export-all --base-path=/var/git
    log_on_failure += USERID
}

You may need to change the location of the git-daemon (the above example is from Fedora) and you will need to update the location of your git repository directory.

The above configuration will share ALL the git repositories in the /var/git directory.

If you do not wish for the all repositories to be public, you may remove the --export-all flag and add a empty file named git-daemon-export-ok to the git repository you wish to still share.

Auto Update Script for MiniMyth

This script will download the newest version of MiniMyth and update your tftp directory.

It first downloads the version file and compares that version number to the current install version of MiniMyth install.  If it determines that the version dose not match it will then download the current release, untar it and move the files to their current locations. It dose not delete any older versions on your server.

  • The TFTPDIR variable is the location on your tftp server where the MiniMyth files are stored.
  • The URL variable is the MiniMyth directory that the install of MiniMyth you wish to use lives.

You may also download the full directory structure with this script from my github repository or from my server.

# /bin/bash
# Matt Rude <m@mattrude.com> 11-Nov-2009
#
TFTPDIR=/tftpboot/PXEClient
URL="http://minimyth.org/download/stable/latest/"
##############################################################################
if [ -e $TFTPDIR/version ]; then
 mv $TFTPDIR/version $TFTPDIR/version.last
 rm -rf $TFTPDIR/verstion.md5
else
 touch $TFTPDIR/version.log
 touch $TFTPDIR/version.last
fi

cd $TFTPDIR
wget -nc $URL/version > /dev/null 2>&1
VER=`cat $TFTPDIR/version`
OLDVER=`cat $TFTPDIR/version.last`

if [ "$VER" = "$OLDVER" ]; then
 chown -R apache:apache $TFTPDIR/*
 exit 0
else
 echo "`date` Upgraded to version: $VER" >> version.log

 rm -rf $TFTPDIR/ram-minimyth-*.tar.bz2.md5
 wget -nc $URL/ram-minimyth-$VER.tar.bz2 > /dev/null 2>&1
 wget -nc $URL/ram-minimyth-$VER.tar.bz2.md5 > /dev/null 2>&1
 MD5STAT=`md5sum -c ram-minimyth-$VER.tar.bz2.md5 |awk ' {print $2 }'`
 if [ "$MD5STAT" = "OK" ]; then
  rm -f $TFTPDIR/kerne*
  rm -f $TFTPDIR/rootf*
  rm -fr $TFTPDIR/conf/default/theme*
  tar -xjf ram-minimyth-$VER.tar.bz2
  rm -rf ram-minimyth-$VER.*
 else
  echo "`date` Minimyth Version $VER Failed the MD5 check" >> version.log
  echo "" > $TFTPDIR/version
  exit 1
 fi

 RAMDIR=$TFTPDIR/ram-minimyth-$VER

 mkdir -p $TFTPDIR/conf/default
 cp $RAMDIR/kernel $TFTPDIR/kernel-$VER
 cp $RAMDIR/rootfs $TFTPDIR/rootfs-$VER
 cp -r $RAMDIR/themes $TFTPDIR/conf/default/themes-$VER

 ln -s kernel-$VER kernel
 ln -s rootfs-$VER rootfs
 cd $TFTPDIR/conf/default
 ln -s themes-$VER themes
 #mythtvosd --template=scroller --scroll_text="minimyth upgraded to: $VER" > /dev/null
ficd $TFTPDIR
mv version.log version.tmp
tail -2 version.tmp > version.log
rm -rf version.tmp
chown -R apache:apache $TFTPDIR/*
exit 0

Postfix: remap from addresses with a generic map

Depending on the software you are using, you may need to change the outbound (from) address of outbound email.  In this How-To, we will change the outbound email address from “apache@example.com” to “webmaster@example.com”.

To start out, go to your postfix directory and modify/create your generic file.  Your generic file will map the two addresses to each other.  At the end of your generic file add a space or tab seperated file simmiler to this.

apache@example.com         webmaster@example.com

Once you have added the entries to your generic file, you need to hash the file so postfix may quickly access it.

postmap /etc/postfix/generic

After you map has been created, you will need to add the entry to your main.cf file to tell Postfix to use your generic map.

smtp_generic_maps = hash:/etc/postfix/generic

Once you have updated your main.cf file, you need to reload postfix

postfix reload

Importing and Syncing a Subversion Repository with Git

Importing a Subversion repository is such a way that you may still sync the changes back in forth, is really just to simple. This how-to assumes you existing repository is running is SVN and you wish to switch it over to git.

Similar to how you may create a new SVN directory from a remote repo with svn, by running a command similar too:

svn checkout https://svn.example.com/svn/project_name

You may run the following to command to download the same remote repository, but then create an git repository and import the data from the svn repo.

git svn clone https://svn.example.com/svn/project_name

This will create a new directory named project_name with all your svn history.

Git is vary fast, but when it’s slow, it’s slow. The above process can take quite a bit of time (wordpress.org’s current trunk will take hours) to complete on an older (more comments not slower) repository since it checks in each comment at a time. If you don’t need the all the history of the project but just need a way to keep your self up to date as you code, you may just download the current revision tag.  So to  download SVN version 1234 and clone it into git, run:

git svn clone -r 1234 https://svn.example.com/svn/project_name

If you would like some of the history you can get that also, so if you would like say the last 234 commits run:

git svn clone -r 1000:1234 https://svn.example.com/svn/project_name

Then, to update your git repo once the SVN repository has been committed to, run

git svn fetch

That should ouput something like:

r1235 = 340621340d856d805714e9bd86fdb11777f710fe (git-svn)
 M    includes/deprecated.php
 M    includes/functions.php

You may now follow the Creating a secure git repository how-to and create a remote server.

Creating a secure Git repository server

Creating a secure git repository server is a pretty simple process.  Basically we will be using SSH to transmit the data over an encrypted channel.  SSH will handle all the authentication and data encryption.  So the first set is creating a user for git to use and creating the git users ssh key.

On your server, from a privileged account, create a user (were going to use git).

adduser git
passwd git

The configuration we will be setting up will store the actual repositories in the git users home directory.  If you don’t like it’s current location, you may modify the /etc/passwd file for your user.

Once we have the user setup, in it’s home directory we need to create our first repository.  Start out by creating the folder, then well go into it and create the git repository files.

mkdir new-project.git
cd new-project.git
git init --bare

With the repository now setup go back to your desktop/laptop system (linux/unix).

From your desktop system, create a empty repository or go to an existing git repo. If you are going to be adding an existing repo, it may not be connected to any other remote repository. if you clone it from a remote source (a directory on the same system counts as a remote source) you will need to modify the repository’s config file and remove those entry’s, look in the config file under the .git (note the ‘dot’) directory.

To create an empty repository, create a directory for the repository, go into it and init the repo.

mkdir new-project.git
cd new-project.git
git init

Since you need something in your repo, and git likes having a readme file (or gitweb dose) let’s create a readme file and commit it.

touch README
git add README
git commit -m 'Added README file, first commit'

Now that we have a commit in our repo, we can add the newly created git repo server and push our new repo to it.

We will start by adding the server to the repo’s configuration.

git remote add origin git@new-git-server.example.com:new-project.git

The above example assumes you’re server’s name is “new-git-server.example.com” and your using a project named “new-project.git” for the user “git”. But it also assumes that your repo is directly in the user git’s home directory. If you store your repo’s in a different directory, you will need to add the list folders after the colon.

After you have successfully added the server, push your new repository to the server.

git push origin master

You may now be asked for the git user’s password, enter it, and your repository should be transferred.

Also check out how to setup a ssh public/private key where no password is required, only the private key on your client system.

The above may look like a lot, but it’s really pretty simple to setup. If you have any problems or questions abut this How-To, please leave a comment or contact me.