Configuring WordPress to use Memcached

When using WordPress self hosted software it’s generally a good idea to cache as much as possible.  Object Caching allows you store parts of the pages in memory for quicker retrieval, since the server will not need to look as much up from the SQL database.

Installing the needed parts

To start out, you will need to have memcached installed on your server. If your using Fedora, you may install memcached via Yum as follows.

yum -y install memcached php-pecl-memcached perl-Cache-Memcached

Configuring memcached

After installing memcached you need to configure it. If working in Fedora, and using the Yum install as talked about above, you will need to change the memcached confuration by modifying it’s sysconfig file located at /etc/sysconfig/memcached.

A default configuration may look like this.

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1"

If you would like to share this memcached server with other webservers, change the address from 127.0.0.1 to the server’s actual address.

To set memcached to start automatically when the server get’s rebooted, run:

chkconfig memcached on

And of course, don’t for get to start it

service memcached start

Configuring WordPress

After memcached is installed, you need to configure the WordPress side.

Next you should install the Memcached Object Cache plugin, but be careful, this is not a normal plugin.  You should not activate this plugin as you would with a normal plugin, but instead download it as you normally would, but then you need to copy the object-cache.php file to your wp-content folder.

From the root of your WordPress install, run the following:

cp wp-content/plugins/memcached/object-cache.php wp-content/

Now we need to configure WordPress to use the memcached server. Add the following near the end of your wp-config.php file.

global $memcached_servers;
$memcached_servers = array('default' => array('127.0.0.1:11211'));

Note that were using the same server and port (127.0.0.1:11211) as was configured above while we were setting up memcached.

Checking in on memcached

Memcached is one of those things that just sort of runs. There’s not much direct feed back, besides the speed difference on your site.

One quick way is memcache-top. memcache-top will show you the current status of your memcached server.

To install, run the following.

wget http://memcache-top.googlecode.com/files/memcache-top-v0.6
chmod +x memcache-top-v0.6
./memcache-top-v0.6

Running ./memcache-top-v0.6 will assumed the default configuration we used here.

Advertisements

Dovecot’s Quota Plugin

The below assumes your using mdbox so dirsize or dict:sql will be the fastest. Since we already have sql setup, were going to go with that.

Enabling the Quota Plugin

mail_plugins = $mail_plugins quota

protocol imap {
  # Space separated list of plugins to load (default is global mail_plugins).
  mail_plugins = $mail_plugins imap_quota
}

plugin {
  # SQL backend:
  quota = dict:User quota::proxy::quota
  quota_rule = *:storage=1024M:messages=100000
  quota_rule2 = Trash:storage=+20%%
  quota_rule3 = Junk:storage=+20%%
}dict {
  quota = mysql:/etc/dovecot/dovecot-dict-mysql.conf
}

The dovecot-dict-mysql.conf file should containe something like this

connect = host=localhost dbname=postfix user=postfix password=postfix
map {
  pattern = priv/quota/storage
  table = quota
  username_field = username
  value_field = bytes
}
map {
  pattern = priv/quota/messages
  table = quota
  username_field = username
  value_field = messages
}

And lastly, add the following table to your email servers database in mySQL.

CREATE TABLE quota (
  username varchar(100) not null,
  bytes bigint not null default 0,
  messages integer not null default 0,
  primary key (username)
);

Building a WordPress Cloud, Cluster Setup

This how-to will explain how to build a WordPress site on 2, 3, or more Rackspace cloud servers, with full load-balancing and redundancy.

To accomplish this, you will setup multiple web-servers and one or more mySQL servers, behind two Rackspace cloud load-balancers. One load-balancer will server all you normal user internet traffic from all the web-servers.  The other load-balancer will server only your secured traffic to your admin sites (Dashboard) from a single, master, server.  You will then set WordPress to only server the admins sites threw a secure connections, this way all uploads will be saved to a single server and may be distributed from there. This also insures that you will be able to see the newly uploaded file, even before it has a chance to propagate to the other servers. The flaw with this configuration is that if the Master server goes down, no posts may be created until the issue is resolved.

The Setup

There are meny different ways we can accomplish this.  Here I am going to show a two server setup, but you can easily expand this into as many servers as you wish.

  • Server1 – Master Database Server, Master Web Server
  • Server2 – Slave Database Server, Slave Web Server or only Slave Web Server

Building Server1

Server1 is our main server, if Server2+went down, the site would still be fully up, just slower.

This How-To assumes your using Fedora hosts for these setup’s.  To start out, we need Apache, php, mySQL installed on the server.

Configuring Spamassassin (MySQL & Virtual Users)

With this setup you will be saving the user preferences and bayes tokens in a MySQL data source. But, while scanning a message if spamd is unable to connect to the server specified in user_scores_dsn (below) or an error occurs when querying the SQL server then spam checking will not be performed on that message.

First we need to build the database table. You may download mine below.

Then import the file into the spamassassin table on your MySQL server.

wget http://wiki.mattrude.com/images/7/7a/Spamassassin.sql
wget http://wiki.mattrude.com/images/a/a7/Spamassassin_userpref.sql
mysql -h localhost -u postfix -ppostfix spamassassin < Spamassassin.sql
mysql -h localhost -u postfix -ppostfix spamassassin < Spamassassin_userpref.sql

To set the version number in the database, run the following.

echo "INSERT INTO bayes_global_vars VALUES ('VERSION','3');" |mysql -u postfix -ppostfix spamassassin
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'required_score', '5.0', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'rewrite_header Subject', '[SPAM]', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'rewrite_header Subject', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'report_safe', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'trusted_networks', '192.168.1.0/24', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_bayes', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'bayes_auto_learn', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'skip_rbl_checks', '0', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_razor2', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_pyzor', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'ok_locales', '1', NULL);

Now setup the configuration.

  • /etc/mail/spamassassin/local.cf
required_score                  5.0
rewrite_header Subject          [SPAM]
trusted_networks                192.168.1.0/24
report_safe                     1
use_bayes                       1
bayes_auto_learn                1
skip_rbl_checks                 0
use_razor2                      1
use_pyzor                       1
ok_languages                    enrewrite_header Subject
user_scores_dsn                 DBI:mysql:spamassassin:localhost
user_scores_sql_username        postfix
user_scores_sql_password        postfix
auto_whitelist_factory          Mail::SpamAssassin::SQLBasedAddrList
user_awl_dsn                    DBI:mysql:spamassassin:localhost
user_awl_sql_table              awl
user_awl_sql_username           postfix
user_awl_sql_password           postfix
bayes_store_module              Mail::SpamAssassin::BayesStore::MySQL
bayes_sql_dsn                   DBI:mysql:spamassassin:localhost
bayes_sql_username              postfix
bayes_sql_password              postfix

The spamd server will not pay attention to SQL preferences by default, even with user_scores_dsn set in the config files. You must startup spamd with the proper options (ie -q or -Q). If the user_scores_dsn option does not exist, SpamAssassin will not attempt to use SQL for retrieving users’ preferences.

SpamAssassin needs to be ran with the options similar to this:

/usr/bin/spamd -d -x -q -Q -u nobody -r /var/run/spamd.pid

I believe the best way of doing this is modify your /etc/init.d/spamassassin init file and change SPAMDOPTIONS to:

SPAMDOPTIONS="-d -m5 -x -q -Q -u nobody"

make sure /etc/sysconfig/spamassassin dosn’t override your settings run the below command to confirm spamassassin is running correctly

ps -eaf |grep spamd

WordPress Auto Backup Script

A quick backup script for WordPress. If you pass a config file containing the block of variables, the script will auto run for your install of WordPress.

# /bin/bash
BKNAME=
DIR=
DBHOST=
DBNAME=
DBUSER=
DBPASS=

# Backup Varibles
BKDIR=

# Restore Variables
RSDIR=
RSARDIR=
RSUSER=
RSGROUP=

OFFSITE=

# Offsite Rsync
RSYNCHOST=
RSYNCUSERNAME=
RSYNCPASSWORD=

# Offiste SCP
SCPUSER=
SCPHOST=
SCPDIR=

if [ `date +%e` =  1 ]; then
 DAY=`date +%b`
 DATE=$DAY.`date +%Y-%m-%d`
else
 DAY=`date +%w`
 DATE=$DAY.`date +%Y-%m-%d`
fi

if [ -e $1 ]; then
 source $1
 CONFIGSTATUS="Using config file found at $1"
else
 echo "No Config file found"
 exit
fi

if [ ! -e $DB_HOST ]; then
  DBHOST=`grep "DB_HOST" $DIR/wp-config.php |sed "s/define('DB_HOST', '//" |sed "s/');//"`
  DBNAME=`grep "DB_NAME" $DIR/wp-config.php |sed "s/define('DB_NAME', '//" |sed "s/');//"`
  DBUSER=`grep "DB_USER" $DIR/wp-config.php |sed "s/define('DB_USER', '//" |sed "s/');//"`
  DBPASS=`grep "DB_PASS" $DIR/wp-config.php |sed "s/define('DB_PASSWORD', '//" |sed "s/');//"`
fi
BKDIRNAME=$BKNAME

if [ `date +%e` = 1 ]; then
 ARCHIVE=1
else
 if [ `date +%w` = 0 ]; then
  ARCHIVE=1
 else
  ARCHIVE=0
 fi
fi

if [ ! -e $BKDIR ];then
 mkdir -p $BKDIR
fi

chown -R root:root $DIR/
chown -R apache:apache $DIR/wp-content $DIR/wp-admin/update.php
chown apache:apache $DIR

if [ -e $DIR/sitemap.xml ]; then
  chown apache:apache $DIR/sitemap.xml
fi

if [ -e $DIR/sitemap.xml.gz ]; then
  chown apache:apache $DIR/sitemap.xml.gz
fi
case "$2" in
restore)
  rm -rf $RSARDIR/$BKNAME.$DAY.*
  rm -rf $RSDIR/$BKNAME
  rm -rf $DIR/$BKNAME.$DAY.*
  rm -rf $DIR/$DBNAME.*
  if [ ! -e $RSARDIR ]; then
   mkdir -p $RSARDIR
  fi
  export RSYNC_PASSWORD=$RSYNCPASSWORD
  rsync -rvzht --delete --stats $RSYNCUSERNAME@$RSYNCHOST::ibackup/odin/$BKNAME/$BKNAME.$DAY.* $RSARDIR/ --port=45873 >> $DIR/$BACKUPNAME.log 2>&1
  cd $RSARDIR
  md5sum -c $BKNAME.$DATE.tgz.md5 > /dev/null 2>&1
  RESTOREMD5=`echo $(($?))`
  if [ $RESTOREMD5 = 0 ]; then
   cd $RSDIR/
   cp $RSARDIR/$BKNAME.$DATE.tgz $RSDIR/$BKNAME.$DATE.tgz
   tar -xzf $RSDIR/$BKNAME.$DATE.tgz
   mv $RSDIR$DIR $RSDIR
   cd $RSDIR/$BKNAME
   md5sum -c $DBNAME.$DATE.sql.md5 > /dev/null 2>&1
   RESTORESQLMD5=`echo $(($?))`
   if [ $RESTORESQLMD5 = 0 ]; then
    cd $RSDIR
    mysql -u $DBUSER -p$DBPASS $DBNAME > $DIR/$BKNAME.$DATE.log 2>&1
  done
  mysqldump -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME > $DIR/$DBNAME.$DATE.sql 2> $DIR/$BKNAME.$DATE.log
  a=$?
  mysqldump -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME --xml > $DIR/$DBNAME.$DATE.xml 2> $DIR/$BKNAME.$DATE.log
  a2=$?
  cd $DIR/
  md5sum $DBNAME.$DATE.sql $DBNAME.$DATE.xml > $DBNAME.$DATE.sql.md5 2>> $DIR/$BKNAME.$DATE.log
  b=$?
  SQLERROR=`echo $(($a+$a2+$b))`
  if [ $SQLERROR != 0 ]; then
   BKNAME=$BKNAME-NOSQL
   echo "Exiting on SQL error"
   echo "DBHOST $DBHOST"
   echo "DBNAME $DBNAME"
   echo "DBUSER $DBUSER"
   echo "DBPASS $DBPASS"
   exit 1
  fi
  tar -czf $BKDIR/$BKNAME.$DATE.tgz --totals $DIR >> $DIR/$BKNAME.$DATE.log 2>&1
  c=$?
  cd $BKDIR/
  md5sum $BKNAME.$DATE.tgz > $BKNAME.$DATE.tgz.md5 2>> $DIR/$BKNAME.$DATE.log
  d=$?
  TOTAL=`ls -lh $BKDIR/$BKNAME.$DATE.tgz |awk '{ print $5 }'`
  ERROR=`echo $(($a+$a2+$b+$c+$d))`
  if [ $ERROR = 0 ]; then
   STATUS=Good
  else
   STATUS=Failed
  fi

  echo "" >> $DIR/$BKNAME.$DATE.log
  echo "Starting GPG encryption" >> $DIR/$BKNAME.$DATE.log
  for a in $GPGKEY
  do
   GPGKEYD="$GPGKEYD -r $a"
  done
  #gpg -e -r $GPGKEY $BKDIR/$BKNAME.$DATE.tgz
  echo "" >> $DIR/$BKNAME.$DATE.log
  if [ $ARCHIVE = '0' ]; then
   case "$OFFSITE" in
    rsync|RSYNC)
      export RSYNC_PASSWORD=$RSYNCPASSWORD
      rsync -rvzht --delete --stats $BKDIR/ $RSYNCUSERNAME@$RSYNCHOST::ibackup/odin/$BKDIRNAME --port=45873 --exclude=Archive/ >> $DIR/$BACKUPNAME.log 2>&1
    ;;

    scp|SCP)
      echo "" >> $DIR/$BKNAME.$DATE.log
      echo "Starting SCP Transmition" >> $DIR/$BKNAME.$DATE.log
      ssh $SCPUSER@$SCPHOST "mkdir -p $SCPDIR"
      ssh $SCPUSER@$SCPHOST "rm -f $SCPDIR/$BKNAME.$DAY.*"
      scp -q $BKDIR/$BKNAME.$DATE.tgz $SCPUSER@$SCPHOST:$SCPDIR/ 2>> $DIR/$BKNAME.$DATE.log
      scp -q $BKDIR/$BKNAME.$DATE.tgz.md5 $SCPUSER@$SCPHOST:$SCPDIR/ 2>> $DIR/$BKNAME.$DATE.log
      #ssh $SCPUSER@$SCPHOST "chmod 666 $SCPDIR/$BKNAME.$DAY.*"
      echo "" >> $DIR/$BKNAME.$DATE.log
    ;;
   esac
  fi

  echo "final error status is: $ERROR" >> $DIR/$BKNAME.$DATE.log
  sed '/tar: Removing leading */d' $DIR/$BKNAME.$DATE.log > $BKDIR/backuptmp.log
  #echo "INSERT INTO Backup_Log ( System, Backup_Job, Label, Output, Bytes, Status, Log ) VALUES ('`hostname -s`', 'Wiki', '$DIR', '$BKNAME.$DATE', '$TOTAL', '$STATUS', '`cat $BKDIR/backuptmp.log`');" |mysql -t -h localhost -u backup Status
  rm -rf $BKDIR/backuptmp.log
  rm -rf $DIR/$DBNAME.$DAY.*
  rm -rf $DIR/$BKNAME.$DAY.*
  ;;
*)
  echo "The valid commands are backup & restore"
  exit 1
  ;;
esac

Converting a MediaWiki database from MySQL to SQLite

So the plane here is to convert a fully working MediaWiki install running on MySQL to run on SQLite instead.  To do this you will need to install a 2nd MediaWIKI install on a test or development system.  Once you are done you can move your new MediaWiki install to where every you would like.

Backup up your Old MediaWiki Installation

To start, from within your working MediaWiki install, first back up your data.

php maintenance/dumpBackup.php --full --uploads > wiki-backup.xml

This will create a file named wiki-backup.xml in your MediaWiki’s root directory, copy that file to a safe place.  We aren’t going to touch the MySQL database until were done, but it’s always a good idea to have backups safe and sound in case you need them.

The backup script run above dose not backup your MediaWiki’s uploaded images and other files.  These files are stored in your ‘images’ folder in the root of MediaWiki’s directory.  You need to back those up also.

tar -czf wiki-images.tgz images/

You should now have everything you need from your old MediaWiki install. Next you will need to install MediaWiki in a new location on your sever (or a development server).

Installing your new SQLite MediaWiki site

You should download and install the newest version of MediaWiki.  I always you the development trunk since this is what’s used on Wikipedia.

During the install process you will be asked what database you would like to use, you much choose ‘SQLite’ since this is the point of reinstalling MediaWiki.  Bring your new install all the way so you have a new install running on your server.  MediaWiki will create a default home page for you and you should be able to modify that page.  If you are unable to get MediaWiki installed or if you have a problem modifying the Main Page after the install, please see the MediaWiki mailing lists or the FAQ for assistance.

Restoring your Data on your new SQLite site

After you have your new SQLite version of MediaWiki installed and working, you need to restore your data.  The database part of this is pretty strate forward.

Start by copying the xml file you created in the first step to your new MediaWiki install.  Then run the following:

php maintenance/importDump.php wiki-backup.xml

Depending on the amount of pages you have, this may take quite some time to process. Once this is done all your pages should be on your new install (expect the Main Page, you will need to copy that manually).
To restore you images and other uploaded files, first you need to extract the tarball you made earlier to temporary location.

mkdir temporary
cp wiki-images.tgz temporary
cd temporary/
tar -xzf wiki-images.tgz

This will create a bunch of folders in the temporary directory, you need to copy everything in those folders into a single folder. The name of the folder doesn’t matter, I’m using tempimages, but you may use what you would like.

cd ../
mkdir tempimages
cp temporary/images/*/*/* tempimages

Now that you have everything in a single folder, import the content of that folder into your new MediaWiki install.

php maintenance/importImages.php tempimages/

And that should do it, you should now have a fully working MediaWiki install using SQLite.

Installing Dovecot with SQLite Support

Following in line with my previous post on Installing Postfix with SQL Support. This post will describe installing Dovecot from source with full SQLite support.

Installing from Source

First start out by downloading the lastest version from Dovecot’s website (the current version as of the writing of the how-to is 1.2.8).

yum -y install sqlite sqlite-devel gcc make patch db4-devel cyrus-sasl-devel

Next download and untar the source code.

wget http://dovecot.org/releases/1.2/dovecot-1.2.8.tar.gz
tar -xzf dovecot-1.2.8.tar.gz
cd dovecot-1.2.8/

Next, you will need to configure the code before compiling.

./configure --with-sqlite
echo $?

Assuming the configure command finishes with out error (the last line should be a “0”). Compile and install Dovecot.

make && make install

Configuring Dovecot for SQLite

First we need to create or modify the dovecot config file for SQLite access.  If you are currenly using MySQL with Dovecot, switching to SQLite is pretty easy and strate forward.  Or you may just use the below dovecot config file.

### Dovecot configuration file ###
### /etc/dovecot.conf ###
protocols = pop3 imap
login_user = postfix
auth_cache_size = 128
auth_cache_ttl = 600
mail_debug = yes

mail_location = maildir:%h/

protocol imap {
 listen = *:143
}

protocol lda {
  postmaster_address = postmaster@mattrude.com
  hostname = odin.mattrude.com
  mail_plugin_dir = /usr/local/lib/dovecot/lda
  auth_socket_path = /var/run/dovecot/auth-master
}

auth default {
  mechanisms = plain login
  userdb sql {
    args = /etc/dovecot-sqlite.conf
  }
  passdb sql {
    args = /etc/dovecot-sqlite.conf
  }
  socket listen {
    master {
      path = /var/run/dovecot/auth-master
      user = virtualmail
      group = virtualmail
    }
    client {
      path = /var/spool/postfix/private/auth
      mode = 0660
      user = postfix
      group = postfix
    }
  }
}

After you have created the main Dovecot config file, you will need to add the SQLite config file (below).

### /etc/dovecot-sqlite.conf ###
driver = sqlite
connect = /etc/postfix/postfix.sqlite
password_query = SELECT password, username AS user
  FROM mailbox WHERE username = '%u' AND domain = '%d'
user_query = SELECT maildir, 1000 AS uid, 1000 AS gid FROM mailbox WHERE
  username = '%u' AND domain = '%d' AND active = '1'

After the config files have been created, we need to create the database file, here is where you will need SQLite installed on the system.

Building the SQLite Database

In order to use the SQLite function, you need a SQLite database. First using SQLite3 run

sqlite3 /etc/postfix/postfix.sqlite

To create the database, then you can copy and past the following scheme into the new database.

CREATE TABLE alias (
  address varchar(255) NOT NULL,
  goto text NOT NULL,
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1');

CREATE TABLE domain (
  domain varchar(255) NOT NULL,
  description varchar(255) NOT NULL,
  aliases int(10) NOT NULL default '0',
  mailboxes int(10) NOT NULL default '0',
  maxquota bigint(20) NOT NULL default '0',
  quota bigint(20) NOT NULL default '0',
  transport varchar(255) NOT NULL,
  backupmx tinyint(1) NOT NULL default '0',
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1' );

CREATE TABLE mailbox (
  username varchar(255) NOT NULL,
  password varchar(255) NOT NULL,
  name varchar(255) NOT NULL,
  maildir varchar(255) NOT NULL,
  quota bigint(20) NOT NULL default '0',
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1',
  local_part varchar(255) NOT NULL );

Then close the database

.quit

Or you may download mine from below and use the same scheme work.

mkdir /var/run/dovecot

Dovecot INIT file

#!/bin/bash
#
#	/etc/rc.d/init.d/dovecot
#
# Starts the dovecot daemon
#
# chkconfig: - 65 35
# description: Dovecot Imap Server
# processname: dovecot
# Source function library.
. /etc/init.d/functions

test -x /usr/local/sbin/dovecot || exit 0

RETVAL=0
prog="Dovecot Imap"

start() {
       echo -n $"Starting $prog: "
	daemon /usr/local/sbin/dovecot
	RETVAL=$?
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/dovecot
	echo
}

stop() {
	echo -n $"Stopping $prog: "
	killproc /usr/local/sbin/dovecot
	RETVAL=$?
	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/dovecot
	echo
}

case "$1" in
  start)
	start
	;;
  stop)
	stop
	;;
  reload|restart)
	stop
	start
	RETVAL=$?
	;;
  condrestart)
	if [ -f /var/lock/subsys/dovecot ]; then
	    stop
	    start
	fi
	;;
  status)
	status /usr/local/sbin/dovecot
	RETVAL=$?
	;;
  *)
	echo $"Usage: $0 {condrestart|start|stop|restart|reload|status}"
	exit 1
esacexit $RETVAL