Configuring WordPress to use Memcached

When using WordPress self hosted software it’s generally a good idea to cache as much as possible.  Object Caching allows you store parts of the pages in memory for quicker retrieval, since the server will not need to look as much up from the SQL database.

Installing the needed parts

To start out, you will need to have memcached installed on your server. If your using Fedora, you may install memcached via Yum as follows.

yum -y install memcached php-pecl-memcached perl-Cache-Memcached

Configuring memcached

After installing memcached you need to configure it. If working in Fedora, and using the Yum install as talked about above, you will need to change the memcached confuration by modifying it’s sysconfig file located at /etc/sysconfig/memcached.

A default configuration may look like this.

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-l 127.0.0.1"

If you would like to share this memcached server with other webservers, change the address from 127.0.0.1 to the server’s actual address.

To set memcached to start automatically when the server get’s rebooted, run:

chkconfig memcached on

And of course, don’t for get to start it

service memcached start

Configuring WordPress

After memcached is installed, you need to configure the WordPress side.

Next you should install the Memcached Object Cache plugin, but be careful, this is not a normal plugin.  You should not activate this plugin as you would with a normal plugin, but instead download it as you normally would, but then you need to copy the object-cache.php file to your wp-content folder.

From the root of your WordPress install, run the following:

cp wp-content/plugins/memcached/object-cache.php wp-content/

Now we need to configure WordPress to use the memcached server. Add the following near the end of your wp-config.php file.

global $memcached_servers;
$memcached_servers = array('default' => array('127.0.0.1:11211'));

Note that were using the same server and port (127.0.0.1:11211) as was configured above while we were setting up memcached.

Checking in on memcached

Memcached is one of those things that just sort of runs. There’s not much direct feed back, besides the speed difference on your site.

One quick way is memcache-top. memcache-top will show you the current status of your memcached server.

To install, run the following.

wget http://memcache-top.googlecode.com/files/memcache-top-v0.6
chmod +x memcache-top-v0.6
./memcache-top-v0.6

Running ./memcache-top-v0.6 will assumed the default configuration we used here.

Dovecot v2.1+ Statistics

There are different “zoom levels” you may use to view the following statistics:

  • command: Per-IMAP command
  • session: Per IMAP/POP3 connection
  • user: Per user (all of user’s sessions summed up)
  • domain: Per domain (all of domain’s users summed up)
  • ip: Per IP address (all sessions from the IP summed up)

Basic Configuration

mail_plugins = $mail_plugins stats
protocol imap {
  mail_plugins = $mail_plugins imap_stats
}
plugin {
  # how often to session statistics
  stats_refresh = 30 secs
  # track per-IMAP command statistics
  stats_track_cmds = yes
}

You’ll also need to give enough permissions for mail processes to be able to write to stats-mail fifo. For example if you use a single “vmail” user for mail access:

service stats {
  fifo_listener stats-mail {
    user = vmail
    mode = 0600
  }
}

Statistics gathered

Statistics gathered using the getrusage() system call:

  • user_cpu: User CPU (seconds.microseconds)
  • sys_cpu: System CPU (seconds.microseconds)
  • min_faults: Minor page faults (page reclaims)
  • maj_faults: Major page faults
  • vol_cs: Voluntary context switches
  • invol_cs: Involuntary context switches
  • disk_input: Number of bytes read from disk
  • disk_output: Number of bytes written to disk

The disk_input and disk_output attempt to count the actual read/write bytes to physical disk, so e.g. reads from OS’s cache aren’t counted. Note that not all operating systems and filesystem support this, instead they simply show these values always as 0.

Statistics gathered by Dovecot’s lib-storage internally:

  • lookup_path: Number of open() and stat() calls (i.e. “path lookups”)
  • lookup_attr: Number of stat() and fstat() calls
  • read_count: Number of read() calls for message data (e.g. index files not counted)
  • read_bytes: Number of message bytes read()
  • cache_hits: Number of cache hits from dovecot.index.cache file


Dovecot’s Quota Plugin

The below assumes your using mdbox so dirsize or dict:sql will be the fastest. Since we already have sql setup, were going to go with that.

Enabling the Quota Plugin

mail_plugins = $mail_plugins quota

protocol imap {
  # Space separated list of plugins to load (default is global mail_plugins).
  mail_plugins = $mail_plugins imap_quota
}

plugin {
  # SQL backend:
  quota = dict:User quota::proxy::quota
  quota_rule = *:storage=1024M:messages=100000
  quota_rule2 = Trash:storage=+20%%
  quota_rule3 = Junk:storage=+20%%
}dict {
  quota = mysql:/etc/dovecot/dovecot-dict-mysql.conf
}

The dovecot-dict-mysql.conf file should containe something like this

connect = host=localhost dbname=postfix user=postfix password=postfix
map {
  pattern = priv/quota/storage
  table = quota
  username_field = username
  value_field = bytes
}
map {
  pattern = priv/quota/messages
  table = quota
  username_field = username
  value_field = messages
}

And lastly, add the following table to your email servers database in mySQL.

CREATE TABLE quota (
  username varchar(100) not null,
  bytes bigint not null default 0,
  messages integer not null default 0,
  primary key (username)
);

SQLite basics

Using SQLite

  • To open a SQLite database.
sqlite3 /path/to/database/file
  • Show the tables in a database
.tables
  • To Exit SQLite
.quit

PHP & SQLite

Connect to the database

try{
$dbHandle = new PDO('sqlite:/var/www/lighttpd/noc/emailserver_stat.sqlite');
}catch( PDOException $exception ){
echo "Can NOT connect to database";
die($exception->getMessage());
}

Create a table if it doesn’t exist

$sqlCreateTable = 'CREATE TABLE status (date date NOT NULL default \'0000-00-00\', item varchar(255) NOT NULL, value varchar(255) NOT NULL)';
$dbHandle->exec($sqlCreateTable);

List the conent of a table

$search_date = "2009-02-11";
$sqlGetView = 'SELECT * FROM status WHERE date = "'.$search_date.'"';
$result = $dbHandle->query($sqlGetView);
echo "<table border='1'>";
 while ($entry = $result->fetch()) {
    echo "<tr><td> " . $entry['value'] . "</td><td>" . $entry['item']. "</td></tr>";
 } echo "</table>";

Display a single value from a table

$search_date = "2009-02-11";
$sqlGetView = 'SELECT value FROM status WHERE item = \'connections\' AND date = "'.$search_date.'"';
$result = $dbHandle->query($sqlGetView);
$pageView = $result->fetch();
$connections = $pageView['value'];echo "$search_date
";
echo 'Number Of Connections: '.$connections.'
';

Configuring Spamassassin (MySQL & Virtual Users)

With this setup you will be saving the user preferences and bayes tokens in a MySQL data source. But, while scanning a message if spamd is unable to connect to the server specified in user_scores_dsn (below) or an error occurs when querying the SQL server then spam checking will not be performed on that message.

First we need to build the database table. You may download mine below.

Then import the file into the spamassassin table on your MySQL server.

wget http://wiki.mattrude.com/images/7/7a/Spamassassin.sql
wget http://wiki.mattrude.com/images/a/a7/Spamassassin_userpref.sql
mysql -h localhost -u postfix -ppostfix spamassassin < Spamassassin.sql
mysql -h localhost -u postfix -ppostfix spamassassin < Spamassassin_userpref.sql

To set the version number in the database, run the following.

echo "INSERT INTO bayes_global_vars VALUES ('VERSION','3');" |mysql -u postfix -ppostfix spamassassin
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'required_score', '5.0', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'rewrite_header Subject', '[SPAM]', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'rewrite_header Subject', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'report_safe', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'trusted_networks', '192.168.1.0/24', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_bayes', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'bayes_auto_learn', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'skip_rbl_checks', '0', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_razor2', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'use_pyzor', '1', NULL);
INSERT INTO userpref (username, preference, value, prefid) VALUES ('@GLOBAL',  'ok_locales', '1', NULL);

Now setup the configuration.

  • /etc/mail/spamassassin/local.cf
required_score                  5.0
rewrite_header Subject          [SPAM]
trusted_networks                192.168.1.0/24
report_safe                     1
use_bayes                       1
bayes_auto_learn                1
skip_rbl_checks                 0
use_razor2                      1
use_pyzor                       1
ok_languages                    enrewrite_header Subject
user_scores_dsn                 DBI:mysql:spamassassin:localhost
user_scores_sql_username        postfix
user_scores_sql_password        postfix
auto_whitelist_factory          Mail::SpamAssassin::SQLBasedAddrList
user_awl_dsn                    DBI:mysql:spamassassin:localhost
user_awl_sql_table              awl
user_awl_sql_username           postfix
user_awl_sql_password           postfix
bayes_store_module              Mail::SpamAssassin::BayesStore::MySQL
bayes_sql_dsn                   DBI:mysql:spamassassin:localhost
bayes_sql_username              postfix
bayes_sql_password              postfix

The spamd server will not pay attention to SQL preferences by default, even with user_scores_dsn set in the config files. You must startup spamd with the proper options (ie -q or -Q). If the user_scores_dsn option does not exist, SpamAssassin will not attempt to use SQL for retrieving users’ preferences.

SpamAssassin needs to be ran with the options similar to this:

/usr/bin/spamd -d -x -q -Q -u nobody -r /var/run/spamd.pid

I believe the best way of doing this is modify your /etc/init.d/spamassassin init file and change SPAMDOPTIONS to:

SPAMDOPTIONS="-d -m5 -x -q -Q -u nobody"

make sure /etc/sysconfig/spamassassin dosn’t override your settings run the below command to confirm spamassassin is running correctly

ps -eaf |grep spamd

Postfix mySQL to SQLite export/import scripts

postfixdb-export.sh:

#/bin/bash
TMPDIR=/tmp/postfix_sqlite
DATABASENAME=postfix.sqlite
MYSQLDATABASE=postfix
MYSQLUSER=postfix
MYSQLPASS=postfix
HTTPDIR=/var/www/mail.mattrude.com/noc

if [ ! -e $TMPDIR ]; then
 mkdir -p $TMPDIR
 chmod a+w $TMPDIR
else
 chmod a+w $TMPDIR
fi

if [ ! -e $HTTPDIR ]; then
 mkdir -p $HTTPDIR
 rm -rf $TMPDIR/csv_md5check.md5
else
 if [ ! -e $HTTPDIR/$DATABASENAME.gpg ]; then
  rm -rf $TMPDIR/csv_md5check.md5
 fi
fi

rm -rf $TMPDIR/alias.csv $TMPDIR/domain.csv $TMPDIR/mailbox.csv
echo "SELECT * FROM alias
INTO OUTFILE '$TMPDIR/alias.csv'
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'" |mysql $MYSQLDATABASE

echo "SELECT * FROM domain
INTO OUTFILE '$TMPDIR/domain.csv'
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'" |mysql $MYSQLDATABASE

echo "SELECT * FROM mailbox
INTO OUTFILE '$TMPDIR/mailbox.csv'
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'" |mysql $MYSQLDATABASE

if [ -e $TMPDIR/csv_md5check.md5 ]; then
 MD5STATUS=`md5sum -c $TMPDIR/csv_md5check.md5 |grep -v ".csv: OK" |wc -l` 2&gt;&amp;1
else
 MD5STATUS=999
fi

if [ $MD5STATUS != '0' ]; then
 rm -rf $TMPDIR/$DATABASENAME $TMPDIR/$DATABASENAME.md5 $TMPDIR/schema.txt
 md5sum $TMPDIR/*.csv &gt; $TMPDIR/csv_md5check.md5
 echo "Database $MYSQLDATABASE is being updated"
 echo "
 CREATE TABLE alias (
  address varchar(255) NOT NULL,
  goto text NOT NULL,
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1');

 CREATE TABLE domain (
  domain varchar(255) NOT NULL,
  description varchar(255) NOT NULL,
  aliases int(10) NOT NULL default '0',
  mailboxes int(10) NOT NULL default '0',
  maxquota bigint(20) NOT NULL default '0',
  quota bigint(20) NOT NULL default '0',
  transport varchar(255) NOT NULL,
  backupmx tinyint(1) NOT NULL default '0',
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1' );

 CREATE TABLE mailbox (
  username varchar(255) NOT NULL,
  password varchar(255) NOT NULL,
  name varchar(255) NOT NULL,
  maildir varchar(255) NOT NULL,
  mailstore varchar(255) NOT NULL default 'mdbox:',
  quota bigint(20) NOT NULL default '0',
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  quota_usage varchar(255),
  quota_usage_date varchar(255),
  active tinyint(1) NOT NULL default '1',
  local_part varchar(255) NOT NULL );
 " &gt; $TMPDIR/schema.txt

 echo ".read $TMPDIR/schema.txt" |sqlite3 $TMPDIR/$DATABASENAME
 echo ".import $TMPDIR/alias.csv alias" |sqlite3 $TMPDIR/$DATABASENAME
 echo ".import $TMPDIR/domain.csv domain" |sqlite3 $TMPDIR/$DATABASENAME
 echo ".import $TMPDIR/mailbox.csv mailbox" |sqlite3 $TMPDIR/$DATABASENAME

 rm -rf $HTTPDIR/$DATABASENAME $HTTPDIR/$DATABASENAME.gpg $HTTPDIR/$DATABASENAME.sig
 cp $TMPDIR/$DATABASENAME $HTTPDIR/$DATABASENAME
 cd $HTTPDIR/
 gpg -b $DATABASENAME
 gpg -se -r E2E70CD8 -r D2FAA865 -r 6503BE11 $DATABASENAME
 rm -f $HTTPDIR/$DATABASENAME
 #md5sum $DATABASENAME &gt; $DATABASENAME.md5
 #bzip2 $DATABASENAME
 #ssh odin.mattrude.com /home/matt/bin/sbin/postfix_sqlite_import.sh
fi

postfixdb-import.sh:

#/bin/bash
DBNAME=postfix.sqlite
DBDIR=/etc/postfix/database
URL=https://mail.mattrude.com/noc/postfix.sqlite
TMPDIR=/tmp/$DBNAME
LOGDIR=$TMPDIR

if [ ! -e $DBDIR ]; then
 mkdir -p $DBDIR
 chmod 700 $DBDIR
 chown 0:0 $DBDIR
fi

if [ ! -e $TMPDIR ]; then
 mkdir -p $TMPDIR
fi

if [ ! -e $LOGDIR ]; then
 mkdir -p $LOGDIR
fi

rm -rf $LOGDIR/postfix_import*.log $TMPDIR/$DBNAME.sig $DBDIR/$DBNAME.sig
wget --no-check-certificate -P $DBDIR $URL.sig &gt; $LOGDIR/postfix_import_sig_download.log  2&gt;&amp;1
SIGDOWNLOAD=$?

if [ $SIGDOWNLOAD != 0 ]; then
 echo "Did not download file $URL.sig; The host may be down or too busy."
 echo "Here is the download log:"
 cat $LOGDIR/postfix_import_sig_download.log
 echo ""
 echo "exiting script"
 exit 1
fi

cd $DBDIR
gpg --verify $DBNAME.sig &gt; /dev/null 2&gt;&amp;1
MD5STATUS=`echo $?`
if [ $MD5STATUS != '0' ]; then
 echo "Update needed, downloading updated $DBNAME."
 cd $TMPDIR
 rm -f $TMPDIR/$DBNAME.sig
 mv $DBDIR/$DBNAME.sig $TMPDIR/$DBNAME.sig
 rm -rf $TMPDIR/$DBNAME.gpg
 wget --no-check-certificate -P $TMPDIR $URL.gpg &gt; $LOGDIR/postfix_import_download.log 2&gt;&amp;1
 if [ -e $TMPDIR/$DBNAME.gpg ]; then
  cd $TMPDIR
  gpg $TMPDIR/$DBNAME.gpg &gt;&gt; $LOGDIR/postfix_import_download.log 2&gt;&amp;1
  echo "sig test 2"
  ls -lh
  gpg --verify $DBNAME.sig &gt; /dev/null 2&gt;&amp;1
  SIGTEST=`echo $?`
  if [ $SIGTEST == '0' ]; then
   cd $DBDIR
   echo "Database file $DBNAME.gpg was successfully downloaded"
   ARCHIVEDATE=`date +%Y%m%d_%H%M`
   mv $DBDIR/$DBNAME $DBDIR/$DBNAME.$ARCHIVEDATE
   mv $TMPDIR/$DBNAME $DBDIR/$DBNAME
   bzip2 $DBDIR/$DBNAME.$ARCHIVEDATE
   chown 0:0 $DBDIR/$DBNAME
   chmod 666 $DBDIR/$DBNAME
   if [ -e $DBDIR/$DBNAME ]; then
    cd $DBDIR
    cp $TMPDIR/$DBNAME.sig $DBDIR/$DBNAME.sig
    MD5TEST2=`gpg --verify $DBNAME.sig |grep -v "gpg: Good signature" |wc -l`
    if [ $MD5TEST2 = '0' ]; then
     echo "$DBNAME was updated successfully!"
     rm -rf $DBDIR/$DBNAME.sig
    else
     echo "$DBNAME was downloaded successfully but the downloaded file dose not match the signature file."
     exit 1
    fi
   else
    echo "$DBNAME was not copied to the database location ($DBDIR)."
    exit 1
   fi
  else
   echo "Database $DBNAME was not updated, the signature check failed."
   echo "Here's the Signature Check Log:"
   cd $TMPDIR
   gpg --verify $DBNAME.sig
   rm -rf $TMPDIR/$DBNAME*
  fi
 else
  echo "Database $DBNAME was not updated, file was not downloaded."
  echo ""
  echo "Here's the download log:"
  cat $LOGDIR/postfix_import_download.log
  rm -rf $TMPDIR/$DBNAME*
  rm -rf $DBDIR/$DBNAME.sig
 fi
fi
rm -rf $TMPDIR/$DBNAME.sig $DBDIR/$DBNAME.sig

WordPress Auto Backup Script

A quick backup script for WordPress. If you pass a config file containing the block of variables, the script will auto run for your install of WordPress.

# /bin/bash
BKNAME=
DIR=
DBHOST=
DBNAME=
DBUSER=
DBPASS=

# Backup Varibles
BKDIR=

# Restore Variables
RSDIR=
RSARDIR=
RSUSER=
RSGROUP=

OFFSITE=

# Offsite Rsync
RSYNCHOST=
RSYNCUSERNAME=
RSYNCPASSWORD=

# Offiste SCP
SCPUSER=
SCPHOST=
SCPDIR=

if [ `date +%e` =  1 ]; then
 DAY=`date +%b`
 DATE=$DAY.`date +%Y-%m-%d`
else
 DAY=`date +%w`
 DATE=$DAY.`date +%Y-%m-%d`
fi

if [ -e $1 ]; then
 source $1
 CONFIGSTATUS="Using config file found at $1"
else
 echo "No Config file found"
 exit
fi

if [ ! -e $DB_HOST ]; then
  DBHOST=`grep "DB_HOST" $DIR/wp-config.php |sed "s/define('DB_HOST', '//" |sed "s/');//"`
  DBNAME=`grep "DB_NAME" $DIR/wp-config.php |sed "s/define('DB_NAME', '//" |sed "s/');//"`
  DBUSER=`grep "DB_USER" $DIR/wp-config.php |sed "s/define('DB_USER', '//" |sed "s/');//"`
  DBPASS=`grep "DB_PASS" $DIR/wp-config.php |sed "s/define('DB_PASSWORD', '//" |sed "s/');//"`
fi
BKDIRNAME=$BKNAME

if [ `date +%e` = 1 ]; then
 ARCHIVE=1
else
 if [ `date +%w` = 0 ]; then
  ARCHIVE=1
 else
  ARCHIVE=0
 fi
fi

if [ ! -e $BKDIR ];then
 mkdir -p $BKDIR
fi

chown -R root:root $DIR/
chown -R apache:apache $DIR/wp-content $DIR/wp-admin/update.php
chown apache:apache $DIR

if [ -e $DIR/sitemap.xml ]; then
  chown apache:apache $DIR/sitemap.xml
fi

if [ -e $DIR/sitemap.xml.gz ]; then
  chown apache:apache $DIR/sitemap.xml.gz
fi
case "$2" in
restore)
  rm -rf $RSARDIR/$BKNAME.$DAY.*
  rm -rf $RSDIR/$BKNAME
  rm -rf $DIR/$BKNAME.$DAY.*
  rm -rf $DIR/$DBNAME.*
  if [ ! -e $RSARDIR ]; then
   mkdir -p $RSARDIR
  fi
  export RSYNC_PASSWORD=$RSYNCPASSWORD
  rsync -rvzht --delete --stats $RSYNCUSERNAME@$RSYNCHOST::ibackup/odin/$BKNAME/$BKNAME.$DAY.* $RSARDIR/ --port=45873 >> $DIR/$BACKUPNAME.log 2>&1
  cd $RSARDIR
  md5sum -c $BKNAME.$DATE.tgz.md5 > /dev/null 2>&1
  RESTOREMD5=`echo $(($?))`
  if [ $RESTOREMD5 = 0 ]; then
   cd $RSDIR/
   cp $RSARDIR/$BKNAME.$DATE.tgz $RSDIR/$BKNAME.$DATE.tgz
   tar -xzf $RSDIR/$BKNAME.$DATE.tgz
   mv $RSDIR$DIR $RSDIR
   cd $RSDIR/$BKNAME
   md5sum -c $DBNAME.$DATE.sql.md5 > /dev/null 2>&1
   RESTORESQLMD5=`echo $(($?))`
   if [ $RESTORESQLMD5 = 0 ]; then
    cd $RSDIR
    mysql -u $DBUSER -p$DBPASS $DBNAME > $DIR/$BKNAME.$DATE.log 2>&1
  done
  mysqldump -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME > $DIR/$DBNAME.$DATE.sql 2> $DIR/$BKNAME.$DATE.log
  a=$?
  mysqldump -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME --xml > $DIR/$DBNAME.$DATE.xml 2> $DIR/$BKNAME.$DATE.log
  a2=$?
  cd $DIR/
  md5sum $DBNAME.$DATE.sql $DBNAME.$DATE.xml > $DBNAME.$DATE.sql.md5 2>> $DIR/$BKNAME.$DATE.log
  b=$?
  SQLERROR=`echo $(($a+$a2+$b))`
  if [ $SQLERROR != 0 ]; then
   BKNAME=$BKNAME-NOSQL
   echo "Exiting on SQL error"
   echo "DBHOST $DBHOST"
   echo "DBNAME $DBNAME"
   echo "DBUSER $DBUSER"
   echo "DBPASS $DBPASS"
   exit 1
  fi
  tar -czf $BKDIR/$BKNAME.$DATE.tgz --totals $DIR >> $DIR/$BKNAME.$DATE.log 2>&1
  c=$?
  cd $BKDIR/
  md5sum $BKNAME.$DATE.tgz > $BKNAME.$DATE.tgz.md5 2>> $DIR/$BKNAME.$DATE.log
  d=$?
  TOTAL=`ls -lh $BKDIR/$BKNAME.$DATE.tgz |awk '{ print $5 }'`
  ERROR=`echo $(($a+$a2+$b+$c+$d))`
  if [ $ERROR = 0 ]; then
   STATUS=Good
  else
   STATUS=Failed
  fi

  echo "" >> $DIR/$BKNAME.$DATE.log
  echo "Starting GPG encryption" >> $DIR/$BKNAME.$DATE.log
  for a in $GPGKEY
  do
   GPGKEYD="$GPGKEYD -r $a"
  done
  #gpg -e -r $GPGKEY $BKDIR/$BKNAME.$DATE.tgz
  echo "" >> $DIR/$BKNAME.$DATE.log
  if [ $ARCHIVE = '0' ]; then
   case "$OFFSITE" in
    rsync|RSYNC)
      export RSYNC_PASSWORD=$RSYNCPASSWORD
      rsync -rvzht --delete --stats $BKDIR/ $RSYNCUSERNAME@$RSYNCHOST::ibackup/odin/$BKDIRNAME --port=45873 --exclude=Archive/ >> $DIR/$BACKUPNAME.log 2>&1
    ;;

    scp|SCP)
      echo "" >> $DIR/$BKNAME.$DATE.log
      echo "Starting SCP Transmition" >> $DIR/$BKNAME.$DATE.log
      ssh $SCPUSER@$SCPHOST "mkdir -p $SCPDIR"
      ssh $SCPUSER@$SCPHOST "rm -f $SCPDIR/$BKNAME.$DAY.*"
      scp -q $BKDIR/$BKNAME.$DATE.tgz $SCPUSER@$SCPHOST:$SCPDIR/ 2>> $DIR/$BKNAME.$DATE.log
      scp -q $BKDIR/$BKNAME.$DATE.tgz.md5 $SCPUSER@$SCPHOST:$SCPDIR/ 2>> $DIR/$BKNAME.$DATE.log
      #ssh $SCPUSER@$SCPHOST "chmod 666 $SCPDIR/$BKNAME.$DAY.*"
      echo "" >> $DIR/$BKNAME.$DATE.log
    ;;
   esac
  fi

  echo "final error status is: $ERROR" >> $DIR/$BKNAME.$DATE.log
  sed '/tar: Removing leading */d' $DIR/$BKNAME.$DATE.log > $BKDIR/backuptmp.log
  #echo "INSERT INTO Backup_Log ( System, Backup_Job, Label, Output, Bytes, Status, Log ) VALUES ('`hostname -s`', 'Wiki', '$DIR', '$BKNAME.$DATE', '$TOTAL', '$STATUS', '`cat $BKDIR/backuptmp.log`');" |mysql -t -h localhost -u backup Status
  rm -rf $BKDIR/backuptmp.log
  rm -rf $DIR/$DBNAME.$DAY.*
  rm -rf $DIR/$BKNAME.$DAY.*
  ;;
*)
  echo "The valid commands are backup & restore"
  exit 1
  ;;
esac

Installing Dovecot with SQLite Support

Following in line with my previous post on Installing Postfix with SQL Support. This post will describe installing Dovecot from source with full SQLite support.

Installing from Source

First start out by downloading the lastest version from Dovecot’s website (the current version as of the writing of the how-to is 1.2.8).

yum -y install sqlite sqlite-devel gcc make patch db4-devel cyrus-sasl-devel

Next download and untar the source code.

wget http://dovecot.org/releases/1.2/dovecot-1.2.8.tar.gz
tar -xzf dovecot-1.2.8.tar.gz
cd dovecot-1.2.8/

Next, you will need to configure the code before compiling.

./configure --with-sqlite
echo $?

Assuming the configure command finishes with out error (the last line should be a “0”). Compile and install Dovecot.

make && make install

Configuring Dovecot for SQLite

First we need to create or modify the dovecot config file for SQLite access.  If you are currenly using MySQL with Dovecot, switching to SQLite is pretty easy and strate forward.  Or you may just use the below dovecot config file.

### Dovecot configuration file ###
### /etc/dovecot.conf ###
protocols = pop3 imap
login_user = postfix
auth_cache_size = 128
auth_cache_ttl = 600
mail_debug = yes

mail_location = maildir:%h/

protocol imap {
 listen = *:143
}

protocol lda {
  postmaster_address = postmaster@mattrude.com
  hostname = odin.mattrude.com
  mail_plugin_dir = /usr/local/lib/dovecot/lda
  auth_socket_path = /var/run/dovecot/auth-master
}

auth default {
  mechanisms = plain login
  userdb sql {
    args = /etc/dovecot-sqlite.conf
  }
  passdb sql {
    args = /etc/dovecot-sqlite.conf
  }
  socket listen {
    master {
      path = /var/run/dovecot/auth-master
      user = virtualmail
      group = virtualmail
    }
    client {
      path = /var/spool/postfix/private/auth
      mode = 0660
      user = postfix
      group = postfix
    }
  }
}

After you have created the main Dovecot config file, you will need to add the SQLite config file (below).

### /etc/dovecot-sqlite.conf ###
driver = sqlite
connect = /etc/postfix/postfix.sqlite
password_query = SELECT password, username AS user
  FROM mailbox WHERE username = '%u' AND domain = '%d'
user_query = SELECT maildir, 1000 AS uid, 1000 AS gid FROM mailbox WHERE
  username = '%u' AND domain = '%d' AND active = '1'

After the config files have been created, we need to create the database file, here is where you will need SQLite installed on the system.

Building the SQLite Database

In order to use the SQLite function, you need a SQLite database. First using SQLite3 run

sqlite3 /etc/postfix/postfix.sqlite

To create the database, then you can copy and past the following scheme into the new database.

CREATE TABLE alias (
  address varchar(255) NOT NULL,
  goto text NOT NULL,
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1');

CREATE TABLE domain (
  domain varchar(255) NOT NULL,
  description varchar(255) NOT NULL,
  aliases int(10) NOT NULL default '0',
  mailboxes int(10) NOT NULL default '0',
  maxquota bigint(20) NOT NULL default '0',
  quota bigint(20) NOT NULL default '0',
  transport varchar(255) NOT NULL,
  backupmx tinyint(1) NOT NULL default '0',
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1' );

CREATE TABLE mailbox (
  username varchar(255) NOT NULL,
  password varchar(255) NOT NULL,
  name varchar(255) NOT NULL,
  maildir varchar(255) NOT NULL,
  quota bigint(20) NOT NULL default '0',
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1',
  local_part varchar(255) NOT NULL );

Then close the database

.quit

Or you may download mine from below and use the same scheme work.

mkdir /var/run/dovecot

Dovecot INIT file

#!/bin/bash
#
#	/etc/rc.d/init.d/dovecot
#
# Starts the dovecot daemon
#
# chkconfig: - 65 35
# description: Dovecot Imap Server
# processname: dovecot
# Source function library.
. /etc/init.d/functions

test -x /usr/local/sbin/dovecot || exit 0

RETVAL=0
prog="Dovecot Imap"

start() {
       echo -n $"Starting $prog: "
	daemon /usr/local/sbin/dovecot
	RETVAL=$?
	[ $RETVAL -eq 0 ] && touch /var/lock/subsys/dovecot
	echo
}

stop() {
	echo -n $"Stopping $prog: "
	killproc /usr/local/sbin/dovecot
	RETVAL=$?
	[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/dovecot
	echo
}

case "$1" in
  start)
	start
	;;
  stop)
	stop
	;;
  reload|restart)
	stop
	start
	RETVAL=$?
	;;
  condrestart)
	if [ -f /var/lock/subsys/dovecot ]; then
	    stop
	    start
	fi
	;;
  status)
	status /usr/local/sbin/dovecot
	RETVAL=$?
	;;
  *)
	echo $"Usage: $0 {condrestart|start|stop|restart|reload|status}"
	exit 1
esacexit $RETVAL

Auto Update Script for MiniMyth

This script will download the newest version of MiniMyth and update your tftp directory.

It first downloads the version file and compares that version number to the current install version of MiniMyth install.  If it determines that the version dose not match it will then download the current release, untar it and move the files to their current locations. It dose not delete any older versions on your server.

  • The TFTPDIR variable is the location on your tftp server where the MiniMyth files are stored.
  • The URL variable is the MiniMyth directory that the install of MiniMyth you wish to use lives.

You may also download the full directory structure with this script from my github repository or from my server.

# /bin/bash
# Matt Rude <m@mattrude.com> 11-Nov-2009
#
TFTPDIR=/tftpboot/PXEClient
URL="http://minimyth.org/download/stable/latest/"
##############################################################################
if [ -e $TFTPDIR/version ]; then
 mv $TFTPDIR/version $TFTPDIR/version.last
 rm -rf $TFTPDIR/verstion.md5
else
 touch $TFTPDIR/version.log
 touch $TFTPDIR/version.last
fi

cd $TFTPDIR
wget -nc $URL/version > /dev/null 2>&1
VER=`cat $TFTPDIR/version`
OLDVER=`cat $TFTPDIR/version.last`

if [ "$VER" = "$OLDVER" ]; then
 chown -R apache:apache $TFTPDIR/*
 exit 0
else
 echo "`date` Upgraded to version: $VER" >> version.log

 rm -rf $TFTPDIR/ram-minimyth-*.tar.bz2.md5
 wget -nc $URL/ram-minimyth-$VER.tar.bz2 > /dev/null 2>&1
 wget -nc $URL/ram-minimyth-$VER.tar.bz2.md5 > /dev/null 2>&1
 MD5STAT=`md5sum -c ram-minimyth-$VER.tar.bz2.md5 |awk ' {print $2 }'`
 if [ "$MD5STAT" = "OK" ]; then
  rm -f $TFTPDIR/kerne*
  rm -f $TFTPDIR/rootf*
  rm -fr $TFTPDIR/conf/default/theme*
  tar -xjf ram-minimyth-$VER.tar.bz2
  rm -rf ram-minimyth-$VER.*
 else
  echo "`date` Minimyth Version $VER Failed the MD5 check" >> version.log
  echo "" > $TFTPDIR/version
  exit 1
 fi

 RAMDIR=$TFTPDIR/ram-minimyth-$VER

 mkdir -p $TFTPDIR/conf/default
 cp $RAMDIR/kernel $TFTPDIR/kernel-$VER
 cp $RAMDIR/rootfs $TFTPDIR/rootfs-$VER
 cp -r $RAMDIR/themes $TFTPDIR/conf/default/themes-$VER

 ln -s kernel-$VER kernel
 ln -s rootfs-$VER rootfs
 cd $TFTPDIR/conf/default
 ln -s themes-$VER themes
 #mythtvosd --template=scroller --scroll_text="minimyth upgraded to: $VER" > /dev/null
ficd $TFTPDIR
mv version.log version.tmp
tail -2 version.tmp > version.log
rm -rf version.tmp
chown -R apache:apache $TFTPDIR/*
exit 0

Installing Postfix with SQLite Support

This How-to will show you how to set up Postfix with SQLite support on a Fedora 11 system.  This will require us to compile Postfix from scratch.

Installing the dependences

yum -y install sqlite sqlite-devel gcc make patch db4-devel cyrus-sasl-devel
echo "postfix:x:89:89::/var/spool/postfix:/sbin/nologin" >> /etc/passwd
echo "postdrop:x:90:90::/var/spool/postfix:/sbin/nologin" >> /etc/passwd
echo "postfix:x:89:" >> /etc/group
echo "postdrop:x:90:" >> /etc/group
ln -s /usr/lib/sasl2/ /usr/local/lib/sasl2

Download and Patch Postfix

Download and unpack the current Postfix Version.

wget http://postfix.energybeam.com/source/official/postfix-2.6.5.tar.gz
tar -xzf postfix-2.6.5.tar.gz
cd postfix-2.6.5

Download and Patch Postfix with the SQLite Postfix patch

wget http://www.treibsand.com/postfix-sqlite/postfix-2.6-20080216_sqlite.patch
patch -ul -d . -p1 < postfix-2.6-20080216_sqlite.patch
echo $?

Building Postfix

To Build with SQLite Support

make -f Makefile.init makefiles 'CCARGS=-DHAS_SQLITE -I/usr/local/include'
'AUXLIBS=-L/usr/local/lib -lsqlite3'
echo $?

Or to Build with SQLite & TLS Support
Requires:

yum -y install openssl-devel
make -f Makefile.init makefiles 'CCARGS=-DHAS_SQLITE -I/usr/local/include -DUSE_TLS'
'AUXLIBS=-L/usr/local/lib -lsqlite3 -lz -lm -lssl -lcrypto'
echo $?

To Build with SQLite, Dovecot, & TLS Support

yum -y install openssl-devel dovecot-devel
make makefiles 'CCARGS=-DHAS_SQLITE -I/usr/include/sasl/ -DUSE_SASL_AUTH -DUSE_CYRUS_SASL  -DDEF_SERVER_SASL_TYPE="dovecot" -DUSE_TLS' -I/usr/local/include
   'AUXLIBS=-L/usr/local/lib -lsqlite3 -lz -lm -lssl -lcrypto -lsasl2'
echo $?

Compiling Postfix

Now Compile Postfix

make
echo $?

And Install it

make install

Building the SQLite Database

In order to use the SQLite function, you need a SQLite database. First using SQLite3 run

sqlite3 /etc/postfix/postfix.sqlite

To create the database, then you can copy and past the following scheme into the new database.

CREATE TABLE alias (
  address varchar(255) NOT NULL,
  goto text NOT NULL,
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1');

CREATE TABLE domain (
  domain varchar(255) NOT NULL,
  description varchar(255) NOT NULL,
  aliases int(10) NOT NULL default '0',
  mailboxes int(10) NOT NULL default '0',
  maxquota bigint(20) NOT NULL default '0',
  quota bigint(20) NOT NULL default '0',
  transport varchar(255) NOT NULL,
  backupmx tinyint(1) NOT NULL default '0',
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1' );

CREATE TABLE mailbox (
  username varchar(255) NOT NULL,
  password varchar(255) NOT NULL,
  name varchar(255) NOT NULL,
  maildir varchar(255) NOT NULL,
  quota bigint(20) NOT NULL default '0',
  domain varchar(255) NOT NULL,
  created datetime NOT NULL default '0000-00-00 00:00:00',
  modified datetime NOT NULL default '0000-00-00 00:00:00',
  active tinyint(1) NOT NULL default '1',
  local_part varchar(255) NOT NULL );

Then close the database

.quit

Or you may download mine from below and use the same scheme work.

Once you have your database file place it in /etc/postfix/ as something like /etc/postfix/postfix.sqlite then run

chown root:root /etc/postfix/postfix.sqlite
chmod 600 /etc/postfix/postfix.sqlite

Configuring Postfix

Now add the maps to you config.

/etc/postfix/main.cf

relay_domains = sqlite:/etc/postfix/sqlite_relay_domains_maps.cf
relay_recipient_maps = sqlite:/etc/postfix/sqlite_relay_recipient_maps.cf
virtual_alias_maps = sqlite:/etc/postfix/sqlite_virtual_alias_maps.cf
virtual_mailbox_domains = sqlite:/etc/postfix/sqlite_virtual_domains_maps.cf
virtual_mailbox_maps = sqlite:/etc/postfix/sqlite_virtual_mailbox_maps.cfvirtual_mailbox_base = /var/spool/virtualmailboxes
virtual_minimum_uid= 1000
virtual_uid_maps = static:1000
virtual_gid_maps = static:1000

sqlite_relay_domains_maps.cf

dbpath = /etc/postfix/postfix.sqlite
query = SELECT domain FROM domain WHERE domain='%s' AND backupmx = '1' AND active = '1'

sqlite_relay_recipient_maps.cf

dbpath = /etc/postfix/postfix.sqlite
query = SELECT goto FROM alias WHERE address='%s' AND active = 1

sqlite_virtual_alias_maps.cf

dbpath = /etc/postfix/postfix.sqlite
query = SELECT goto FROM alias WHERE address='%s' AND active = '1'

sqlite_virtual_domains_maps.cf

dbpath = /etc/postfix/postfix.sqlite
query = SELECT domain FROM domain WHERE domain='%s' AND backupmx = '0' AND active = '1'

sqlite_virtual_mailbox_maps.cf

dbpath = /etc/postfix/postfix.sqlite
query = SELECT maildir FROM mailbox WHERE username='%s' AND active = '1'

Setting up a fresh install of Postfix

mkdir /var/spool/virtualmailboxes/
echo "virtualmail:x:1000:1000::/var/spool/virtualmailboxes:/sbin/nologin" >> /etc/passwd
echo "virtualmail:x:1000:" >> /etc/group
chmod 700 /var/spool/virtualmailboxes/
chown -R virtualmail:virtualmail /var/spool/virtualmailboxes/
rm -f /usr/lib/sendmail
ln -s /usr/sbin/sendmail /usr/lib/sendmail

Adding SQLite entry’s

First add a Domain

echo "INSERT INTO domain ( domain, description, transport )
VALUES ( 'laptop.mattrude.com', 'laptops domain', 'virtual' );" |sqlite3 /etc/postfix/postfix.sqlite

Then add a user

echo "INSERT INTO mailbox ( username, password, name, maildir, domain, local_part )
VALUES ( 'matt@laptop.mattrude.com', 'password', 'Matt', 'laptop.mattrude.com/matt@laptop.mattrude.com/', 'laptop.mattrude.com', 'matt' );" |sqlite3 /etc/postfix/postfix.sqlite

Last we need to add the mailboxes alias

echo "INSERT INTO alias ( address, goto, domain )
VALUES ( 'matt@laptop.mattrude.com', 'matt@laptop.mattrude.com', 'laptop.mattrude.com' );" |sqlite3 /etc/postfix/postfix.sqlite

Next, contue on to my Dovecot SQLite how-to to finish your email server.