Evan Sims

Almost 9 months after the fact, my post about automating server backups to SpiderOak has proven to be popular amongst Googlers, so I thought it was time to update it with the latest build of my script.

If you don’t have SpiderOak yet, you can download it using this link to start with 3GB of space. If you decide you want to purchase a plan for more storage space, I recommend checking out their Facebook page where they currently have a promo code there worth 20% off all of their plan options.

You can find instructions for setting up the Linux client on their site.

Next, download, configure and save my backup script to your server. The latest version of this script can be found on my /scripts repo at GitHub, and I warmly welcome any contributions users make to it, be it bug fixes or new features.

#!/bin/bash

#  Automated Cloud Backups to SpiderOak
#  -----
#  Feel free to use this in any environment and
#  modify it to your heart's content.
#  -----
#  Author: Evan Sims <hello@evansims.com>
#  Updates: https://github.com/evansims/scripts/

##### START OF CONFIGURATION ####

  # MySQL Database:
  mysqlUser=""
  mysqlPass=""

  # MongoDB Database:
  mongoUser=""
  mongoPass=""
  mongoHost=""

  # Backup target directory:
  backup="/home/username/backups/"

  # Backup these directories or files:
  folders[0]="/home/username/files/"
  folders[1]="/var/logs/"

  # If you're doing backups to a git repo and want to commit
  # these updates, enable this.
  gitCommit=0
  gitPushRepo="" # git push $gitPushRepo

  # 7 day backups are the default.
  # Your $backup directory will be filled with Monday, Tuesday, etc. folders.
  # Each weekday folder will be overwritten when the backup cycles around.
  stamp=$(date -u +%A)
  backupCurrent=$backup$stamp

  # OR uuncomment the following lines for ~30 day backups.
  # Your $backup directory will be filled with folders labled 01 through 31
  # representing each day of the month.
  #stamp=$(date -u +%d)
  #backup=$backup$stamp

  # OR uncomment the following lines for unlimited backups.
  # Your $backup directory will have a new folder created for each day
  # a backup is built. Keep an eye on space usage if you're going to use
  #this method.
  #stamp=$(date -u +%F)
  #backup=$backup$stamp

##### END OF CONFIGURATION ####

temp="$PWD/backup.tmp"
echo "Starting Backup..."

if [ ! -d "$backupCurrent" ]; then
        mkdir "$backupCurrent"
fi

if [ ! -d "$backupCurrent/mongodb" ]; then
        mkdir "$backupCurrent/mongodb"
fi

if [ ! -d "$backupCurrent/mysql" ]; then
        mkdir "$backupCurrent/mysql"
fi

if [ -z $mongoUser ] && [ -z $mongoPass ]; then
  echo "-----"
  echo -ne "Exporting MongoDB collections ... "
  mongodump -h $mongoHost -u $mongoUser -p $mongoPass -o "$backupCurrent/mongodb" 2>"$temp" >/dev/null
  tar -cjf "$backupCurrent/mongodb.tar.bz2" "$backupCurrent/mongodb/"* 2>"$temp"
  rm -R "$backupCurrent/mongodb/"*
  mv "$backupCurrent/mongodb.tar.bz2" "$backupCurrent/mongodb/mongodb.tar.bz2"
  echo "Done."
fi

if [ -z $mysqlUser ] && [ -z $mysqlPass ]; then
  echo "-----"
  echo -ne "Preparing to backup MySQL ... "
  databases=( $(mysql -u"$mysqlUser" -p"$mysqlPass" --skip-column-names --batch -e "show databases;" 2>"$temp") );
  echo "found ${#databases[@]} databases.";

  for i in ${databases[@]}; do
          if [ $i != "information_schema" ] && [ $i != "mysql" ] && [ $i != "phpmyadmin" ]; then
                  echo -ne "Optimizing and backing up database $i ... "
                  mysql -u"$mysqlUser" -p"$mysqlPass" -D "$i" --skip-column-names --batch -e "optimize table $i" 2>"$temp" >/dev/null
                  mysqldump -u"$mysqlUser" -p"$mysqlPass" --opt $i | bzip2 -c > "$backupCurrent/mysql/$i.sql.bz2"
                  echo "Done."
          fi
  done
fi

echo "-----"
for f in "${folders[@]}"
do
        if [ -d "$f" ]; then
                echo -ne "Backing up directory $f ... "
                d=$(basename $f)
                d="$d.tar.bz2"
        elif [ -f "$f" ]; then
                echo -ne "Backing up file $f ... "
                d=${f##*/}
                d="$d.tar.bz2"
        else
                echo "Error! $f could not be found."
                exit 1
        fi

        tar -cjf "$backupCurrent/$d" -C / ${f:1}
        echo "Done."
done

if [ ! -s $temp ]; then
        rm -f "$temp"
fi

if [ -z $gitCommit ]; then
  cd $backup
  echo "-----"
  echo "Committing changes to git ... "
  git add .
  git commit -m "Backup $stamp"

  if [ -z $gitPush ]; then
    git push $gitPush
  fi

  echo "Done."
fi

echo "-----"
echo -ne "Pushing data to SpiderOak ... "

SpiderOak --empty-garbage-bin 2>"$temp" >/dev/null
SpiderOak --batchmode 2>"$temp" >/dev/null
SpiderOak --purge-historical-versions all 2>"$temp" >/dev/null
SpiderOak --vacuum 2>"$temp" >/dev/null

echo "Done."

echo "-----"
echo "Backup Complete!"

exit 0

Setup this script to run as a Cron task once a day, during a period of time that your machine won’t be being actively used. In the case of a web or database server, I’d recommend graphing your access logs and seeing when you have the lowest point of traffic on average, and run it during this time.

Although this script is designed for Linux, and makes use of the SpiderOak Linux client’s command interface, I believe the OS X client supports the same commands. The script could potentially be modified to support Mac with relative ease.

I hope folks out there find this useful. Have fun.

After several months of work, I’m very pleased to finally open the doors on WoW Instant. Instant is a rapid search engine for not only quickly looking up spells and items, but finding upgrades, comparing gear based on class and spec-specific usability weights, and building and sharing collections of items with guildmates.

I began working on Instant in late 2010 as an excuse to challenge myself and learn more about the Warcraft databases, and towards learning a variety of web technologies that ultimately became the foundation for what you see today (namely MongoDB, HTML5 and CSS3.)

I hope you enjoy what I’ve launched with today, but rest assured I’m not finished yet. This is just the tip of the iceberg, but I do hope you like it and please let me know if there’s any features in particular you’d like to see me implement going forward.

Update! I’ve posted an updated version of this script right here. Enjoy.

If you run a server, you need to be doing backups. There’s no ifs, ands or buts about it. I don’t care how small or big a site you run. It’s important. Whether it’s a hardware failure, a software bug that obliterates your database, a hacker or just negligence on your hosting provider’s side (all of which has happened to me) I assure you something will eventually go wrong. And you will wish to God you’d been doing them.

Over the years I’ve tried a lot of backup and sync solutions, but the three I’ve settled on using in my home, office and server environments are rsync, Dropbox and SpiderOak. Rsync I use mostly for my internal network sync or pushing code to my servers, so I’ll leave that one for a later post. In this post I’ll focus on the server end.

A while back I wrote a custom bash script for managing my backups, which I have setup to run as a cron task each night. It grabs all the user home directories, optimizes and exports the MySQL databases, and copies my MongoDB stores. It neatly compresses all of them and places them in a backup directory. It then fires up Dropbox or SpiderOak to push the backups to the cloud, and ultimately sync it all back to my local machines here in the office. It works flawlessly and gives me a lot of peace of mind.

Because Dropbox and SpiderOak both provide Linux CI binaries, this is all in all a pretty simple solution to setup and maintain. I will note that I switched to SpiderOak for my setup over Dropbox as I found the SpiderOak network to be more reliable, and their Linux binaries to be much more dependable. I also like that I can run SpiderOak on demand with the –batchmode flag rather than have it constantly taking up server memory when I’m not using it. SpiderOak’s attention to security and it’s encryption policies are also greatly appreciated, especially in light if tge recent discoveries about Dropbox’s own policies.

Step 1: Install Dropbox or SpiderOak

Both products offer Linux installation guides on their Wikis. These setup instructions have changed several times even in recent months, so I won’t go through the effort of echoing them here and likely leading you astray when they change down the road.

Dropbox instructions here, and SpiderOak here and here.

Step 2: Backup Script

Here is a modified version of my script. You can use it as a template to customize and build upon for your own needs.

#!/bin/bash

mysqlUser="MYSQL ROOT USER"
mysqlPass="MYSQL ROOT PASSWORD"

backup="/path/to/backups/"

folders[0]="/home/user1/"
folders[1]="/home/user2/"
folders[2]="/some/other/path/"

temp="$PWD/backup.tmp"

echo "Starting Backup..."

stamp=$(date -u  +%F)
backup=$backup$stamp

if [ ! -d "$backup" ]; then
  mkdir "$backup"
fi

echo "-----"
echo -ne "Preparing to backup MySQL tables... "
databases=( $(mysql -u"$mysqlUser" -p"$mysqlPass" --skip-column-names --batch -e "show databases;" 2>"$temp") );
echo "found ${#databases[@]} databases.";

for i in ${databases[@]}; do
  if [ $i != "information_schema" ] && [ $i != "mysql" ] && [ $i != "phpmyadmin" ]; then
    echo -ne "Optimizing and backing up database $i ... "
    mysql -u"$mysqlUser" -p"$mysqlPass" -D "$i" --skip-column-names --batch -e "optimize table $i" 2>"$temp" >/dev/null
    mysqldump -u"$mysqlUser" -p"$mysqlPass" --opt $i | bzip2 -c > "$backup/db_$i.sql.bz2"
    echo "Done."
  fi
done

echo "-----"

for f in "${folders[@]}"
do
  if [ -d "$f" ]; then
    echo -ne "Backing up directory $f ... "
    d=$(basename $f)
    d="$d.tar.bz2"
  elif [ -f "$f" ]; then
    echo -ne "Backing up file $f ... "
    d=${f##*/}
    d="$d.tar.bz2"
  else
    echo "Error! $f could not be found."
    exit 1
  fi

  tar -cjf "$backup/$d" -C / ${f:1}
  echo "Done."
done

if [ ! -s $temp ]; then
  rm -f "$temp"
fi

echo "-----"
echo "Pushing data to SpiderOak..."

SpiderOak --batchmode

echo "-----"
echo "Backup Complete!"
exit 0

Note that if you choose to use Dropbox or installed SpiderOak as a daemon, you’ll want to remove lines 62 through 65. I just prefer to use SpiderOak on-demand.

Step 3: Schedule It

Setup a Cron task to run the script. Be sure to think about user permissions for the script and run the task under the appropriate user/group.

* 3 * * * /path/to/backup.sh

This will run the backup every morning at 3am.

Step 4: Profit?

Aside from occasionally purging old backups you no longer need from your storage directory and SpiderOak, you should be golden. The script will run, the backups will be created and synced to your service of choice, and you’ll have one less thing to worry about.

• If you’d like to give SpiderOak a try, you can sign up using this link for 3GB of free backup storage.
• If you’d rather use Dropbox, you can sign up using this link for 2.25GB of free backup storage.

Hopes this helps someone out there. If you have any suggestions on improvements, please share them in the comments.

Requires SASS 3 and CSS3 PIE for legacy Internet Explorer support. Yes, even IE6.

@mixin rgba-background($color, $opacity) {
    position: relative;

    background: $color;
    background: rgba($color, $opacity);
    -pie-background: rgba($color, $opacity);

    @include PIE;
}

@mixin PIE {
    behavior: url("#{$basepath}/css/pie.htc");
}

Example usage:

#example { @include rgba-background(#0070FF, 0.5); }

#example2 { @include rgba-background(mix(#000, #0070FF), 0.5); }

Customize as necessary. Enjoy.