Backing up my (online) brain

February 22, 2008 at 11:22 PM | categories: backups, gmail, linux | View Comments

Files from my computer are disappearing.

No worries though, this is intentional.

As time goes on, more and more of my files are ending up in places like:

  • Google
  • Flickr
  • Facebook

And things are great. I can access these services from anywhere in the world, from any computer. What a great time to be alive!

But there is a bit of uneasiness in the back of my mind about all of this. Yes, these services liberate me (as well as my data), but at the same time, I've become a new breed of slave -- my data is no longer mine to control.

Privacy concerns aside, the biggest thing that haunts me is what if one of these services were to blow up? My data would be gone. When the data resided on my computer I could back it up myself anytime I wanted and have some sense of safety. With data on Google's server, honestly I feel pretty darn safe, but the cost of losing my data is still quite high.

My goal: Get my data back into my hands.

I'd like to start writing up my experiences on wrangling all of my data from my online services back onto media I control.

If you want to follow along, I'll be tagging these posts 'Backups'.


The first service I'll start with is Gmail. Google, in the realm of free mail hosts, is very accommodating when it comes to accessing your mail in bulk. They have both POP3 and IMAP access. Retrieving your mail with any number of clients is trivial.

But there's more to Gmail than just your email: tags. POP3 won't get the tags, but IMAP will organize tags into folders. However, most IMAP clients only download messages on-the-fly as you read them, not all at once (like for backups).

Solution: OfflineIMAP.

OfflineIMAP will completely download all your Gmail messages into folders corresponding to all your tags. It actually does much more than this, OfflineIMAP is a full two-way synchronization of your email. Once you've downloaded your email you can take it with you while you're offline, read it, delete it and when you come back online you synchronize with gmail again deleting those same messages off of the server and retrieving new messages.

OfflineIMAP works really well for it's stated purpose: using IMAP offline. It is also useful for backing up IMAP accounts but you have to understand how it works so that you don't accidentally start deleting mail from your gmail account. Since we're using OfflineIMAP as a backup device instead of a tool to read our email I would recommend that you never open the email that it downloads with any sort of mail reader. If you do, and you accidentally move messages to a different location (mutt does this to read messages for instance), OfflineIMAP might think you deleted the message and it will delete the same message off your server (In Gmail's case it actually just removes the tag and keeps the message in All Mail, but still..) If you ever need to use the downloaded email, its safer to use a copy instead.

Caveat Emptor. Onward to configuring OfflineIMAP.

On Ubuntu:
sudo apt-get install offlineimap

Create a file called ~/.offlineimaprc :

accounts = GMail
maxsyncaccounts = 3

[Account GMail]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = /storage/Gmail_Backup

[Repository Remote]
type = IMAP
remotehost =
remoteuser =
remotepass = your_password
ssl = yes
maxconnections = 1

#Setting realdelete = yes will Really Delete email from the server.
#Otherwise "deleting" a message will just remove any labels and 
#retain the message in the All Mail folder.
realdelete = no

The things you need to change are:

  • localfolders - This is the full path to where it will save the email
  • remoteuser - This is your gmail email address
  • remotepass - This is your gmail password (required for automated backups, otherwise it asks everytime)

You can now run offlineimap and you should start to see it download your email. By default, OfflineIMAP gives you a graphical or textual interface that shows you what it is doing. This is good especially for the first time you download your email.

After you've downloaded all of your email, you'll want to automate the task so it runs at least every day:

Put the following in your crontab:

00 23 * * * offlineimap -u Noninteractive.Quiet

This will run OfflineIMAP everyday at 11PM.

Again, because OfflineIMAP is not technically a backup solution in itself, you might also care to make a seperate rsyncd copy of the mail somewhere else:

rsync -a /storage/Gmail_Backup/ /storage/Gmail_Backup_Copy

You can similarly run rsync in your crontab after your offlineimap finishes. By making this seperate copy, you avoid the possible deletion of emails by OfflineIMAP. This second copy will also retain messages that you delete later on in Gmail (The offlineimap syncd copy will not). This may or may not be what you want.

Stay tuned for the next installment: Flickr backups!

Read and Post Comments

Automated Time Lapse Photography in Python

February 14, 2008 at 11:32 PM | categories: python | View Comments

Last night at the Utah Python User Group meeting, John Harrison had an awesome game he has made called Marshie Attacks. It combines PyGame, PySight, a projector and lasers. You point the laser at Marshie and the isight camera detects if the laser is on target, and if so, he blows up. Very cool stuff indeed.

His game depends on the Macintosh PySight library for video capture, but is otherwise cross platform. I decided to take a stab at getting images off my webcam in linux. Using gstreamer, it wasn't hard at all.

I don't have a projector, so I can't make a full Linux port Marshie Attacks, yet. This first stab is just to show how to grab images from a v4l2 device in Linux. This program takes time lapse photos, and saves them to disk with a configurable frequency.

Requirements include:
  • Python (2.5)
  • gstreamer 0.10
  • Python Imaging Library

Download the code

Read and Post Comments