Leon's Weblog

July 22, 2008

File Synchronization with Unison

Filed under: Software Dev — Leon @ 8:51 am

Unison is a universal tool for synchronizing files. Although the program is no longer actively developed, it has enough useful features to make it my tool of choice for many tasks and projects. Here are a few scenarios for which I find Unison to be particularly useful:

Application Deployment
While Unison is in no way a replacement for version control, it can be used to release (web/intranet) applications from staging to production environments. This approach has several advantages. First, it is faster (and safer) than doing a full copy of a large site. Before the changes are committed, the program displays a summary of changed files and allows you to use diff to view/confirm the changes that were made. Since, platforms like ASP.NET can compile pages on-the-fly (in memory) synchronizing only the changed files saves the server processing time and improves the users’ experience. Also, synchronization is bidirectional (unlike rsync) so changes made directly on the production copy can be detected (just don’t ask who made them). Of course all of this can be achieved by writing custom deployment scripts but running Unison is far easier (especially if you have a frequent release schedule).

Synchronizing Documents with Mobile Gadgets
I run a central file server that hosts all of my documents. Although I can access the documents remotely, I often make create replicas for my laptop, PDA, and flash drive (as needed) for times when I am not connected or the internet connection is too slow. Unison is particularly useful here because it is available for Windows, Linux, and Mac and can synchronize local files (for flash drive), network shares, and over SSH. This was the only tool that I found that can safely and securely synchronize files from my Linux server to my windows laptop without compromising any functionality on either platform. Furthermore, if you have more than two replicas of the same files, you can safely synchronize the replicas two at a time to propagate changes.

Backup
There are many backup and disaster recovery solutions; however on Unix/Linux, everything is just a file. It’s often easier and more useful to just make a copy of everything to an external disk and maintain it by synchronizing. To recover, just copy the files back. I wouldn’t recommend this approach on a critical corporate server; but, for a personal server I find this approach is good enough.

Unison is free. Give it a shot.

July 3, 2008

Backup Fully, Backup Often

Filed under: Personal — Leon @ 9:59 am

Recently, the power supply on my server failed damaging the motherboard and all attached hard drives. I used many precautionary measures to protect the data on the server but they were not enough to avoid going through data recovery. The data was on a journaling file system (ReiserFS v.3) but that doesn’t help when the disks are fried and un-readable. The data was also mirrored across two 250GB drives which, as luck would have it, were both unusable. Sure there were several server backups as well but none were recent or complete enough to be usable.

My data recovery quest started with some anecdotal attempts to get the drives to work. The USB SATA adaptors did not work nor did the trick of putting the disks in the freezer (as silly as that sounds some have had luck with this approach so I figured is was worth a shot). It was time to enlist professional help so I contacted CBL Data Recovery who have had a long history recovering data from various disasters.

Pros:

  • CBL performs an assessment of the damage and only charges you if they are able to recover the data.
  • The prices are reasonable compared to other services that I have seen that change 10K and above.
  • Friendly service

Cons:

  • The recovery process took over a week. Apparently, the disk platters got damaged as well as the disk circuit board.
  • The customer service representatives were not very helpful and did not appear technically inclined. The CBL engineers that I talked to were much more aware of the situation.
  • Many of recovered text files had some binary data after the EOF flag which caused some Linux programs to crash when opening the files. This was fixable but time consuming.

Ultimately, CBL was able to recover all the data from the drives. Time to rebuild my server and think of a better backup strategy.