Recently I bought an external disk of 1 terabyte. This comes to replace my old 250 MB disk.
The fists thing I made was to format the disk and create 1 partition to store all the information. As I wanted to use the disk with Mac and PC, I formatted it as NTFS. I also acquire a Mac utility to support writing to NTFS.
The disk worked fine until I tried to use backup tools for Mac. Most of the backup tools for Mac require a disk formatted in Mac format. But in my case it is NTFS. A solution might be to create 2 partitions, one for Mac and one for PC, but the one of Mac might not be available in PC. Still I want the compatibility with both systems.
My backup requirements for Mac were easy:
- Back up only data files
- Maintain only the image at the time the backup is made
- Do not backup OS and Apps. In case of failure, those can be restored from the installation disk and packages.
I was about to format the disk with the 2 partitions when I realize I already had 500MB of information in the disk. I was in risk to lose part of my information because I have no other disk to use as backup during the format operation.
I include another requirement:
- The backup files can be read from both systems. In case I want to review the files in any system.
The backup is simplified as I’m not really interested in maintain version of files.
I started looking for some scripts and backup ideas but none of those I found were of my entirely satisfaction.
At last I decide to create my own. The easiest seems to be the RSYNC command to create the copy of my Mac’s data.
My Backup Script initially was only to RSYNC the folder I want to backup, creating a copy of into the external disk.
The first time it took hours to backup all files.
The second time again took hours to make the backup.
No matter what files were changed the RSYNC always took hours to finish.
I didn’t do any deep investigation, but I’m convinced the RSYNC command can’t handle properly files stored under NTFS format. This because when the RSYNC started, many files are detected as changed, no matter no single change was made.
After been disappointed by RSYNC, I create my own script to sync files.
The required operation for the script:
- Receive 2 folders, source and destination
- Validate those folders
- Compare current files to detect differences
- Copy from source to destination the newly created files
- Delete from destination the deleted files
- Update destination if changes were made to source
Compare current files to detect differences, as needed: Copy from source to destination the newly created files Delete from destination the deleted files Update destination if changes were made to source
Using existing BASH commands all these can be done.
The script in review
The basic script usage is:
./myBackup.sh sourceDir destDir
or
./myBackup.sh sourceDir destDir X
Given the command, the script starts. We will later discuss the functions.
First validate the parameters
validateSource "$1"
...
validateDest "$2"
If source does not exist, the script won’t continue.
The list of files from both directories is retrieved using the FIND command.
cd "$source"
find -s * > $currentDir/__source__.lis
...
cd "$dest"
find -s * > $currentDir/__dest__.lis
Using the DIFF command we can get the differences.
diff __source__.lis __dest__.lis > __diff__.lis
Now, in the __diff__.lis file we have the list of differences. Reading that file we can process the fists 2 steps of the synchronization.
I had problems performing 2 operations as I read the file, so, I read the differences file twice, one for deletion, one to copy files.
First loop is to delete files. The files deleted from source are deleted from destination in this loop.
while read line
do
# < Copy file from source to dest
# > Delete file from dest
if [ "${line:0:1}" == ">" ]
then
delDest "$dest" "${line:2}"
fi
done < __diff__.lis
Second loop is to copy files. The files created in source and absent in destination are copied in this loop.
while read line
do
# < Copy file from source to dest
# > Delete file from dest
if [ "${line:0:1}" == "<" ]
then
copySource2Dest "$source" "$dest" "${line:2}"
fi
done < __diff__.lis
To detect differences in files, the evaluation is:
- Compare changed date
- Compare file size
- If required, compare checksum
For this comparison, we loop the file list of source directory.
while read line
do
{process here}
done < __source__.lis
Within the loop, we get parameters for each file. Directories are skipped from this validation.
getFileProps "$source" "$line"
sourceCreate=$st_birthtime
sourceChange=$st_mtime
sourceSize=$st_size
getFileProps "$dest" "$line"
destCreate=$st_birthtime
destChange=$st_mtime
destSize=$st_size
Once we retrieve the parameters, the comparisons are very straight forward.
if [ $sourceChange -ne $destChange ]
then
different=1
fi
if [ $sourceSize -ne $destSize ]
then
different=1
fi
The script is prepared to receive 3 parameters; the third is any value and represents the need to compare checksums of each file (source and destination). The third parameter is stored in a variable $deepSearch
if [ ! $deepSearch == "" ]
then
set $(cksum "$source/$line")
sourceCheckSum=$1
set $(cksum "$dest/$line")
destCheckSum=$1
if [ $sourceCheckSum -ne $destCheckSum ]
then
different=1
fi
fi
If all the validations tells us the files are different, then, the file need to be copied.
if [ $different -eq 1 ]
then
needEnter=0
copySource2Dest "$source" "$dest" "$line"
fi
Finally, just delete the temporary files.
rm __source__.lis
rm __dest__.lis
rm __diff__.lis
Supporting functions
The source directory, if a parameter was given, validate it is really a directory.
function validateSource
{
localSource=$1
if [ "$localSource" == "" ]
then
echo "Usage:"
echo " $0 <sourceDir> [<destinationDir>]"
echo " "
localSource=""
elif [ ! -d "$localSource" ]
then
echo "\"$localSource\" directory does not exist."
localSource=""
fi
source=$localSource
}
The destination directory is not really required, but it is highly recommended. If it was not wiven, a default backup directory is selected.
function validateDest
{
localDest=$1
if [ "$localDest" == "" ]
then
localDest=/bak
fi
if [ ! -d "$localDest" ]
then
mkdir "$localDest"
fi
dest=$localDest
}
Sometimes we will need to copy files from source to destination. In case of directory, create it in destination, in case of file, copy and set the last changed date to the one of source, this can be done with TOUCH.
function copySource2Dest
{
# if it is a directory, create it
# if it is a file, copy it
if [ -d "$1/$3" ]
then
mkdir "$2/$3"
else
cp "$1/$3" "$2/$3"
if [ -f "$2/$3" ]
then
touch -r"$1/$3" "$2/$3"
fi
fi
}
When a file or directory was deleted from source, do the same in destination. Before deleting, validate the file exists in destination.
function delDest
{
if [ -d "$1/$2" ]
then
rm -R "$1/$2"
elif [ -f "$1/$2" ]
then
rm "$1/$2"
fi
}
To compare source and destination files, we use the properties returned by the STAT command. The returned values will be used by the script to detect changes.
function getFileProps
{
export $(stat -s "$1/$2")
}
Installation
I have copied the script to /bin/usr in order to make the script available to be used within any directory. Otherwise, the script need to be in the same directory where it is called or within the PATH.
Future development
A way to maintain previous versions of files is highly recommended. Once versioning is implemented, also the restore mechanism will be required.
So far the script is working fine, still, any suggestion is well received.
The Script
