Avoid uploading duplicated files in FTP using MSDOS script

I was uploading files using the script I made in a previous post, and all was working smoothly until I reach a directory full of images used for an address book application, there were about 2,000 files. The current version of the script made no comparison for file existence (actually it deletes the old file in the server before attempting to save the new one) or date comparison. In this case, and assuming those files change don’t change too often, I start thinking in a new process to remove the duplicated files and only upload the non existing files.

I made a new FTP script to read the remote files and then delete the existing files from the local PC. This process is a dumb process and only task is to delete all files from the list. It is not aware of date changes or new file versions. Saying these, if you use this script, always work with a backup of your files, never with your original files as they might be deleted.

The script usage is:

deleteDuplicatedFiles.cmd <Root directory to validate>

Current version of the script has no recursive processes implemented

The operation is as follows:

1. Set working variables and defaults

set user=<ftp user>

set pass=<ftp password>

set site=<ftp site>

set base=<remote path to validate>

set file=<FTP commands file>

set outputFile=<local file to store file listing>

2. Start creating the commands file

echo user %user% %pass% >> %file%

echo cd %base% >> %file%

echo ls >> %file%

echo bye >> %file%

3. Execute the commands´ file

ftp -n %site% < %file% > %outputFile%

4. Call to delete the preexisting files

call :deleteFiles "%workDir%" %outputFile%

5. The deleteFiles subroutine reads all lines from the output file and, assuming is a file name, it validate its existing using another subroutine

FOR /F "usebackq delims==" %%i IN (`type %localFile%`) do (

call :deleteSingleFile %%i

)

6. I’m using deleteSingleFile in case another validation is required or needed to implement, currently it only check the file existence and then delete the file

if exist %deleteFile% del %deleteFile%

Here you can find a copy of the script in case you want to try it: deleteDuplicatedFiles.zip

DISCLAIMER: this script is delivered as is and I take no responsibility of any deleted or damage file. Use this script cautiously and always work on a backup to preserve the original files.

FTP Uploading files using MSDOS script

Recently I was working in a static Web Site and at the time I need to upload it to the server, I realize there were several files and directories to upload. As I’ve never been a big fan of FTP programs other than the command line provided in each OS, and, instead of searching if there were already a public solution, I started thinking in an automation script to upload the files.

The basic idea of an automation script is to read the contents of a directory, create a commands script for FTP and then execute the generated script. But it also needs to be recursive within internal directories.

As I was working with Windows, the script I created is for MS DOS

The way to call it is:

uploadfiles <Root directory to upload>

Where the given root directory is where the desired files to upload are stored.

Taking a little walkthrough of the script, it works like these:

1. Set working variables and defaults.

set user=<set FTP User>

set pass=<FTP Password>

set site=<site to connect>

set base=<target directory>

set file=<temporary commands file>

2. Next thing to do is to start creating the commands file, some of the used values have been set in step 1

echo user %user% %pass% >> %file%

echo cd %base% >> %file%

echo pwd >> %file%

echo bin >> %file%

echo hash >> %file%

3. The directory processing logic was implemented in a subroutine, we use it to process the root directory

call :processDir . "%workDir%" 0 %file% x

4. Once the directory was processed, we return just to call the FTP to use the commands file and upload all found files.

ftp -n %site% < %file%

The processDir subroutine is called recursively to process all found subdirectories. Once the script is finished, it is intended to work within a FTP session, once in the session if might change directories in remote site, but not in local PC, so the script need to handle all calls to local files as a relative path from where the script was called. This routine works as follows:

1. Local variables are set, from received parameters

set localPath=%~1

set localDir=%~2

set localID=%3

set localFile=%4

set quotes=1

set fullpath=%localPath%\%localDir%

2. For each processed directory, the script assumes that all need to be created

echo mkdir %localDir% >> %localFile%

echo cd %localDir% >> %localFile%

echo pwd >> %localFile%

3. The is finds all the directories of the current one and call itself to process them

FOR /F "usebackq delims==" %%i IN (`dir /b/ad`) do (

...

if "!localquote!"=="0" call :processDir %fullpath% %%i 1 %localFile% %5x

if "!localquote!"=="1" call :processDir "%fullpath%" "%%i" 1 %localFile% %5x

)

4. Next step is to process all files form current directory

FOR /F "usebackq delims==" %%i IN (`dir /b/a-d`) do (

...

if "!localquote!"=="0" (

echo ls %%i >> %localFile%

echo dele %%i >> %localFile%

echo put %fullpath%\%%i >> %localFile%

)

if "!localquote!"=="1" (

echo ls "%%i" >> %localFile%

echo dele "%%i" >> %localFile%

echo put "%fullpath%\%%i" >> %localFile%

)

)

The process is always trying to detect is the files are received with quotes or spaces within its name to make the proper handle, but, I haven’t fully test it for such cases.

Testing the script, let’s say we have the script and the desired directory to upload (web) in d:\temp, we execute it like this:

D:\Temp>uploadFiles.cmd Web

The screen output, in this case, is:

Processing .\Web\Administration

Processing .\Web\bin

Processing .\Web\Controls

Processing .\Web\HtmlTemp

Processing .\Web\Images

Processing .\Web\JS

Processing .\Web

Processing FTP

Having configured the script with these values:

set user=myUser

set pass=myPassword

set site=ftp.mySite.com

set base=/www/

set file=\command.ftp

The created commands file is:

user myUser myPassword

cd /www/

pwd

bin

hash

mkdir Administration

cd Administration

pwd

ls CountryAccess.aspx

dele CountryAccess.aspx

put .\Web\Administration\CountryAccess.aspx

ls CountryAccess.aspx.cs

dele CountryAccess.aspx.cs

put .\Web\Administration\CountryAccess.aspx.cs

ls CountryAccess.aspx.designer.cs

dele CountryAccess.aspx.designer.cs

put .\Web\Administration\CountryAccess.aspx.designer.cs

...

pwd

mkdir JS

cd JS

pwd

ls flash_detect.js

dele flash_detect.js

put .\Web\JS\flash_detect.js

ls GeneralFunctions.js

dele GeneralFunctions.js

put .\Web\JS\GeneralFunctions.js

cd ..

pwd

ls Web.config

dele Web.config

put .\Web\Web.config

cd ..

pwd

Here you can find a copy of the script in case you want to test it yourself: uploadFiles.zip