SynchronEX logo

SynchronEX - Advanced File Synchronizer, Backup & FTP

Documentation

Contents

0. Introduction
1. Installation & Setup (GUI Wizards)
2. Advanced Project Examples
3.1 Command Line Syntax 
3.2 Project File Syntax
3.3 Project File Syntax Details
3.4 Advanced API
3.5 Built-In File Systems

4. Technology Background & Special Topics
5. FAQ / Tips & Tricks
6. FAQ / Trouble Shooting

Quick Links:

Incremental FTP Upload

XVS/UXVS Version Tracking ( Journaling virtual file system )

RCY ( Recycle journaling file system )
TFS ( Time-stamp-only virtual file system )

HTTPS ( Secure WebDAV/WebFolders over HTTPS / SSL)
SFTP ( Secure FTP over SSL/TLS )

FTPS ( Secure FTP over SSL/TLS )
XFTP(S) ( Extended FTP file system for bi-directional sync )

IGZ ( GZip auto-compression file system - working on any other file system )


0. Introduction

SynchronEX = Easy-To-Use GUI Frontend + Robust File Synchronizer Engine

SynchronEX is optimized for automation. Independent from GUI, the core engine can be called from command line, from batch files, from scheduled jobs (Wizard interfacing to Windows Job Scheduler) or from other programs. The SynchronEX GUI frontend enables easy setup of projects through Wizards and also displays results of simulations and exec runs nicely. Extra tools like a duplicate file finder complete this software. SynchronEX projects can be executed directly from the Windows Explorer simply like shown below. The project configuration files of SynchronEX are human readable and can also be evolved into complex automation scripts (Python language). A most simple project script would be like this:

#C:/texts/config.syncpy
roots = [ 'C:/texts' , '//laptop/c/texts' ]
excludefiles_rel = ['*.bak','*.obj','bigfile.doc']

  

Windows explorer integration

image on server     

 

 

This way you can organize all your "projects" in regular Windows link folders

  image on server

 

 

Newsletter

Order a free newsletter to be informed about updates, improvements, tips & tricks. Your data is only used for that purpose.

Your e-mail:        


Download / Update

SynchronEX Homepage: Download / Update.

 

Buy Now / Registration

SynchronEX Homepage: Buy Now.

 

Features

SynchronEX Homepage: Feature list


1. Installation & Setup

1.1 Installation

Run the SynchronEX-Setup.exe installer. 

Later versions can be installed over previous versions without prior un-install.


1.2 Usage:

  • GUI : Run SynchronEX from the Windows start menu. After a 'Welcome' dialog run the project Wizard in order to set up your first project. Press the "Simulation" button. If ok, press the "Run" button.  
  • Scheduled Jobs : Consider Menu/Tools/Schedule Task after your projects works upon direct execution. 
  • Parametric scripting & programming: Consider reading the project file syntax, the example.syncpy & template.syncpy in the SynchronEX directory (GUI Wizard: Script Template) and the example scripts below and the 
  • Command Line Usage: command line syntax
     

Windows Explorer Annotation : Changing the default Explorer (double-click) action for .syncpy projects (context menu): 

  • Go to "Windows Explorer/Extras/Folder Options/Types/.syncpy/Advanced ...
  • Select a new default action. 
  • Also new commands can be added : See command line syntax

Annotations for command line usage:

  • The command line option '-s' runs everything as simulation: You can play save without really executing file operations. See command line syntax

  • SynchronEX can be installed/copied to your favorite 'bin' directory (e.g. C:\bin). Consider re-running Menu/Config/Re-Register .syncpy Explorer Context Commands or run 'synctree.exe --setup'

General Annotations:

  • Unless you are doing simple one-way incremental backups, the usage of a synchronization database (-y / syncdb=1) is recommended in order to detect concurrent change collisions. Hard synchronization (syncremove=1) automatically sets syncdb=1
  • Slash & Backslash usage ( \  / ) : In the manual scripted project files usage of slash / is preferred for directory notation like in roots = ['c:/home/texts','//remote/texts']. Otherwise use double-backslash like in 'c:\\home\\texts' or python-raw(r)-string notation like r'c:\home\texts'.

1.3 Project Setup Using GUI Wizards

The SynchronEX Project Wizards can invoked by Menu/File/NewProject(Wizard)


Wizard :: Directory-to-Directory Backup

Enter the source root directory and the target root directory. Optionally follow "Next" to enter files & directories for exclusion/inclusion and optionally follow "Next" to the advanced options.

Path names can be entered either in Windows notation with back-slash ( \  e.g.  C:\data\some\files ) or in modern Unix notation with normal slash ( C:/data/some/files ) or mixed.
Note: In the resulting project script back-slashes are usually escaped as double-backslash in conformity with the Python script language style.


Wizard :: Fast Incremental FTP / FTPS Upload

SynchronEX' fast incremental FTP uses a shadow folder for computing the changed files. Direct comparison over FTP would be slow & weak because file dates are not preserved during FTP transfer, ASCII transmission alters the file, file names may be mangled etc ...
(See TFS option in order to avoid consuming extra disk space). 

If you do not already have a shadow folder, simply create an empty one like C:/www-sync. You can also copy the current source folder contents to the shadow folder in order to start incremental FTP from current state. All file which are not present in the shadow folder or which are changed in comparison to the shadow folder are transfered to the FTP remote tree and after successful transfer synchronized to the shadow folder.

Thus, the shadow folder stands as an exact copy of the current FTP remote tree. ( As a side effect this shadow folder is also very useful as kind of backup copy of the current remote tree, while editing the "edit tree". )

Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. Thus this encrypted password cannot be used or decrypted on an other PC.

TFS: Consider also using the "TFS:" file system (TimeStampOnlyVFS) for use with the shadow folder. Thus the shadow files on TFS virtual file system have zero-length, do not consume disk space and carry only time stamp information. Example:

roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }

FTP File System
FTPS File System
XFTP File System
TFS File System


Wizard :: XFTP ( Bi-Directional FTP/FTPS Synchronization )

XFTP is is an eXtended FTP file system, which stores the original file modification time stamp for a file like  

  ftp:<dirname>/<filename>

in an auxiliary file 

    ftp:<dirname>/XVS/.time.<filename>  

(see also XVS File system). 

For other options of this Wizard see FTP Upload Wizard

The extra time stamp information is written as GMT flat time value. Thus the XFTP file system is suitable for full-featured bi-directional synchronization throughout all time-zones.
XFTPS is to XFTP as FTPS to FTP : it uses SSL/TLS as transport layer. 


Wizard :: Bi-directional synchronization (e.g. Laptop <-> PC/Server)

Use this wizard to set up a true bi-directional synchronization. E.g. between a Laptop and a PC folder.

Enter the 2 root folders and follow the wizard. An especially important selection has to be done on the 3rd wizard page: Detect collisions & synchronize removals. See the extra help of that wizard page.

Note: if you check the "target only" option for a root, that root is never a source (master) folder but only a target (backup) location.


Wizard :: Native DAV Synchronization (HTTP / HTTPS)

WebDAV is a modern full featured bi-directional WAN file system, often considered as more reliable replacement for FTP. This file system is supported by current web servers like  Apache with mod_dav and other DAV servers. DAV over HTTPS is excellent for secure encrypted WAN file exchange.

Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. This encrypted password cannot be used or decrypted on an other PC.

DAV File System
XFTP File System


Wizard Page :: Exclude & Include Files & Directories

Filename wildcards ( *  ? ) can be used anywhere in exclude & include masks. The * wildcard matches also across directories.

Annotations:

Relative path excludes & includes match in any directory. Relative paths are usually used like *.tmp;*.bak;special.file (Files) or CVS;resv (Directories).
For relative paths you can use also patterns mixed with relative subdirectories like   ( e.g. relative file exclude pattern "resv/*.dat"    matches  "C:/texts/sub/sub/sub/resv/myfile.dat" when "C:/texts" is a root)

'Rooted' (absolute(1)) paths match starting at the root folder. 
Example: rooted exclude file pattern "data/sub/my*.txt"  matches on  C:/texts/data/sub/mytext.txt  when 'C:/texts' is root, but does not match C:/texts/sub/sub/sub/data/sub/mytext.txt )

Annotation on Directory Exclusion & Inclusion:

Currently excluded directories can only be revoked by inclusion on the same tree level ! Inclusions at deeper levels have to be bridged - e.g. like..
 
excludedirs_abs = ["*"] 
includedirs_abs = ["sub","sub/data","sub/data/*"]. 

As intended, this example wouuld not re-include a directory "sub/otherdata" because it matches exclude through "*" and does not match a re-inclusion pattern.
A revoking inclusion just like ... 

includedirs_abs = ["sub/data"]. 

.. is not sufficient to work as intended. 
[ In the future this behavior may be changed for trivial tail-only-wildcard patterns in order to detect "include bridges" automatically. ]

Footnotes:

"rooted/absolute": absolute in relation to the root directory (not "absolute" in the sense of absolute file system paths)

Wizard :: Options

Synchronize removals

(Script: syncremove=1)

When checking this option, file/directory removals (remove actions) are also synchronized. 

The trigger for a removals is : File/directory was there during last synchronization or backup; File/directory is missing now => Remove action. Thus it is a kind of "Edge-Trigger". Thus removals can only be detected after an initial first sync run, which creates the sync database as a kind of starting point ! 

Detecting file removals requires the synchronzation database. Thus syncdb=1 (and detecting collisions) is implicit with syncremove=1. Thus it is not necessary, but optional to set syncdb=1 explicitly.

Track a synchronization database and detect collisions of concurrent file changes

(Script: syncdb=1)

When checking this option, the status of files during a synchronization run is remembered in a 'synchronization database' file. This enables the detection of file synchronization collisions upon concurrent file changes. It also enables the detection of removed files (see syncremove=1 above).

An example of a file change collision: You have roots C:/texts and //laptop/texts. After an (initial) successful synchronization the file C:/texts/myfile.txt and //laptop/texts/myfile.txt are both edited : a concurrent file change. Upon next synchronization run it is unclear which file should be the new master file and only the human can decide. Thus when enabled syncdb=1 option, SynchronEX reports a file collision and does not copy this file at all.

This database file has by default the name <syncpy-project-filename>.syncdb and sits by default in the same directory as the .syncpy project file itself. E.g. When you have a project file C:/texts/laptop.syncpy and syncdb=1 option, then a synchronization database file C:/texts/laptop.syncpy.syncdb is created and tracked. You can alter the location of the sync database file by force when setting e.g. ..

syncdbpath="c:/outside/mysync.syncdb" 

.. in  the project script (See scripting documentation).

Continue operation upon errors

The default behavior is to stop the synchronization/backup run whenever any file cannot be copied/overwritten as intended. The 2nd option allows to continue the sync operation when an access error occurs for single files. 

A 3rd option allows to continue the overall project script operation (e.g. multiple syncs and sub syncs ), when the a partial sync run cannot be completed (e.g. when a root/drive/network cannot be accessed ). This option is usually used in Super-Sync project files or in more complex scripted projects.

Threshold time

Only files younger as the date/time specified are copied in synchronizations/backups.

Simple log file:

A simple list of copied (and removed) files is written to this log file.  The default behavior is to overwrite this log file upon each sync/simulation run.

Note: This kind of log-file can be very useful for post-operations upon synchronized files it in scripts. E.g. after an synctree.do() do a Pythonic loop like:  for file in open("_syncsrc.lst"): print file;

Full log file:

The output shown in the console window is also logged to this "full log file". The default behaviour is to append the log contents to the file upon each sync/simulation run. Thus the log file maybe reset manually by deleting it.

The full log file the the file best for tracking synchronizations and backups exactly  e.g. for tracing back problems later. This is especially recommended for unsupervised automatic runs.

Note: When setting up a scheduled sync/backup task with the Scheduler Wizard, the log file is usually set up from inside the Scheduler wizard ( and passed to the project by the -L command line option ).

Verbose console output

Additional sync/copy information is displayed in the console (and full log file). During FTP uploads also a detailed (debugging level) log of all FTP commands is displayed.

Check if project file is located in one of the roots folders (helps avoid muddling sync projects)

By default this option is checked: SynchronEX wants the .syncpy project file to be located in one the sync / backup root folders. This behavior helps to avoid copying project files unintentionally to strange folders. Uncheck this option if you want to store the project file outside of a root folder.

XVS version tracking of overwritten and removed files ( on local & Windows-network-UNC paths )

XVS enables to preserve and track historic versions of your valuable files. This provides the most save sync and backup mode you can think of.

Example: An overwrite of D:/backup/texts/hello.txt first copies D:/backup/texts/XVS/001.hello.txt , 002.hello.txt , etc... as kind of "in-place" backup files.

Thus you can look up and roll back the files you changed to any prior state (manifest during a synchronization run) !

When checking this XVS option checkbox in the options dialog, the xvs: file system tag is attached to all sync roots. In case you want XVS only for one of the roots (e.g. for a backup target), then do not check this checkbox, but prepend 'xvs:' to that root folder (1st wizard page). E.g. have 2 roots like  "C:/texts"   and   "xvs:D:/backup/texts" .

For more XVS documentation see XVS.

 


2. Project Examples ( Use Cases )

This chapter presents projects at project script file level (& command line examples) . Consider using a wizard in the GUI before reading here.

2.1 Synchronize your C:/texts and all sub-directories on Laptop and PC

# c:/texts/config.syncpy
# ( the # marks a comment )
roots = [ 'c:/texts' , '//laptop/c/texts' ]
excludefiles_rel = ['config.syncpy','*.bak']

Double-click config.syncpy for simulation. If everything
seems to be ok, right-click config.syncpy for action "Execute".

or run from command line:

C:/texts> synctree -s     (simulation)

and then

C:/texts> synctree     (real execution)

The configuration file config.syncpy it taken by default, otherwise spec like synctree mysync.syncpy.
 


2.2 Simple command-line-update-sync from . to ./backup without project file

synctree -z -r backup

                 |  |
                 | have . (default) and ./backup as sync roots
                
                 no project file

This is in fact a "nested" sync, because "./backup" is a subdirectory of "."
Of course nested sync roots are handled correctly in order to prevent from (endless) recursive copying.

2.3 Manual incremental (or initial) backup to ZIP archive

#myzipbackup.syncpy
roots = ['.','zip:c:/data/backup.zip']
roots_no_src = [1]

roots_no_src = [1] means : the 2nd root (zip:c:/data/backup.zip) is target only / not a source. This defines a one-way backup.

The zip file is create on the fly if it doesn't exist. If it already exists, it is an incremental backup. Zip file locations maybe use the same way as ordinary directories. Even intra zip directories like zip:c:/data/backup.zip//subdir are possible (note the double slash after .zip ! ).

The same example as manual call:

synctree -z -r zip:c:/data/backup.zip
                 |  |
                 | have . (default) and zip-file as sync roots
                
                 no project file

and including the one-way backup restriction:

synctree -z -r zip:c:/data/backup.zip -n1

-n1    : sync root backup is not a source!

 
2.4 3-point synchronization in a software testing project

#config.syncpy >>>
roots = ['d:/devel/myprj','c:/backup/myprj','//testmachine/devel/myprj']
roots_no_src = [1] # backup is only a sink
excludefiles_rel = ['*.bak','*.sym','*.obj','*.pyc']
includefiles_rel = ['special.obj']
excludefiles_abs = ['config*.syncpy']
excludedirs_rel = ['CVS','rx*']
excludedirs_abs = ['data/bigdata']
syncdb=1 # detect Y-collisions of concurrent changes !

syncdb=1 allows save multi-point authoring. Concurrent changes of files are
detected.

2.5 Super-sync binding multiple sub-syncs

#C:/config.syncpy (the super-sync) >>>
sub_syncs = ['D:/devel/datasheets/config.syncpy',
           'C:/texts/presentations/config.syncpy',
           'D:/mapdata/data.syncpy']
syncdb=1   # such option is inherited by the 
# no roots ... ( as we do not synchronize really something 
#                here, but could do that of course too)

Annotation: If you provide additional parameters in this super-sync-file, the sub-syncs inherit those parameters. (Command line options are effective for all sub syncs too.)


2.6 Grabbing new photos from a digital camera

#D:/Photos/incoming/config.syncpy
roots = ['D:/Photos/incoming/config.syncpy','H:/dcim/100K4800']   #Kodak Camera mounted on H:
roots_no_src = [0] # update one-way from camera!
excludefiles_rel = ['thumbs.db','desktop.ini']


2.7 2-timer sync with different parameters

#C:/home/texts/config.syncpy >>>

roots = ['C:/home/texts','//laptop/c/home/texts']
excludefiles_rel = ['config.syncpy','*.bak']

synctree.do()

roots = ['C:/home/texts','D:/backup/texts']
roots_no_src = [1]
excludefiles_rel = ['*.bak','*.tmp', '*.lst', '*.dat', '~$*']

synctree.do()


2.8 Synchronize non-recursive (no sub directories)

#C:/home/texts/config.syncpy >>>
roots = ['C:/home/texts','//laptop/c/home/texts']
excludedirs_abs = ['*']

Command line variant: C:/home/texts> synctree -z -r //laptop/c/home/texts -x * 


2.9 Hard sync for 2 equally ranked repositories (copy + remove) 

#C:/home/texts/config.syncpy >>>
roots = ['C:/home/texts','//laptop/c/home/texts']
syncremove=1

syncremove is an advanced option, which implicitly uses syncdb=1. If you have a file remove action on one instance (except roots_no_src members) the remove action is detected through syncdb and is also "synchronized" to the other non-roots_no_src instances. Having syncremove=1, it is strongly recommended watching a simulation run (-> remove count) before really executing. Or use syncremove option "-+" in explicit cases.

2.10 Working in-sync on two machines with a mobile (ZIP) disk in between

Also this hopping sync case without direct network connection is no problem with SynchronEX. It is consistent, valid and effective practice, including collision detection on concurrent changes, when syncdb=1 or syncremove=1. 

HOME-Computer    <--->    Mobile (ZIP) Disk    <--->    WORK-Computer

For example on your home computer you could have this:

# c:/texts/config.syncpy
roots = ['C:/texts','Z:/texts']  # ZIP mounted on Z:
excludefiles_rel = ['config.syncpy','*.bak']
syncremove=1

... and on your work computer you have this

 

# d:/texts/config.syncpy
roots = ['d:/texts','H:/texts']  # ZIP mounted on H:
excludefiles_rel = ['config.syncpy','*.bak']
syncremove=1

You could also place both project files central on the ZIP-disk for example as home.syncpy & work.syncpy. With syncdb=1 or syncremove=1 provided, you could also carry a piece of a server 'away' on disk, work on it and re-sync with valid consistent collision detection!

2.11 Find duplicate files from current directory or in 'roots'

C:/texts> synctree --find_dup=fname,content --more20

C:/texts> synctree --find_dup=content > duplist.txt

fname   : filenames must be equal
content : the content must match exactly
--more20: stop display after one page
> duplist.txt : writes the results into a text file for comfortable analysis


Starting recursively from the current dir (c:/texts), this lists the duplicate files that are equal in filename and content, but may differ in time stamp (time criteria missing). The second example writes the duplicates into a text file. Also in the second example comparison is solely based on the file contents. This finds duplicates equal in content, but maybe differing in filename! Read more.

For checking duplicate files in a .syncpy configuration, right-click 'Find Duplicates' on that project file according to the installation.

2.11b Find empty directories starting from current directory or in 'roots'

C:/texts> synctree --find_emptydirs --more10

C:/texts> synctree --find_emptydirs > emptylist.txt

fname   : filenames must be equal
content : the content must match exactly
--more0 : enable keyboard key 's' to stop & page display

> emptylist.txt : writes the results into a text file for comfortable analysis

.. or right-click 'Find Empty Dirs' on a .syncpy project file according to the installation.

2.12 Rotating incremental backup every week to another zip archive

#c:/texts/cycbackup.syncpy

weekperiod = 60*60*24*7
ix = time.time() / weekperiod % 4     # 4 weeks turnaround
bkdir = 'zip:C:/backup/work%02d.zip' % ix  # work00.zip,work01.zip,...

roots = ['C:/workdir', bkdir ]
roots_no_src = [1]               # don't copy from backup

Rotating backup to C:/backup/work00.zip .. work03.zip ... stepping every week. It is not necessary to initialize the zip files beforehand, they are created on-the-fly if not already existing. 
Idea: This script could be evolved to a hierarchical multi-period backup rotating for days, weeks, months ( shutil.copy2( bkdir , "work_month%02d_backup.zip" % iMonth) ...)

Zip files can be used in same style as ordinary directories. Also intra zip directories like zip:c:/data/backup.zip//subdir are valid backup roots (note the double slash after backup.zip ! )

2.13  Incremental FTP upload (of a web site)

#C:/www-edit/ftpupload.syncpy
roots = ['C:/www-edit', 'C:/www-sync']
ftp_shadow={ 1 : 'ftp://user:pwd@myserver.yyy/subdir' }
ftp_ascii=ftp_executable = ['*.php','*.pl','*.py','*.cgi']
excludefiles_rel = ['*.syncpy','*.syncdb','*.bak','*.cdr']
excludedirs_rel = ['_*','CVS']
roots_no_src = [1]   # -> 'C:/www-sync' is target only
syncremove=1

This project uploads (incrementally) your changed web files to your ftp-account in a rush. In this example the website tree is edited in C:/www-edit. The shadow copy is hold/maintained in C:/www-sync. Removed files are also removed on the ftp server.

Password handling: If you don't want to specify the user:pwd as shown you may also set only the user like in 'ftp://user@myserver.yyy/subdir'. Thus, the password is requested interactively. (If you have your password in the project file, don't forget to add '*.syncpy' to the excludefiles_rel list in order to not upload your password.) If your username or password contains the '@' character (example: myname@xdomain): quote it like myname\@xdomain. This is necessary, because unquoted '@' delimits user:pwd from the ftp server name. The GUI wizard takes care of this issue.

Each file is first transferred over FTP, then it is copied to the shadow dir. Thus, on ftp failure or broken transmission, the file will be re-transmitted on the next run.

ftp_ascii=ftp_executable = ['*.php','*.pl','*.py','*.cgi'] ensures that your CGI-scripts are uploaded in ASCII mode and that their executable flag is set on the server. You could also narrow down such declaration to special directories like ftp_ascii=ftp_executable = ['cgi-bin/*.pl','*.cgi'] 

This FTP mechanism can also be used for remote ftp data backup on the intranet/internet. But consider also WEBDAV for that purpose.

See also the Auto-Periodic FTP example here.

Annotation: ftp_shadow is an alias for shadow_copy (new in v3).

2.14  Script example: Recycle backup of files for remove

#C:/music/mp3.syncpy
roots = ['C:/music', 'Z:/mp3']
syncremove=1
src_listfile='_syncsrc.testsync.lst'

synctree.do()
if simulation:
  #on sim. run backs up files for remove
  for line in open(src_listfile):
    if line.startswith("remove "):
      rfile=line[7:].strip()
      tfile='C:/backupmusic/'+os.path.basename(rfile)
      print "rback",rfile,'->',tfile
      shutil.copy2(rfile,tfile)

During simulation mode this sync backs up files for remove to C:/backupmusic. On exec run the hard sync is executed normally. Of course one could program arbitrary execution behavior, for example a joint recycle backup + sync job in one script by calling .do() twice and setting appropriate simulation flag.

2.14b  Script example: Moving a source tree to target

roots = ['src_folder','dst_folder']
roots_no_src = [1]
src_listfile='_mysync.lst'
synctree.do()
if not simulation:
  for line in open(src_listfile):
    if not line.startswith("remove "):
      path=line.strip()
      print "move-remove",path
      try: os.remove(path)
      except: print "couldn't remove",path

After successful copy operation, the source files are removed.

2.15  Auto-Periodic FTP synchronization

There are many possibilities to execute a SynchronEX script periodically by an external scheduler (Windows Task Schedule (Wizard in SynchronEX), crond for Windows, ...). A call like this should be scheduled:

c:\bin\synctree.exe c:\www-edit\ftpupload.syncpy  >> c:\ftpupload.log

However you can also script a periodic sync inside a project file itself like this:

#C:/www-edit/periodicftp.syncpy
roots = ['C:/www-edit', 'C:/www-sync']
ftp_shadow = { 1 : 'ftp://user:pwd@myserver.yyy/subdir' }
ftp_ascii = ftp_executable = ['*.php','*.pl','*.py','*.cgi']
excludefiles_rel = ['*.syncpy','*.syncdb','*.bak','*.cdr']
excludedirs_rel = ['_*','CVS']
roots_no_src = [1]   # -> 'C:/www-sync' is target only
syncremove=1

if not simulation:
  while 1:
    synctree.do()
    config.ftp_release_all()
    print time.asctime(),": waiting 1 hour ..."
    time.sleep(3600)      # waiting 1 hour

 

2.16  Synchronization to Windows mounted WebDAV (DAV; Webfolders like http://server/dav ) shares

This example synchronizes to a WEBDAV folder. WEBDAV folders are useful for WEB uploads as an alternative solution for FTP.
WEBDAV folders can also be used as WAN synchronization point for sharing files with other users.

This script mounts the WEBDAV folder to the drive letter Z:

# C:/www-edit/davsync.syncpy 

print "mounting DAV share to drive letter Z: ..."
os.system("net use Z: http://my.server.yyy/davsub")
##os.system("net use Z: https://server/dav /u:login password")

roots = ['C:/www-edit', 'Z:/mydavfolder']
excludefiles_rel = ['*.syncpy','*.syncdb','*.bak','*.cdr']
excludedirs_rel = ['_*','CVS']
roots_no_src = [1]   # -> 'Z:/davfolder' is target only
##syncremove=1

For using DAV as WAN synchronization location consider removesync=1 and commenting out the line roots_no_src...  like ##roots_no_src=...

Annotations: 

  • repeated trials for mounting Z: will cause a warning message like  "..device  already used". You can savely ignore such message. Or you can use alternative commands suppressing the message: 
    print os.popen("net use Z: http://my.server.yyy/davsub").read() 
  • WEBDAV shares are supported in recent Windows OS versions (XP,2000,3000). 
    There is also a native DAV support (including https) built into SynchronEX. See >>DAV File System

2.17  Sending a notification email after job

Sending a notification email after the job is done:

Login and password can be omitted. Or the password can be loaded from a secured/secret file like: password=open("c:/secure/syncpwd.txt").read()

The mail text body could be rendered from the configured logfile like: 
text="Log:"+open(logfile).read()

2.18 Bidirectional X-FTP Synchronization with Read/Write-Lock

In this example a stale lock has to be erased manually. (Automatic lock timeout to be done)


2.19 Catching an Error and Sending an Email

roots = ['C:/www-edit', 'C:/www-sync']
ftp_shadow={ 1 : 'ftp://user:pwd@myserver.yyy/subdir' }
ftp_ascii=ftp_executable = ['*.php','*.pl','*.py','*.cgi']
excludefiles_rel = ['*.syncpy','*.syncdb','*.bak','*.cdr']
excludedirs_rel = ['_*','CVS']
roots_no_src = [1]   # -> 'C:/www-sync' is target only
syncremove=1

try: 
    synctree.do()
except Exception,v:
    xfile.smtp_mail(
        "smtp.myserver.com",login="mylog",password="mypass",
        from_addr="me@myserver.com",to_addr="you@myserver.com",
        subject="SX job had an error",text="Message:"+str(v))

2.20 Searching the Log File for Errors and Sending an Email

synctree.do()
roots = ['\\temp\\1', '\\temp\\2']
roots_no_src = [1]
logfile='mysync.log'
errorcontinue=1
synctree.do()
errors=xfile.re.findall('(?:Error:|\*\*).*', open(logfile).read())
if errors:
    print errors
    xfile.smtp_mail(
        "smtp.myserver.com", login="mylog", password="mypass",
        from_addr="me@myserver.com", to_addr="you@myserver.com",
        subject="SX job log contains errors", text="Message:"+'\n'.join(errors))

 


3. Script Syntax and Command Line Syntax

For examples of syntax application see Examples.

3.1 Command Line Syntax

Usage: 

synctree.exe [OPTIONs] [PRJSCRIPTFILE]

PRJSCRIPTFILE: 

The ".syncpy" configuration script file. Default: config.syncpy

OPTIONs:

-s           force simulation run; files are not really copied
-y use a sync database (.syncdb) and detect Y-collisions
-z dont use config file (OPTIONSs only mode)
-n0,2 1st and 3rd root directory are not sources
-r /foo directory "/foo" is added to the roots list
-e foo.c "foo.c" is added to the excludefiles_rel
-i foo.obj "foo.obj" is added to the includefiles_rel
-x tmp "tmp" is added to the excludedirs_rel
-k foo.tmp "foo.tmp" is killed everywhere
-d 1.2.02 set threshold date
-t 13:20 set threshold time
-v verbose list (incl. filemtime, size)
-w / -W don't wait for ENTER at the end (-W : wait if error)
-l src.log create log list of all source files
-L sync.log full log of console output to file (append mode)
-+ sync file removals also (syncremove=1)
-c read project file from console/stdin
-p don't prompt for interactive questions, just exit with error
-h small help
--help big help
--remove src.lst :remove files in the list! (create with '-l -s')
--more       stop display after every 10 lines (--more0/20/40/60) --remove=src.lst remove files listed in src.lst! (create with '-l -s') --find_dup=fname,size,time,content,exact find duplicate files based on selected criteria --find_dups  equal to --find_dup=fname,size --find_emptydirs  find empty directories in roots or current working directory --regkey     enter license key directly --setup      bind file classes --template   open a project template for edit --help       big help --gui        start the SynchronEX GUI --guirun      run project in GUI

More annotations on command line options.

3.2 Project File Syntax : Quick Reference

A Complete Synchronex Project File Example

#C:/src/mysrc/test.syncpy

roots = ['C:/www-edit', '//remotecomputer/mysrc', 'd:/backup/mysrc',
'sftp://usr@server.dom/subdir/']

roots_no_src = [2] # 2 --> 'd:/backup/mysrc' is target only

excludefiles_rel = ['config.syncpy', '*.pyc', '*.obj', '*.bak', '~$*']

includefiles_rel = ['special.obj','my*.pyc']

killfiles_rel = ['*.bak']

excludefiles_abs = ['dist/myexe.exe']

excludedirs_rel = ['CVS','tmp']

excludedirs_abs = ['data/bigdata']

# don't copy files older than this:
thres_time = "1.1.2002/14:00"

# record a synchronization database for advanced collision detection (concurrent
# changes), remove control etc.
syncdb = 1

# run these SynchronEX projects in addition:
sub_syncs = ['C:/somewhereelse/backup.syncpy', '../relative/serversync.syncpy']

 

Overview of Parameters in SynchronEX Project Files (.syncpy) / By Example

(See also the file "template.syncpy" in your SynchronEX program folder or open the "General Template Wizard" in the SynchronEX GUI)

roots = ['C:\\www-edit\\', 'Z:/backup/www-edit/']
roots = ['.', '//UNC_remotecomputer/mysrc', 'd:/backup/mysrc/', u'unicode/dir\u043a/]
roots = ['C:\\www-edit\\', 'sftp://user@server.dom/public_html/',
'https://me@davserver.dom/sub/dir/]

config.RegisterLimitedVFS( 
"volX:",
iter_root="c:/backup/mbackup/media%INDEX%", wait_media=0,
auto_mkdir=1, initial_remove=0, count_copy=1,
limit_bytes=14400000, sector_size=512
)
roots = ['C:\\www-edit\\', 'volX:']

roots_no_src = []
roots_no_src = [0,2] # 1st and 3rd root are not source (target only/backup)

automkdir_targets = [0,2] # auto-create this roots if not existing
# (must be target-only too)

excludefiles_rel = ['*.syncpy', '*.log', '*.bak', 'secret.txt']
includefiles_rel = ['special*.bak']

excludefiles_abs = ['presentations/*big*.ppt', 'pagefile.sys']
includefiles_abs = ['presentations/out/big*.ppt']
# note: '_abs' (absolute) means: path from the sync root (not totally absolute)

excludedirs_rel = ['CVS', 'logfiles', 'backup??', '_*']
includedirs_rel = ['backup01']

excludedirs_abs = ['data/bigfiles*', 'install/3rdParty', 'System Volume Information']
includedirs_abs = ['data/bigfiles1']

# those files are removed upon each sync exec! :
killfiles_rel = ['~*.tmp']
killfiles_abs = ['data/temp*.dat']

# define Python callback functions for sophisticated file/dir exclusion:
def cb_is_exclude(rpath):
return os.path.getsize(os.path.join(root[0], rpath)) > 100000
# thus copy only files smaller than 100000 bytes)
def cb_is_exclude_dir(rpath): 
return 'SUBSTRING' in rpath 
# thuss copy only dirs not having 'SUBSTRING' in the name

# don't xvs-autoarchive versions of these files:

xvs_excludefiles = ['C:/data/big*.dat','D:/cache/*','*.tmp']
# note: These are really absolute paths; or paths relative to the current working directory
# (Thus files in different roots can be addressed sperately)

# track synchronization in database (which allows collision detection etc.)
syncdb = 1

# non-default path for the sync database file:
syncdbpath = 'c:/outside/mysync.syncdb'

# synchronizes file-deletions as well (syncdb=1 is set implicitly)
syncremove = 1

# don't sync files older that this date/time:
thres_time = "2.3.2002/14:00:03"

# parameter for seeing time stamps withing 2s granularity as equal
# (use this only for old FAT Filesystems)
even_time = 1

# force simulation mode ON by project script
simulation = 1

# create new directories only when files have to be created there too
lazy_mkdir = 1

verbose = 1 # extra verbose log output

# continue upon more or less severe error types:
errorcontinue = 4 # currently known levels: 0,1,2,3,4

logfile = '_backup.log' # define a log file (append mode)
logfile = '_backup.log', 'w' # define a log file (overwrite mode)
src_listfile = '_backup.lst' # a short/easy parseable log file (overwrite mode)
src_listfile = '_backup.lst', 'a' # a short/easy parseable log file (append mode)

# check if the project file (prjname.syncpy) resides in one of the roots
check_cwd = 1 # (DEFAULT)

# shadow actions (copy/remove) to certain roots to other target root paths too
# (usually used for fast WAN upload syncs by comparing local roots)
shadow_copy = { 1 : 'ftp://user@myserver1.yyy/subdir/' }
2 : 'https://user@myserver2.yyy/subdir/' }
# note: '1' means 2nd root, '2' means 3rd root, ('0' would mean 1st root)
# example roots: roots = ['C:/www-edit', 'C:/sync/server1', 'C:/sync/server2']


# define which files are transfered as ASCII and which are executable
ftp_ascii = ['*.py', '*.pl', '*.php', '*.tcl', '*.cgi', '*.txt']
ftp_executable = ['*.py', '*.pl', '*.php', '*.tcl', '*.cgi']
ftp_execumask = '755' # umask used to set on executables


# force sync copy even for files which show a collision (concurrent changes since
# last sync on more than root). The younger file is copied over the older.
collision_override = 1

symlinks = 1 # don't ignore symbolic links

# follow symlinks and copy the content:
symlink_follow = 1 # 0/DEFAULT : copy symlinks as symlinks

# explicit call to start synchronization (only need when multiple
# syncs have to be run; or for sophisticated error control and log processing):
synctree.do()

# run other SynchronEX project files in addition to the current one:
sub_syncs = ['C:/texts/backup.syncpy','D:/images/backup.syncpy']

3.3 Project File Syntax : Details

roots = [ "C:/web", "C:/web-sync" ]

2 or more roots folders for synchronization/backup.

Wide unicode root folder names (on unicode file systems like NTFS) can be coded like this:

roots = ['.', u'wide-unicode/dir-\u043a\u043a\u043a/]  # /dir-ккк

( u in front of string-delimiter and '\uXXXX' 16-bit HEX character code )

syncdb=1    /    commandline: -y

A synchronization database "<project-file>.syncdb" is tracked. This enables high 
quality synchronization with detection of Y-collisions. A collision is 
detected, when 2 or more instances of a file have changed since last synchronization run.

syncremove=1   /   commandline: -+

This option defines a hard synchronization: positive and negative. If you remove a file
somewhere (except roots_no_src members), the remove action is also synchronized
to the other instances. Usage of a sync database (syncdb=1) is defined implicitely by this
option.

roots_no_src = [0,2,...]   /   commandline:  -n0,2

Defines one-way copying (backup mode). roots_no_src is a list of root indices - starting from 0 - for roots not allowed as sources (master folders). Those roots are 'target-only'.

Example: you have

roots = ['d:/backup0','c:/mydir']

.. then

roots_no_src = [0]

defines that the d:/backup0 directory is not a source, only c:/mydir is a source.

excludefiles_rel   /   excludefiles_abs   /  commandline:  -e
excludedirs_rel   /   excludedirs_abs   /  commandline:  -x
includefiles_rel   /   includefiles_abs   /  commandline: -i
includedirs_rel   /   includedirs_abs 

List of file/directory patterns for exclusion & inclusion. see also Wizard Page Exclude & Include.
See also annotation on wildcards.


cb_is_exclude / cb_is_exclude_dir

Extra pythonic callback function for checking exclude/include of files or directories. If this callback is a valid non-None function, this callback is called for each file / directory. The callback function must take the rooted path as 1st and only argument. The function must return one of these values:

0 or False  : don't force exclude but check other exclude masks
1 or True   : exclude ! (don't check other masks)
2           : include ! (don't check other masks) 

Example: Copy only files smaller than 100000 bytes) : 

def cb_is_exclude(rpath): 
    return 100000 < os.path.getsize(os.path.join(root[0],rpath))

def cb_is_exclude_dir(rpath): 
    return 'ZZZ' in rpath   # ZZZ string in dirname?

Precedence of exclude/include patterns / rules:

  • by default all is included (except XVS files) - lowest priority
  • except excludes
  • but includes included = excludes from exclude = force (overrides)
  • dynamic exclude by functions cb_is_exclude / cb_is_exclude_dir (exclude with highest priority)

thres_time='Dec-31-03/14:00' / commandline:  -t  -d

If this threshold time parameter is set, no files before that time are copied. The thres_time value may eigther US or European string format like "31.01.2003/15:00" or a flat 'seconds since 1970' (c-time/time.time()) value.

An example for a dynamic threshold time (copy only files of the last 24 hours):

thres_time = time.time() - 24*60*60

sub_syncs = ['other.syncpy','d:/data/mirror.syncpy']

Defines a super project calling other projects. If you don't provide 'roots=' in the super-project, the super-project doesn't copy anything itself. Anyway if you set additional parameters in  the super-project, those parameters are inherited by the sub-projects (including the simulation flag). But in most practical cases a super-project file contains only a single "sub_sync=" line.

logfile="my.synclog"
logfile="my.synclog","w"

Configures a log file. All text which appears in the console output also goes to this log file By default new content is appended to an already existing log file (open mode "a"). The second example (open mode "w") overwrites the existing log file. 

shadow_copy={ 1 : 'd:/backup/diff-dir' }

shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }

shadow_copy copies/removes files which are written/removed at the attached root. The integer-key [1 in the example] defines the index of the attached root.

Example:

roots = ['c:/www-edit', c:/www-shadow]
shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }

Thus changed files, which are synchronized from C:/www-edit (root #0) to the C:/www-shadow (root #1) folder, are also copied to (or removed at) the ftp server (attached to root #1 by shadow_copy=... ). 

Annotations on FTP-URL Format:

The FTP-URL format including username (user) and optional password (pwd) and optional port number (25) is the same as used in internet browsers. If the password (pwd) is omitted, then it is requested interactively.

If your username or password contains the '@' character (example: myname@xdomain), then it has to be quoted by backslash like myname\@xdomain. This is necessary, because unquoted '@' delimites the "user:pwd" from the ftp server name.

FTP Passive Mode (PASV):

Some old ftp servers do not support passive ftp mode (PASV). In that case an error like "command not supported" is yielded. Also some network routers may cut off passive ftp transfers ("connection reset by peer", ...). You may watch FTP debug output and/or switch to non-passive mode by inserting:

#C:\www-edit\ftp-nonpassive.syncpy
verbose = 1
ftp_passive = 0
roots=...
...

ftp_ascii = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]

Files matched by this mask are schedule for ASCII FTP transfer. ASCII transfer is usually required by CGI scripts. 

ftp_executable = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]

Files matched by this mask are set as executable on the FTP server by "SITE CHMOD 755 <fname>" command. In case your FTP server (like Microsoft IIS FTP Server) doesn't support unix executable flags or SITE CHMOD and errors are reported when CHMODing masked files, set ftp_executable = [] 

ftp_execumask=755

Sets the "umask" file attributes, which are set by the FTP SITE CHMOD command for files matching the "ftp_executable" mask. Default: 755

lazy_mkdir=0

When lazy_mkdir is set to 1, the directories are only created during synchronization, when files are created. Thus empty directories, would not be re-created on the other side.

errorcontinue=0

Continue operation upon errors? :

  • 0 = never
  • 1 = upon single file errors
  • 2 = <reserved value>
  • 3 = if project continuable
  • 4 = even if root error in subprojects ( see "sub_sync" )

Python Scripting / synctree.do()

You may add any python code to .syncpy project file. The python modules os,shutil,fnmatch,glob,string,re,time and more are available (and imported). 

If you do not have at least one call to synctree.do() or synctree.dontdo(), a synctree.do() call is done automatically after parsing the project script.


Annotation on filename wildcards in exclude & include masks:

The mask filters match using filename wildcards ?, * and [seq].  
seq: character ranges, e.g. [A-Za-z] .

A more complex example: excludefiles_abs = ['data/my*/test[A-Z]/db??store*.dat']
Real filenames cannot contain ? or * but [ or ]. if you want to match an explicit file/dirname like synctest[4] you have to quote the brackets like excludefiles_abs = ['data/synctest[[]4[]]/*']. In roots and sub_syncs you must not quote, because these are direct paths, not patterns.

3.4 Project File Syntax : Advanced API


xfile.X_remove_recursive

Removes files and folders recursively! By default a confirmation dialog protects this operation.

Example:

xfile.X_remove_recursive("/my/path/to/folder", removethis=0, confirm=1)

 


Annotations on Command Line Parameters:

--more/--more0/--more20/--more40/--more60

Enables the integrated 'more' console display filter. Thus console output stops after 10,20,40 or 60 lines. --more0 switches on a more-filter, which is initially unlocked ('f' key), but is the same as --more10 + pressing key 's'/SPACE.

The more-filter reacts to the following keyboard keys:

  • any key during hold [except Ctrl-C and 'f'] :: display next page
  • 'f' during hold :: (follow) unlock the more-filter 
  • 's' / SPACE during operation :: (stop) lock the more-filter again
  • Ctrl-C : break execution ( single files are always copied completely before the break is done )

--find_dup=fname,size,time,content,exact

Finds duplicate files based on the selected criteria (exiting option). 

fname the filenames must be equal
size the file sizes must be equal
time the file time stamps must be equal
content the file contents must be equal (implies 'size')
exact all criteria above must match

See also the graphical duplicate file finder in the GUI !

--find_dups

Finds duplicate files. same as --find_dup=fname,size. This is a exiting option not starting a sync.

--find_emptydirs

Finds empty directories in roots or '.'

This is a exiting option not starting a synchronization.

3.5 Built-In File Systems

This chapter presents a list and short description of current built-in file systems. ( SynchronEX' Extensible Virtual File System Technology enables also custom made file systems. )

Normal Local or Windows UNC Network File System

Examples:

   roots = [ 'c:/www', 'd:/web' ]

   roots = [ 'c:/www', '//server/web' ]

 

ZIP File System (ZIP:)

SynchronEX synchronizes seamless into zip archives or even into intra-ZIP paths

Examples:

roots = [ 'c:/www', 'zip:d:/backup/web.zip' ]

roots = [ 'c:/www', 'zip:d:/backup/web.zip//sub/subdir' ]
roots_no_src = [1]

XVS File System (XVS:)

See also XVS Overview, Recycle (RCY) File System.

XVS is a sort of journaling file system on top of the Local File System, which keeps past versions of files by automatically backing them up to a XVS folder ( ./XVS or ./.XVS relative to the file folder), whenever a file is to be overwritten or (re)moved by SynchronEX. The XVS folder is created automatically on demand. 

An XVS backup file like is created like :   /sub/folder/XVS/005.normal-file-name.txt .

The index number (005 in this example) is counted upwards for each archived version of a file. An XVS backup file is created, when the original file is about to be (1) replaced by a copy-operation, (2) written/overwritten, (3) removed/moved.

The default XVS backup folder "XVS" can be changed via config.SetXVSFolder. Use XVS folder names other '.XVS' or 'XVS' only, when there is a good reason for. (These 2 standard XVS folder names are on internal folder exclusion in the style of "excludedirs_rel" by default.)

Example 1:

   roots = [ '/work/www', 'xvs:/dist/www' ]

Example 2:

   config.SetXVSFolder('.XVS')   # changes default 'XVS' or '.XVS'

   roots = [ 'c:/www', 'xvs://server/web' ]   # XVS on a UNC share

UXVS File System (UXVS:)

Operates the same way as the XVS File System, just it can work on top of ANY other filesystem - including Wide Area Network file systems like sftp, DAV, etc.  
Note: On the local file system UXVS it is a little slower - thus prefer XVS there.
Note: The default backup folder for UXVS is ./.XVS (from SynchronEX 4).

Example:

   roots = [ 'c:/www', 'uxvs:sftp://www@myserver.dom/htdocs' ]


Recycle File System (RCY:)

See also XVS File System.

On Windows: Archives past versions of file which are about to be overwritten or (re)moved by automatically moving them to the Windows Recycle-Bin folder ("Trash basket").

On UNIX: Archives past versions of overwritten or (re)moved files on local file systems by automatically moving them to a "./.recycle" folder. Same operation as with the ./XVS or ./.XVS folders of the XVS File System, just uses a differently named folder (which is intended to be cleaned less considerate than .XVS).

Example:

   roots = [ 'c:/www', 'rcy:d:/www-sync' ]

Time-Stamp-Only File System (TFS:)

TimeStampOnlyVFS upon Local File System. Usually only used for target-only roots. Files on TFS are created with zero-length (no content), but carry time stamp information. E.g. Consider using the TFS file system for use with FTP/WEBDAV or other shadow folders. Thus the shadow files on TFS have zero-length, do not consume disk space and carry only time stamp information. 

Example:

roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }

NULL File System (NULL:)

Dummy sink file system similar to "/dev/null" on UNIX, but it is a sink for folder trees.

Examples:

#example: MO/Zip/Floppy drive read test
roots = [ 'z:/data', 'null:' ]

#example: full non-incremental copy / mirroring
roots = [ 'c:/www', 'null:' ]
shadow_copy={ 1: '//remote/non-incremental-mirror-dir' }

#example: force FTP full non-incremental copy 
##roots = [ 'c:/www-edit', 'c:/www-shadow' ]
roots = [ 'c:/www-edit', 'null:' ]
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }

FTP File system (FTP:)

The plain FTP File system. Suitable mainly for using it with shadow_copy / ftp_shadow. Files on bare FTP files systems cannot be compared directly, because FTP is a weak directory protocol for synchronization: File time stamps cannot be preserved during copy and the files itself are (partially) exposed to ASCII conversion etc. 

Example:

roots = [ 'c:/www', 'c:/www-shadow' ]
roots_no_src = [1]
shadow_copy={ 1: 'ftp://me@webserver.com/www' } # password asked interactively
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }

Advanced Configuration:

  • Maximum number of retry attempts on interrupted FTP transfer (default=2):
    config.setFTPRetry(3)
    config.setFTPRetry(retrymax=3,retrywindow=55)

    retrywindow is the number of seconds after which the retry-counter during long FTP file transfers is reset (anew retrymax attempts)
     
  • Using an extra transfer file during FTP(S),SFTP,DAV in up-direction is enabled by default (in order to avoid partial files on live servers). the use of transfer file can be switched off with
    config.useXFERFile(0)
     
  • Change the connection timeout (Time after which a new connection will be used)

    config.setFTPConnTimeout(300.0)   # change timeout to 300 seconds

    (Default is 120 seconds)

See also: XFTP filesystem for direct bi-directional transfer compatible with plain FTP.

FTPS / FTP over SSL/TLS File system (FTPS:) 

FTPS is a FTP File system using secure SSL/TLS as transport layer. FTPS over SSL/TLS is supported by most modern FTP servers. If it doesn't work with your FTP server you will get an error like "550 command not understood" after the FTP "TLS" command (set "verbose=1"). 

Example:

roots = ['c:/www', 'c:/www-shadow']
roots_no_src = [1]
verbose = 1      # verbose display of FTP commands
shadow_copy = {1:'ftps://me@webserver.com/www'} # password asked interactively

shadow_copy = {1:'ftps://me:pwd@webserver.com/www' } 

See also: XFTPS file system

Advanced Configuration:

  • Set x509 Client Keys & Certificates by adding this to the project script:

    config.SetSSLKeyCertFiles('www.davserver.dom',
      key_file='/path/keyfile.x509', cert_file='/path/cerfile.x509')

    (The host name must be the same as used in the connection URL)

For advanced configuration see also FTP file system documentation.

XFTP / XFTPS File system (XFTP: / XFTPS:)

This is an eXtended FTP File system, which stores extra original file modification time stamp information in   ftp:<dirname>/XVS/.time.<filename> (see XVS File system). The extra time stamp information is written as GMT flat time value. Thus the XFTP file system is suitable for full-featured bi-directional synchronization throughout all time-zones. XFTPS is to XFTP as FTPS to FTP : uses SSL/TLS as transport layer. 

Example Project Script:

roots = [ 'C:/www-edit', 'xftp://me@myserver/root' ]
excludedirs_rel = ['.ftplock']
ftp_ascii = []
flock=roots[1]+'/.ftplock'
if not simulation: xfile.X_mkdir( flock )
try:synctree.do()
finally:
    if not simulation: xfile.X_rmdir( flock )

It is not recommended to use XFTP for normal incremental FTP uploads, because direct file comparison over (X)FTP is rather slow.  Thus if you have many upload files and a small number of download files, it is recommended to split the job into 2 parts and use a multi-project. 

For advanced configuration see also FTP file system documentation.

SFTP / SSH File Transfer Protocol ( SFTP: )

SFTP is the secure encrypted SSH(2) File Transfer Protocol. It is a modern, very reliable, secure and fast WAN file transfer protocol. Despite its name, SFTP is technically very different from FTP / FTPS and uses different software. 

Most UNIX Servers offer SSH/SFTP access through the "sshd" daemon. Yet SFTP support often has to be enabled extra in the configuration for sshd in a file like /etc/ssh/sshd_config - by a line like:

#sshd config file /etc/ssh/sshd_config
...
Subsystem sftp /usr/lib/sftp-server

Logon can happen by password or by RSA/DSA private key (with optional key password). Required passwords are asked interactively if the are not set inside the script - and can interactively entered passwords can optionally be saved machine-encrypted after confirmation.

Example Project Scripts:

# wizard_SFTP_Sync
# open password logon (not recommended unless "sycntree -c" from other software)

roots = ['C:/www-edit', 'sftp://myself:directpassword@sftpserver.dom/subdir_from_myself']
# wizard_SFTP_Sync
# password will be asked interactively and optionally be stored machine-encrypted after confirmation

roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom/subdir_from_myself']
# how to point to a absolute server folder (relative from the root folder / ):

roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom//subdir_from_root']
# override the the default '~/.ssh/known_hosts' file
config.setSFTPKnownHostsFile('~/sshsx/known_hosts.txt')

# trigger private key logon (per server)
config.setSFTPPrivateKey('sftpserver.dom', '~/sshsx/private_RSA_key_sftpserver.txt')
config.setSFTPPrivateKey('otherserver.dom', '~/sshsx/private_DSA_key_otherserver.txt', 'funny_pwd')

# change connection timeout to 120 seconds (default: 60)
config.setSFTPConnTimeout(120.0)

# suppress interactive questions for keys, known hosts etc. / just raise errors
config.setInteractive(0) # like -p from commandline

roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom:2277/subdir_from_myself']

Private RSA/DSA Key Logon (with or without extra key password)

The following default locations are searched for private SSH keys: ~/.ssh/id_rsa , ~/.ssh/id_dsa

Otherwise you can set a path to a private key (per server) explicitely by this function:

config.setSFTPPrivateKey(host, priv_file_or_key, password=None)

Note: in case of an encrypted private key, the key password can also be entered interactively - and optionally be stored machine-encrypted by confirmation for later auto-use.

Note: the priv_file_or_key can be an immediate RSA/DSA key string as well (instead of a path to a file).

A Known Hosts File different from default '~/.ssh/known_hosts' can be set like:

config.setSFTPKnownHostsFile(know_hosts_file)

Advanced Configuration:

Change the connection timeout (Time after which a new connection will be used):

config.setSFTPConnTimeout(120.0)

Notes: ~ is %USERPROFILE% on Windows. Thus '~/.ssh/known_hosts' on Windows for example expands to %USERPROFILE%/.ssh/known_hosts . Check the latter location by copying the folder path into the Windows Explorer address navigation. Typically this ends up in a folder like "C:\Documents and Settings\user\.ssh"

WebDAV File System ( DAV: / HTTP: / HTTPS: )

This is the DAV over HTTP(S) file system - also called "WebDAV" or WebFolders". WebDAV is a modern full-featured bi-directional WAN file transfer protocol extending the one-way Hyper Text Transfer Protocol (HTTP). Thus DAV file/folder URLs have the same layout than normal HTTP-URLs - e.g.: 'https://myself@davserver.com/davroot' . DAV is often considered to be a more reliable replacement for FTP. DAV is supported by current web servers like  Apache (with mod_dav installed) and other DAV servers. A DAV server like Apaches mod_dav usually has to be configured (".htaccess", "http_conf") in order to allow (password protected) DAV read/write/property-change access to (certain) folders. If the web server of your ISP enables DAV, you should have documentation from your ISP about how to access and use DAV. DAV over HTTPS is excellent for secure encrypted WAN file exchange.

Example:

#wizard_Native_DAV_Sync
roots = ['C:/www-edit', 'https://myself@www.davserver.dom/davroot']

Advanced Configuration:

Proxy support for DAV (HTTP & HTTPS): 

  • Windows: A proxy server is used automatically, if there is a http proxy Windows setting (in MS InternetExplorer) - or alternatively, if the environment variable http_proxy is set: 
    SET http_proxy="http://www.someproxy.com:3128"
  • UNIX: A proxy server is used on UNIX automatically, if the environment variable http_proxy (case sensitive) is set:
    % http_proxy="http://www.someproxy.com:3128"
    % export http_proxy
  • Use of proxy can be forced inside the project script like:
    config.setProxy('http','http://www.someproxy.com:3128')  #forces proxy ON
    config.setProxy('http',None)  # forces proxy OFF
  • Proxy with authentication:
     config.setProxy('http','http://user@pwd:www.someproxy.com:3128')

Extra Transfer File:

  • By default now (v3.0.7) SynchronEX uses an extra transfer file during DAV upload (in order to avoid partial files on live servers). 
    That method employs the DAV HTTP MOVE method, which is supported by almost any DAV server ( required in the minimal DAV standard RFC 2518 )
     
  • Usage of the transfer file method can be switched off with:

    config.useXFERFile(0)

DAV over HTTPS: Set x509 Client Keys & Certificates:

Set x509 Client Keys & Certificates by adding this to the project script:

config.setDAVKeyCertFiles('www.davserver.dom',
  key_file='/path/keyfile.x509', cert_file='/path/cerfile.x509')


(the host name must be the same as used in your DAV URL)

Change the connection timeout (Time after which a new connection will be used) 

config.setDAVConnTimeout(20.0)   # change timeout to 20 seconds

(Default is 10 seconds)

 

XWebDAV File System ( XDAV: / XHTTP: / XHTTPS: )

A special DAV File system, which stores file time stamps in extra files and not in DAV-properties. Suitable for simple DAV-Servers which do not support the DAV-Method "PROPPATCH" (for example Tomcat). Similar to XFTP vs. FTP

Example:

roots = ['C:/www-edit', 'xhttps://myself@davserver/davroot']

For special configuration see also (normal) DAV file system documentation.

GZip Auto-Compression File System

Compresses and decompresses files individually on the fly on any other file system. 

The (created) compressed files on the target file system have an extra extension '.sx.gz' attached by default.

Examples:

roots = ['C:/www-edit', 'igz:http://myself@davserver/www-gz-backup']

roots = ['C:/www-edit', 'igz:sftp://myself@server/gz-backup']

Change the default file extension for compressed files.

config.SetIGZExtension('.gz')

Auto-Encryption / Batch-Conversion File System

Encrypts and decrypts or batch-converts files individually on the fly by using any command line stream coder (pipe), which is installed on the system (e.g. GPG). Works on top of any other SynchronEX file system

The encrypted files on the target file system have an extra extension '.sx.enc' added by default.

Examples:

config.SetIENCPipe('ienc:', 
                   'gpg -e -r USER1 --batch --always-trust', 
                   'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
                   )  
roots = ['/home/myself/www-edit', 'ienc:http://myself@davserver/www-encrypted-backup']

config.SetIENCPipe('ienc:', 
                   'gpg -e -r USER1 --batch --always-trust', 
                   'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
                   key='_mygpgpassphrase_'
                   )  
roots = ['C:/original', 'ienc:D:/encrypted']

config.SetIENCPipe('ienc:', 
                   'gpg -e -r USER1 --batch --always-trust', 
                   'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
                   keyfile='C:/mygpgpassphrase_is_in_here.txt'
                   )  
roots = ['C:/www-edit', 'ienc:igz:sftp://myself@server/encrypted-gz-backup']

.


4. Technology Background Knowledge

The SynchronEX software is about incremental multi-root file&object-tree synchronization, backup, updating backup, FTP, incremental ZIP archive update, file tree mirroring, reliable copying of file trees, scheduling projects, finding file duplicate files, finding empty directories, advanced browsing and finding of files & other file tree related work.  

SynchronEX = robust engine + GUI frontend. The engine can work independently and silently. E.g. can be called from batch jobs, schedules or other software.

SynchronEX internally uses a sophisticated extensible virtual file system (VFS) technology. Thus SynchronEX bridges seamlessly into FTP, WEBDAV, ZIP and other widely used file system protocols. Other VFS add special capabilities to local file systems. E.g.:  XVS,LimitedMediaVFS,TFS,NULL .... : see File Systems

Synchronization is based on file time stamps. 

A Simulation mode enables evolving arbitrary complex projects without risk. 

SynchronEX can use a synchronization database file when advanced comparisons have to be done, which cannot be accomplished simply by direct file/directory comparison: A Y-collision detection feature (-y /syncdb=1) provides protection against dangerous overwrites of concurrent file changes. Hard synchronization including the synchronization of file/directory removals also makes use of the synchronization database.

Usage of SynchronEX is very simple for common tasks (GUI Wizards). On the other hand you are able to schedule arbitrary complex patterns, or even evolve your project script into a full blown Python automated program.

SynchronEX uses human readable project scripts which can be set up with wizards and can be evolved to more complex scripts/programs. It is recommended to place projects files into one of the root folders of interest. 

Exclude & Include: There exist 5 channels of exclude/include masks: For files OR directories, relative OR rooted (absolute) + a logical callback function. SynchronEX' mask policy is "Take all minus excludes but take includes.". Rooted (absolute) masks have priority over relative masks. In addition a logical callback function, which is called for each file or directory, can bias exclusion 3-state: force exclude, force include, delegate decision to upper masks.

Complex python scripts can be directly executed inside the project script. An arbitrary number of synch-jobs can be executed programmatically (e.g. loops;  synctree.do() calls ) in a project script. ( If no such call is done explicitly, a synctree.do() is automatically scheduled after parsing the project script. )

Copy directions in synchronizations are scheduled based on file time stamps. Exact 1-hour distances with no change in file size, are scheduled as "IN SYNC" because ( common incompatible daylight savings handling on different computers ).

SynchronEX takes special mean to defend against synchronization inconsistencies upon cable interrupts/OS crash. E.g. it uses special time stamps during copy and block OS interrupt signals.

Commandline paramaters take priority over project file settings.

4.1 Special Topics

XVS/UXVS Version Tracking / Journaling File System

XVS enables auto-back-up of former versions of files which are changed or deleted during a synchronization run or backup run. This is the most save mode at all for file synchronizations and backup tasks: You can not anymore loose files or loose old versions of your files unintentionally. XVS can be used together with syncremove=1 option ( hard synch mode ) in order to set up extremely effective synchronizations.

XVS principle

The principle of XVS is simple: If any file on an XVS tagged file system (e.g. root "xvs:C:/texts" ) is overwritten or removed during a synchronization run, the old file is first moved to a XVS backup location. E.g. file C:/texts/mytext.txt  is backed up as C:/texts/XVS/000.mytext.txt , then on next overwrite to C:/texts/XVS/001.mytext.txt, then C:/texts/XVS/002.mytext.txt and so on. The ./XVS sub-folder is generated automatically on the fly when necessary.

You can tag any local or network root path in SynchonEX for XVS processing. ( This does not alter anything in your real file system of that disk drive. It is just a virtual file system inside SynchronEX itself. )

Thus you can prepend a "xvs:" tag to a root directory in order to declare it as XVS virtual file system. In the GUI you can also simply use the XVS checkbox or the dedicated XVS wizard which activates the XVS for all roots by default. Example:

XVS option

roots = [ "xvs:C/texts", "xvs://server/texts" ]

This way you get historic backups for all versions of the files. At any time you can remove XVS backup files, e.g. when they consume a lot of disk space. The GUI tool in Menu/Tools/XVS Cleanup can be used to quickly bulk sweep XVS backup files.

The default maximum of backup versions of a file is 10000. If more versions are to be generated, an error is thrown. You can change this number e.g. by 

config.SetXVSLimit(100) 

.. in a script.

XVS system can be applied to local/UNC filesystems only. The UXVS file system ("Universal XVS") is the same as the SXV file system, but works on _any_ other filesystem. UXVS works a little slower generally:

roots = [ "uxvs:sftp://user@server/texts", "xvs://server/texts" ]

 

"What has XVS to do with CVS or MS SourceSafe?"

Answer: Not much. XVS works completely automatic without needing to know anything about  repositories, tags, check-in/check-out. Is is probably the most easy-to-use method for version tracking at all. It works without any problems in any office or development environment where valuable documents of any nature are edited iteratively
But of course it is not a total replacement for CVS with controlled version tracking ( multi-year, branches etc. )

However even in source code development environments there are situations where it is reasonable to combine XVS and a version tracking system like CVS. E.g. XVS could take care of fast iterations, personal iterations and automatic daily versioned backups while CVS/SourceSafe does the long-term manual controlled version tracking. 

Another special application with CVS repositories: SynchronEX & XVS are reported to successfully and consistently keep a CVS repository itself mirrored on 2 or more locations for certain applications, where there is no permanent network connetions  (see also "collision detection" for consistency ).


5. FAQ / Tips & Tricks

Forcing synchronization upon collisions (syncdb=1) ?

First execute the project, overview the collisions and do all necessary manual merge work. 
In order to really force synchronization solely based upon file times, simply erase the .syncdb database file of that project and re-execute the project.

 

Finding files duplicate in content but differing in filename ?

  1. install a diff, filecompare, fc, .. tool 
  2. cd to your root. e.g.  cd /myfiles
  3. execute: synctree --find_dup=content,fname > lA.txt
  4. execute: synctree --find_dup=content > lB.txt
  5. then for example: diff lA.txt lB.txt

 

Synchronizing two computers with a mobile (e.g. ZIP-Drive) Disk in between ?

Also this case is no problem with SynchronEX and very save, valid and effective with syncdb=1 / syncremove=1

HOME-Computer    <--->    ZIP-Disk    <--->    WORK-Computer

For example on your home computer you have this:

# c:/texts/config.syncpy
roots = ['C:/texts','Z:/texts']  # ZIP mounted on Z:
excludefiles_rel = ['config.syncpy','*.bak']

... and on your work computer you have this

 

# d:/texts/config.syncpy
roots = ['d:/texts','H:/texts']  # ZIP mounted on H:
excludefiles_rel = ['config.syncpy','*.bak']

You could also place both project files central on the ZIP-disk for example as home.syncpy & work.syncpy. With syncdb=1 provided, you could also carry a pice of a server 'away' and work on it and re-sync with valid collision detection.

 


6. FAQ / Trouble Shooting

" Warning Message: project / working directory not in roots ; consider check_cwd=0 "

  • The checkbox is on the 3rd option page of the wizard. or write "check_cwd=0" in the script.
  • "not in roots": By default SynchronEX wants the project script file (.syncpy) to be placed in one of the roots folders of the project in order to prevent from executing wrong projects. check_cwd=0 allows the project to be executed from anywhere

" I have lots of collisions (syncdb=1). How do I get rid of them? "

  • You have real collisions if you have edited/changed files on both (more than 2) sides. In that case you'd have to create a correct synchronized state by manually copying the files appropriately.
  • Maybe the daylight saving settings of two machines, which are used during synchronization, are incompatible. In case you want to force copying the most youngest files: Clear the .syncdb database file of this project, simulate & execute if directions ok.

" How to log on to a Windows folder on a remote machine automatically (with password)"

  • Insert the following on top of your project script:
    #force auto-logon on remote windows folder:
    os.system(r"net use \\COMPUTERNAME\RemoteFolder /u:user pwd")
    roots = ['c:/myfolder', r'\\COMPUTERNAME\RemoteFolder']
    ....
    #force auto-logon on remote windows folder with drive letter mapping:
    os.system(r"net use Z: \\COMPUTERNAME\RemoteFolder /u:user pwd")
    roots = ['c:/myfolder', 'Z:/']
    ....
    More on Windows command "net use".

" I want to call a SynchronEX project from C, Delphi, Script ... with console window hidden "

  • use e.g. a function like "_popen" (C/C++) of that programming language and call "synctree.exe <options>..."

" How do I get a detailed log of FTP commands issued? "

  • Select the "verbose" mode in the options wizard page (script: "verbose=1").

" SynchronEX won't create directories on my FTP server "

  • In order to be fast, SynchronEX creates directories over FTP upon "directory missing error"
  • Maybe your server yields a non-standard error message => please set verbose=1 and report the error message to us.

" Is it possible to compare two directories with SynchronEX but copy the changed files to a third directory? "

  • Yes, use the differential (shadow) synchronization wizard ( shadow_copy={...} )

" A backup to ZIP archive always appends?  "

  • A replacing would be done, by forcing deletion of the ZIP file in the script: 

    os.remove( root[1] ) 

    A true overwrite/delete of a file intra-ZIP is not a job for a fast sync software like SynchronEX, because the whole ZIP file had to be rewritten anyway. E.g. try to delete a intra-zip file in XP explorer or winzip: it takes a long time. Thus SynchronEX always appends to the ZIP file. 
    When you force the removal + complete rewrite as above, its (nearly) as fast as an in-place deletion of files in a zip file would be

" I have problems with SynchronEX on local area networsk & Windows firewall"

Go to: Control Panel -> Network Connections -> Local Area Network -> Properties

On the General Tab: 

  • Internet Protocol (TCP/IP) -> Properties -> Advanced -> WINS -> NetBIOS setting is either set to Default or Enable NetBios over TCP/IP

  • File and Printer Sharing for Microsoft Networks must be checked

On the Advanced Tab:

  • Enable Microsoft Firewall by checking the checkbox.

  • Press Settings push button to configure advanced firewall settings:

On the Services tab:

  • Uncheck all checkboxes to disable all default services. 

  • If your computer hosts an FTP site, or web server, etc. you’ll have to leave the appropriate check box checked.

  • Press the Add push button to add Port 137 UDP to firewall

  • Press the Add push button to add Port 138 UDP to firewall

  • Press the Add push button to add Port 139 TCP to firewall

On the ICMP tab

  • Uncheck all checkboxes to disable all ICMP message support.  You can check Echo (ping) if you want PING to work.


" How to synchronize file from very long paths (>260 chars in path name)? "

There are 2 alternative options:

  • use  config.SetAutoLongPaths() .
  • use Windows path escapes r"\\?\"  and r"\\?\UNC\"
See: http://msdn.microsoft.com/en-us/library/windows/desktop/...#maxpath

 

 

SynchronEX box



 SynchronEX main page.

Buy Now

 Change Log


Stars:

SynchronEX 5 stars


Other Products

Xellsoft's HarvEX : eBay Automation Software

Xellsoft Custom Software


Related Resources

Python scripting in SynchronEX