|
|
|
|||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||
|
|
|
#C:/texts/config.syncpy
|
Windows explorer integration
|
This way you can organize all your "projects" in regular Windows link folders
|
Order a free newsletter to be informed about updates, improvements, tips & tricks. Your data is only used for that purpose.
Download / Update.
SynchronEX Homepage: Buy
Now.
SynchronEX Homepage: Feature list
Run the SynchronEX-Setup.exe installer.
Later versions can be installed over previous versions without prior un-install.
project file syntax, the example.syncpy
& template.syncpy
in the SynchronEX directory (GUI Wizard: Script Template) and the example scripts below and the command
line syntaxWindows Explorer Annotation : Changing the default Explorer (double-click) action for .syncpy projects (context menu):
command line syntax
Annotations for command line usage:
The command line option '-s' runs everything as simulation: You can play
save without really executing file operations. See command line syntax
General Annotations:
The SynchronEX Project Wizards can invoked by Menu/File/NewProject(Wizard)
Enter the source root directory and the target root directory. Optionally follow "Next" to enter files & directories for exclusion/inclusion and optionally follow "Next" to the advanced options.
Path names can be entered either in Windows notation with back-slash (
\ e.g. C:\data\some\files ) or in
modern Unix notation with normal slash (
C:/data/some/files ) or mixed.
Note: In the resulting project script back-slashes are usually escaped as
double-backslash in conformity with the Python script language style.
SynchronEX' fast incremental FTP uses a shadow folder for computing the changed files. Direct
comparison over FTP would be slow & weak because file dates are not
preserved during FTP transfer, ASCII
transmission alters the file, file names may be mangled etc ...
(See TFS option in order to avoid consuming extra disk space).
If you do not already have a shadow folder, simply create an empty one like C:/www-sync. You can also copy the current source folder contents to the shadow folder in order to start incremental FTP from current state. All file which are not present in the shadow folder or which are changed in comparison to the shadow folder are transfered to the FTP remote tree and after successful transfer synchronized to the shadow folder.
Thus, the shadow folder stands as an exact copy of the current FTP remote tree. ( As a side effect this shadow folder is also very useful as kind of backup copy of the current remote tree, while editing the "edit tree". )
Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. Thus this encrypted password cannot be used or decrypted on an other PC.
TFS: Consider also using the "TFS:" file system (TimeStampOnlyVFS) for use with the shadow folder. Thus the shadow files on TFS virtual file system have zero-length, do not consume disk space and carry only time stamp information. Example:
roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }
FTP File System
FTPS File System
XFTP File System
TFS File System
XFTP is is an eXtended FTP file system, which stores the original file modification time stamp for a file like
ftp:<dirname>/<filename>
in an auxiliary file
ftp:<dirname>/XVS/.time.<filename>
(see also XVS File system).
For other options of this Wizard see FTP Upload Wizard
The extra time stamp information is written
as GMT flat time value. Thus the XFTP file system is suitable for full-featured bi-directional
synchronization throughout all time-zones.
XFTPS is to XFTP as FTPS to FTP : it uses SSL/TLS as transport layer.
Use this wizard to set up a true bi-directional synchronization. E.g. between a Laptop and a PC folder.
Enter the 2 root folders and follow the wizard. An especially important selection has to be done on the 3rd wizard page: Detect collisions & synchronize removals. See the extra help of that wizard page.
Note: if you check the "target only" option for a root, that root is never a source (master) folder but only a target (backup) location.
WebDAV is a modern full featured bi-directional WAN file system, often considered as more reliable replacement for FTP. This file system is supported by current web servers like Apache with mod_dav and other DAV servers. DAV over HTTPS is excellent for secure encrypted WAN file exchange.
Password Encryption: When checking this option, the password is not stored in plain manner in the project script, but encrypted. The encryption is done using a finger-print of the current PC. This encrypted password cannot be used or decrypted on an other PC.
DAV File System
XFTP File System
Filename wildcards ( * ? ) can be used anywhere in exclude & include masks. The * wildcard matches also across directories.
Annotations:
Relative path excludes & includes match in any directory.
Relative paths are usually used like *.tmp;*.bak;special.file (Files) or
CVS;resv (Directories).
For relative paths you can use also patterns mixed with relative subdirectories like ( e.g.
relative file exclude pattern "resv/*.dat"
matches "C:/texts/sub/sub/sub/resv/myfile.dat" when
"C:/texts" is a root)
'Rooted' (absolute(1)) paths match starting at the root
folder.
Example: rooted exclude file pattern "data/sub/my*.txt" matches
on C:/texts/data/sub/mytext.txt
when 'C:/texts' is root, but does not match C:/texts/sub/sub/sub/data/sub/mytext.txt )
Annotation on Directory Exclusion & Inclusion:
Currently excluded directories can only be revoked by inclusion on the
same tree level ! Inclusions at deeper levels have to be bridged - e.g. like..
excludedirs_abs = ["*"]
includedirs_abs = ["sub","sub/data","sub/data/*"].
As intended, this example wouuld not re-include a directory "sub/otherdata"
because it matches exclude through "*" and does not match a
re-inclusion pattern.
A revoking inclusion just like ...
includedirs_abs = ["sub/data"].
.. is not sufficient to work as intended.
[ In the future this behavior may be changed for trivial tail-only-wildcard
patterns in order to detect "include bridges" automatically. ]
Footnotes:
"rooted/absolute": absolute in relation to the root directory (not "absolute" in the sense of absolute file system paths)
(Script: syncremove=1)
When checking this option, file/directory removals (remove actions) are also synchronized.
The trigger for a removals is : File/directory was there during last synchronization or backup; File/directory is missing now => Remove action. Thus it is a kind of "Edge-Trigger". Thus removals can only be detected after an initial first sync run, which creates the sync database as a kind of starting point !
Detecting file removals requires the synchronzation database. Thus syncdb=1 (and detecting collisions) is implicit with syncremove=1. Thus it is not necessary, but optional to set syncdb=1 explicitly.
(Script: syncdb=1)
When checking this option, the status of files during a synchronization run is remembered in a 'synchronization database' file. This enables the detection of file synchronization collisions upon concurrent file changes. It also enables the detection of removed files (see syncremove=1 above).
An example of a file change collision: You have roots C:/texts and //laptop/texts. After an (initial) successful synchronization the file C:/texts/myfile.txt and //laptop/texts/myfile.txt are both edited : a concurrent file change. Upon next synchronization run it is unclear which file should be the new master file and only the human can decide. Thus when enabled syncdb=1 option, SynchronEX reports a file collision and does not copy this file at all.
This database file has by default the name <syncpy-project-filename>.syncdb and sits by default in the same directory as the .syncpy project file itself. E.g. When you have a project file C:/texts/laptop.syncpy and syncdb=1 option, then a synchronization database file C:/texts/laptop.syncpy.syncdb is created and tracked. You can alter the location of the sync database file by force when setting e.g. ..
syncdbpath="c:/outside/mysync.syncdb"
.. in the project script (See scripting documentation).
The default behavior is to stop the synchronization/backup run whenever any file cannot be copied/overwritten as intended. The 2nd option allows to continue the sync operation when an access error occurs for single files.
A 3rd option allows to continue the overall project script operation (e.g. multiple syncs and sub syncs ), when the a partial sync run cannot be completed (e.g. when a root/drive/network cannot be accessed ). This option is usually used in Super-Sync project files or in more complex scripted projects.
Only files younger as the date/time specified are copied in synchronizations/backups.
A simple list of copied (and removed) files is written to this log file. The default behavior is to overwrite this log file upon each sync/simulation run.
Note: This kind of log-file can be very useful for post-operations upon synchronized files it in scripts. E.g. after an synctree.do() do a Pythonic loop like: for file in open("_syncsrc.lst"): print file;
The output shown in the console window is also logged to this "full log file". The default behaviour is to append the log contents to the file upon each sync/simulation run. Thus the log file maybe reset manually by deleting it.
The full log file the the file best for tracking synchronizations and backups exactly e.g. for tracing back problems later. This is especially recommended for unsupervised automatic runs.
Note: When setting up a scheduled sync/backup task with the Scheduler Wizard, the log file is usually set up from inside the Scheduler wizard ( and passed to the project by the -L command line option ).
Additional sync/copy information is displayed in the console (and full log file). During FTP uploads also a detailed (debugging level) log of all FTP commands is displayed.
By default this option is checked: SynchronEX wants the .syncpy project file to be located in one the sync / backup root folders. This behavior helps to avoid copying project files unintentionally to strange folders. Uncheck this option if you want to store the project file outside of a root folder.
XVS enables to preserve and track historic versions of your valuable files. This provides the most save sync and backup mode you can think of.
Example: An overwrite of D:/backup/texts/hello.txt first copies D:/backup/texts/XVS/001.hello.txt , 002.hello.txt , etc... as kind of "in-place" backup files.
Thus you can look up and roll back the files you changed to any prior state (manifest during a synchronization run) !
When checking this XVS option checkbox in the options dialog, the xvs: file system tag is attached to all sync roots. In case you want XVS only for one of the roots (e.g. for a backup target), then do not check this checkbox, but prepend 'xvs:' to that root folder (1st wizard page). E.g. have 2 roots like "C:/texts" and "xvs:D:/backup/texts" .
For more XVS documentation see XVS.
This chapter presents projects at project script file level (& command line examples) . Consider using a wizard in the GUI before reading here.
2.1 Synchronize your C:/texts and all sub-directories on Laptop and PC
# c:/texts/config.syncpy |
Double-click config.syncpy for simulation. If everything
seems to be ok, right-click config.syncpy for action "Execute".
or run from command line:
C:/texts> synctree -s (simulation)
and then
C:/texts> synctree (real execution)
The configuration file config.syncpy it taken by
default, otherwise spec like synctree mysync.syncpy.
2.2 Simple command-line-update-sync from . to ./backup without
project file
synctree -z -r backup
| |
| have . (default) and ./backup as sync roots
|
no project file
This is in fact a "nested" sync, because "./backup" is a
subdirectory of "."
Of course nested sync roots are handled correctly in order to prevent from (endless) recursive copying.
2.3 Manual incremental (or initial) backup to ZIP archive
#myzipbackup.syncpy |
roots_no_src = [1] means : the 2nd root (zip:c:/data/backup.zip)
is target only / not a source. This defines a one-way backup.
The zip file is create on the fly if it doesn't exist. If it already exists, it
is an incremental backup.
Zip file locations maybe use the same way as ordinary directories. Even intra
zip directories like
zip:c:/data/backup.zip//subdir are possible (note the double slash after .zip ! ).
The same example as manual call:
synctree -z -r zip:c:/data/backup.zip
| |
| have . (default) and zip-file as sync roots
|
no project file
and including the one-way backup restriction:
synctree -z -r zip:c:/data/backup.zip -n1
-n1 : sync root backup is not a source!
2.4 3-point synchronization in a software testing project
#config.syncpy >>> |
syncdb=1 allows save multi-point authoring. Concurrent changes of files are
detected.
2.5 Super-sync binding multiple sub-syncs
#C:/config.syncpy (the
super-sync) >>> |
Annotation: If you provide additional parameters in this
super-sync-file, the sub-syncs inherit those parameters. (Command line
options are effective for all sub syncs too.)
2.6 Grabbing new photos from a digital camera
#D:/Photos/incoming/config.syncpy |
2.7 2-timer sync with different parameters
#C:/home/texts/config.syncpy >>> |
2.8 Synchronize non-recursive (no sub directories)
#C:/home/texts/config.syncpy >>> |
Command line variant: C:/home/texts> synctree -z -r
//laptop/c/home/texts -x *
2.9 Hard sync for 2 equally ranked
repositories (copy + remove)
#C:/home/texts/config.syncpy >>> |
syncremove is an advanced option, which implicitly uses syncdb=1. If you have a file remove action on one instance (except roots_no_src members) the remove action is detected through syncdb and is also "synchronized" to the other non-roots_no_src instances. Having syncremove=1, it is strongly recommended watching a simulation run (-> remove count) before really executing. Or use syncremove option "-+" in explicit cases.
2.10 Working in-sync on two machines with a mobile (ZIP) disk in between
Also this hopping sync case without direct network connection is no problem with
SynchronEX.
It is consistent, valid and effective practice, including collision detection on
concurrent changes, when syncdb=1 or syncremove=1.
HOME-Computer <--->
Mobile (ZIP) Disk <--->
WORK-Computer
For example on your home computer you could have this:
# c:/texts/config.syncpy |
... and on your work computer you have this
# d:/texts/config.syncpy |
You could also place both project files central on the ZIP-disk for example as
home.syncpy & work.syncpy. With syncdb=1 or syncremove=1 provided, you could
also carry a piece of a server 'away' on disk, work on it and re-sync with valid
consistent collision
detection!
2.11 Find duplicate files from current directory or in
'roots'
C:/texts> synctree --find_dup=fname,content --more20
C:/texts> synctree --find_dup=content >
duplist.txt
fname : filenames must be equal
content : the content must match exactly
--more20: stop display after one page
> duplist.txt : writes the results into a text file for comfortable analysis
Starting recursively from the current dir (c:/texts), this lists the duplicate
files that are equal in filename and content, but may differ in time stamp (time
criteria missing). The second example writes the duplicates into a text file.
Also in the second example comparison is solely based on the file contents. This
finds duplicates equal in content, but maybe differing in filename! Read more.
For checking duplicate files in a .syncpy configuration, right-click 'Find Duplicates' on that project file according to the installation.
2.11b Find empty directories starting from current directory or in
'roots'
C:/texts> synctree --find_emptydirs --more10
C:/texts> synctree --find_emptydirs > emptylist.txt
fname : filenames must be equal
content : the content must match exactly
--more0 : enable keyboard key 's' to stop &
page display
> emptylist.txt : writes the results into a text
file for comfortable analysis
.. or right-click 'Find Empty Dirs' on a .syncpy project file according to the installation.
2.12 Rotating incremental backup every week to another zip archive
#c:/texts/cycbackup.syncpy |
Rotating backup to C:/backup/work00.zip ..
work03.zip ... stepping every week. It is not necessary to initialize the
zip files beforehand, they are created on-the-fly if not already
existing.
Idea: This script could be evolved to a hierarchical multi-period backup
rotating for days, weeks, months ( shutil.copy2(
bkdir , "work_month%02d_backup.zip" % iMonth) ...)
Zip files can be used in same style as ordinary directories. Also intra zip
directories like
zip:c:/data/backup.zip//subdir are valid backup roots (note the double
slash after backup.zip ! )
2.13 Incremental FTP upload (of a web site)
#C:/www-edit/ftpupload.syncpy |
This project uploads (incrementally) your changed web files to your ftp-account in a rush. In this example the website tree is edited in C:/www-edit. The shadow copy is hold/maintained in C:/www-sync. Removed files are also removed on the ftp server.
Password handling: If you don't want to specify the user:pwd as shown you may also set only the user like in 'ftp://user@myserver.yyy/subdir'. Thus, the password is requested interactively. (If you have your password in the project file, don't forget to add '*.syncpy' to the excludefiles_rel list in order to not upload your password.) If your username or password contains the '@' character (example: myname@xdomain): quote it like myname\@xdomain. This is necessary, because unquoted '@' delimits user:pwd from the ftp server name. The GUI wizard takes care of this issue.
Each file is first transferred over FTP, then it is copied to the shadow dir. Thus, on ftp failure or broken transmission, the file will be re-transmitted on the next run.
ftp_ascii=ftp_executable = ['*.php','*.pl','*.py','*.cgi'] ensures that your CGI-scripts are uploaded in ASCII mode and that their executable flag is set on the server. You could also narrow down such declaration to special directories like ftp_ascii=ftp_executable = ['cgi-bin/*.pl','*.cgi']
This FTP mechanism can also be used for remote ftp data backup on the
intranet/internet. But consider also WEBDAV for that purpose.
See also the Auto-Periodic FTP example here.
Annotation: ftp_shadow is an alias for shadow_copy (new in v3).
2.14 Script example: Recycle backup of files for remove
#C:/music/mp3.syncpy |
During simulation mode this sync backs up files for remove to C:/backupmusic. On exec run the hard sync is executed normally. Of course one could program arbitrary execution behavior, for example a joint recycle backup + sync job in one script by calling .do() twice and setting appropriate simulation flag.
2.14b Script example: Moving a source tree to target
roots = ['src_folder','dst_folder'] |
After successful copy operation, the source files are removed.
2.15 Auto-Periodic FTP synchronization
There are many possibilities to execute a SynchronEX script periodically by an external scheduler (Windows Task Schedule (Wizard in SynchronEX), crond for Windows, ...). A call like this should be scheduled:
c:\bin\synctree.exe c:\www-edit\ftpupload.syncpy >> c:\ftpupload.log
However you can also script a periodic sync inside a project file itself like this:
#C:/www-edit/periodicftp.syncpy |
2.16 Synchronization to Windows mounted WebDAV (DAV; Webfolders like http://server/dav ) shares
This example synchronizes to a WEBDAV folder. WEBDAV folders are useful for WEB uploads as an alternative solution for FTP.
WEBDAV folders can also be used as WAN synchronization point for sharing files
with other users.
This script mounts the WEBDAV folder to the drive letter Z:
# C:/www-edit/davsync.syncpy |
For using DAV as WAN synchronization location consider removesync=1 and commenting out the line roots_no_src... like ##roots_no_src=...
Annotations:
2.17 Sending a notification email after job
Sending a notification email after the job is done:
Login and password can be omitted. Or the password can be loaded from a secured/secret file like: password=open("c:/secure/syncpwd.txt").read()
The mail text body could be rendered from the configured logfile like:
text="Log:"+open(logfile).read()
2.18 Bidirectional X-FTP Synchronization with Read/Write-Lock
In this example a stale lock has to be erased manually. (Automatic lock timeout to be done)
2.19 Catching an Error and Sending an Email
roots = ['C:/www-edit', 'C:/www-sync'] |
2.20 Searching the Log File for Errors and Sending an Email
synctree.do() |
For examples of syntax application see Examples.
Usage:
synctree.exe [OPTIONs] [PRJSCRIPTFILE]
PRJSCRIPTFILE:
The ".syncpy" configuration script file. Default: config.syncpy
OPTIONs:
-s force simulation run; files are not really copied
-y use a sync database (.syncdb) and detect Y-collisions
-z dont use config file (OPTIONSs only mode)
-n0,2 1st and 3rd root directory are not sources
-r /foo directory "/foo" is added to the roots list
-e foo.c "foo.c" is added to the excludefiles_rel
-i foo.obj "foo.obj" is added to the includefiles_rel
-x tmp "tmp" is added to the excludedirs_rel
-k foo.tmp "foo.tmp" is killed everywhere
-d 1.2.02 set threshold date
-t 13:20 set threshold time
-v verbose list (incl. filemtime, size)
-w / -W don't wait for ENTER at the end (-W : wait if error)
-l src.log create log list of all source files
-L sync.log full log of console output to file (append mode)
-+ sync file removals also (syncremove=1)
-c read project file from console/stdin
-p don't prompt for interactive questions, just exit with error
-h small help
--help big help
--remove src.lst :remove files in the list! (create with '-l -s')
--more stop display after every 10 lines (--more0/20/40/60) --remove=src.lst remove files listed in src.lst! (create with '-l -s') --find_dup=fname,size,time,content,exact find duplicate files based on selected criteria --find_dups equal to --find_dup=fname,size --find_emptydirs find empty directories in roots or current working directory --regkey enter license key directly --setup bind file classes --template open a project template for edit --help big help --gui start the SynchronEX GUI --guirun run project in GUI
More annotations
on command line options.
#C:/src/mysrc/test.syncpy |
(See also the file "template.syncpy" in your SynchronEX program folder or open the "General Template Wizard" in the SynchronEX GUI)
roots = ['C:\\www-edit\\', 'Z:/backup/www-edit/'] |
roots = [ "C:/web", "C:/web-sync" ]
2 or more roots folders for synchronization/backup.
Wide unicode root folder names (on unicode file systems like NTFS) can be coded like this:
roots = ['.', u'wide-unicode/dir-\u043a\u043a\u043a/] # /dir-ккк
( u in front of string-delimiter and '\uXXXX' 16-bit HEX character code )
syncdb=1 / commandline: -y
A synchronization database "<project-file>.syncdb" is tracked. This enables high
quality synchronization with detection of Y-collisions. A collision is
detected, when 2 or more instances of a file have changed since
last synchronization run.
syncremove=1 / commandline: -+
This option defines a hard synchronization: positive and negative. If you remove a file
somewhere (except roots_no_src members), the remove action is also synchronized
to the other instances. Usage of a sync database (syncdb=1) is defined implicitely
by this
option.
Defines one-way copying (backup mode). roots_no_src is a list of root indices - starting from 0 - for roots not allowed as sources (master folders). Those roots are 'target-only'.
Example: you have
roots = ['d:/backup0','c:/mydir']
.. then
roots_no_src = [0]
defines that the d:/backup0 directory is not a source, only c:/mydir is a source.
excludefiles_rel / excludefiles_abs
/ commandline: -e
excludedirs_rel / excludedirs_abs
/ commandline: -x
includefiles_rel / includefiles_abs
/ commandline: -i
includedirs_rel / includedirs_abs
List of file/directory patterns for exclusion & inclusion. see also Wizard Page Exclude & Include.
See also annotation on wildcards.
cb_is_exclude / cb_is_exclude_dir
Extra pythonic callback function for checking exclude/include of files or
directories. If this callback is a valid non-None function, this callback is
called for each file / directory. The callback function must take the rooted
path as 1st and only argument. The function must return one of these values:
0 or False : don't force exclude but check other
exclude masks
1 or True : exclude ! (don't check other masks)
2 : include ! (don't
check other masks)
Example: Copy only files smaller than 100000 bytes) :
def cb_is_exclude(rpath):
return 100000 < os.path.getsize(os.path.join(root[0],rpath))
def cb_is_exclude_dir(rpath):
return 'ZZZ' in rpath # ZZZ string in dirname?
thres_time='Dec-31-03/14:00' / commandline: -t -d
If this threshold time parameter is set, no files before that time are copied. The thres_time value may eigther US or European string format like "31.01.2003/15:00" or a flat 'seconds since 1970' (c-time/time.time()) value.
An example for a dynamic threshold time (copy only files of the last 24 hours):
thres_time = time.time() - 24*60*60
sub_syncs = ['other.syncpy','d:/data/mirror.syncpy']
Defines a super project calling other projects. If you don't
provide 'roots=' in the super-project,
the super-project doesn't copy anything itself. Anyway if you set additional
parameters in the super-project, those parameters are inherited by the
sub-projects (including the simulation flag). But in most practical cases a
super-project file contains only a single "sub_sync="
line.
logfile="my.synclog"
logfile="my.synclog","w"
Configures a log file. All text which appears in the console output also goes to this log file By default new content is appended to an already existing log file (open mode "a"). The second example (open mode "w") overwrites the existing log file.
shadow_copy={ 1 : 'd:/backup/diff-dir' }
shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }
shadow_copy copies/removes files which are written/removed at the attached root. The integer-key [1 in the example] defines the index of the attached root.
Example:
roots = ['c:/www-edit', c:/www-shadow]
shadow_copy={ 1 : 'ftp://user:pwd@myserver.xxx:25/subdir' }
Thus changed files, which are synchronized from C:/www-edit (root #0) to the
C:/www-shadow (root #1) folder, are also copied to (or removed at) the ftp
server (attached to root #1 by shadow_copy=... ).
Annotations on FTP-URL Format:
The FTP-URL format including username (user) and optional password (pwd) and optional port number (25) is the same as used in internet browsers. If the password (pwd) is omitted, then it is requested interactively.
If your username or password contains the '@' character (example: myname@xdomain),
then it has to be quoted by backslash like myname\@xdomain. This is
necessary, because unquoted '@' delimites
the "user:pwd"
from the ftp server name.
FTP Passive Mode (PASV):
Some old ftp servers do not support passive ftp mode (PASV). In that case an error like "command not supported" is yielded. Also some network routers may cut off passive ftp transfers ("connection reset by peer", ...). You may watch FTP debug output and/or switch to non-passive mode by inserting:
#C:\www-edit\ftp-nonpassive.syncpy
verbose = 1
ftp_passive = 0
roots=...
...
ftp_ascii = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]
Files matched by this mask are schedule for ASCII FTP transfer. ASCII transfer is usually required by CGI scripts.
ftp_executable = [ '*.cgi', '*.pl', 'cgi-bin/*.py', ... ]
Files matched by this mask are set as executable on the FTP server by "SITE CHMOD 755 <fname>" command. In case your FTP server (like Microsoft IIS FTP Server) doesn't support unix executable flags or SITE CHMOD and errors are reported when CHMODing masked files, set ftp_executable = []
ftp_execumask=755
Sets the "umask" file attributes, which are set by the FTP SITE CHMOD command for files matching the "ftp_executable" mask. Default: 755
lazy_mkdir=0
When lazy_mkdir is set to 1, the directories are only created during synchronization, when files are created. Thus empty directories, would not be re-created on the other side.
errorcontinue=0
Continue operation upon errors? :
Python Scripting / synctree.do()
You may add any python code to .syncpy project file. The python modules
os,shutil,fnmatch,glob,string,re,time and more are available (and imported).
If you do not have at least one call to synctree.do() or synctree.dontdo(),
a synctree.do() call is done automatically after parsing the project script.
Annotation on filename wildcards in exclude & include masks:
The mask filters match using filename wildcards ?, * and [seq].
seq:
character ranges, e.g. [A-Za-z] .
A more complex example: excludefiles_abs = ['data/my*/test[A-Z]/db??store*.dat'].
Real filenames cannot contain ? or * but [ or ]. if you want to match an
explicit file/dirname like synctest[4] you have to quote the brackets like
excludefiles_abs = ['data/synctest[[]4[]]/*'].
In roots and sub_syncs you must not quote,
because these are direct paths, not patterns.
xfile.X_remove_recursive
Removes files and folders recursively! By default a confirmation dialog protects
this operation.
Example:
xfile.X_remove_recursive("/my/path/to/folder", removethis=0, confirm=1)
Annotations on Command Line Parameters:
--more/--more0/--more20/--more40/--more60
Enables the integrated 'more' console display filter. Thus console output stops after 10,20,40 or 60 lines. --more0 switches on a more-filter, which is initially unlocked ('f' key), but is the same as --more10 + pressing key 's'/SPACE.
The more-filter reacts to the following keyboard keys:
--find_dup=fname,size,time,content,exact
Finds duplicate files based on the selected criteria (exiting
option).
| fname | the filenames must be equal |
| size | the file sizes must be equal |
| time | the file time stamps must be equal |
| content | the file contents must be equal (implies 'size') |
| exact | all criteria above must match |
See also the graphical duplicate file finder in the GUI !
Finds duplicate files. same as --find_dup=fname,size. This is a exiting option not starting a sync.
Finds empty directories in roots or '.'
This is a exiting option not starting a synchronization.
This chapter presents a list and short description of current built-in file systems. ( SynchronEX' Extensible Virtual File System Technology enables also custom made file systems. )
roots = [ 'c:/www', 'd:/web' ]
roots = [ 'c:/www', '//server/web' ]
SynchronEX synchronizes seamless into zip archives or even into intra-ZIP paths
roots = [ 'c:/www', 'zip:d:/backup/web.zip' ]
roots = [ 'c:/www', 'zip:d:/backup/web.zip//sub/subdir' ]
roots_no_src = [1]
See also XVS Overview, Recycle (RCY) File System.
XVS is a sort of journaling file system on top of the Local File System, which keeps past versions of files
by automatically backing them up to a XVS folder ( ./XVS or ./.XVS
relative to the file folder), whenever a file is to be overwritten or
(re)moved by SynchronEX. The XVS folder is created automatically on
demand.
An XVS backup file like is created like : /sub/folder/XVS/005.normal-file-name.txt .
The index number (005 in this example) is counted upwards for each archived version of a file. An XVS backup file is created, when the original file is about to be (1) replaced by a copy-operation, (2) written/overwritten, (3) removed/moved.
The default XVS backup folder "XVS" can be changed via config.SetXVSFolder. Use XVS folder names other '.XVS' or 'XVS' only,
when there is a good reason for. (These 2 standard XVS folder names are
on internal folder exclusion in the style of "excludedirs_rel" by
default.)
roots = [ '/work/www', 'xvs:/dist/www' ]
config.SetXVSFolder('.XVS') # changes default 'XVS' or '.XVS'
roots = [ 'c:/www', 'xvs://server/web' ] # XVS on a UNC share
roots = [ 'c:/www', 'uxvs:sftp://www@myserver.dom/htdocs' ]
See also XVS
File System.
On Windows: Archives past versions of file which are about to be overwritten or (re)moved by automatically
moving them to the Windows Recycle-Bin folder ("Trash basket").
On UNIX: Archives past versions of overwritten or (re)moved files on local file systems by automatically
moving them to a "./.recycle" folder. Same operation as with the ./XVS or ./.XVS folders of the XVS
File System, just uses a differently named folder (which is intended to be cleaned less considerate than .XVS).
roots = [ 'c:/www', 'rcy:d:/www-sync' ]
TimeStampOnlyVFS upon Local File System. Usually only used for target-only roots. Files on TFS are created with zero-length (no content), but carry time stamp information. E.g. Consider using the TFS file system for use with FTP/WEBDAV or other shadow folders. Thus the shadow files on TFS have zero-length, do not consume disk space and carry only time stamp information.
roots = ["C:/www", "tfs:C:/www-shadow"]
roots_no_src = [1]
shadow_copy={ 1:"ftp://...." }
Dummy sink file system similar to "/dev/null" on UNIX, but it is a sink for folder trees.
#example: MO/Zip/Floppy drive read test
roots = [ 'z:/data', 'null:' ]#example: full non-incremental copy / mirroring
roots = [ 'c:/www', 'null:' ]
shadow_copy={ 1: '//remote/non-incremental-mirror-dir' }
#example: force FTP full non-incremental copy
##roots = [ 'c:/www-edit', 'c:/www-shadow' ]
roots = [ 'c:/www-edit', 'null:' ]
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }
The plain FTP File system. Suitable mainly for using it with shadow_copy / ftp_shadow. Files on bare FTP files systems cannot be compared directly, because FTP is a weak directory protocol for synchronization: File time stamps cannot be preserved during copy and the files itself are (partially) exposed to ASCII conversion etc.
roots = [ 'c:/www', 'c:/www-shadow' ]
roots_no_src = [1]
shadow_copy={ 1: 'ftp://me@webserver.com/www' } # password asked interactively
shadow_copy={ 1: 'ftp://me:pwd@webserver.com/www' }
config.setFTPConnTimeout(300.0) # change timeout to 300 seconds
(Default is 120 seconds)
See also: XFTP filesystem for direct bi-directional transfer compatible with plain FTP.
FTPS is a FTP File system using secure SSL/TLS as transport layer. FTPS over SSL/TLS is supported by most modern FTP servers. If it doesn't work with your FTP server you will get an error like "550 command not understood" after the FTP "TLS" command (set "verbose=1").
roots = [ 'c:/www', 'c:/www-shadow' ]
roots_no_src = [1]
verbose=1 # verbose display of FTP commands
shadow_copy={ 1: 'ftps://me@webserver.com/www' } # password asked interactively
shadow_copy={ 1: 'ftps://me:pwd@webserver.com/www' }
See also: XFTPS file system
For special configuration see also FTP file system documentation.
This is an eXtended FTP File system, which stores extra original file modification time stamp
information in ftp:<dirname>/XVS/.time.<filename>
(see XVS File system). The extra time stamp information is written
as GMT flat time value. Thus the XFTP file system is suitable for full-featured bi-directional
synchronization throughout all time-zones. XFTPS is to XFTP as FTPS to FTP :
uses SSL/TLS as transport layer.
roots = [ 'C:/www-edit', 'xftp://me@myserver/root' ]
excludedirs_rel = ['.ftplock']
ftp_ascii = []
flock=roots[1]+'/.ftplock'
if not simulation: xfile.X_mkdir( flock )
try:synctree.do()
finally:
if not simulation: xfile.X_rmdir( flock )
It is not recommended to use XFTP for normal incremental FTP uploads, because direct file comparison over (X)FTP is rather slow. Thus if you have many upload files and a small number of download files, it is recommended to split the job into 2 parts and use a multi-project.
For special configuration see also FTP file system documentation.
SFTP is the secure encrypted SSH(2) File Transfer Protocol. It is a modern, very reliable, secure and fast WAN file transfer protocol. Despite its name, SFTP is technically very different from FTP / FTPS and uses different software.
Most UNIX Servers offer SSH/SFTP access through the "sshd" daemon. Yet SFTP support often has to be enabled extra in the configuration for sshd in a file like /etc/ssh/sshd_config - by a line like:
#sshd config file /etc/ssh/sshd_config
...
Subsystem sftp /usr/lib/sftp-server
Logon can happen by password or by RSA/DSA private key (with optional key password). Required passwords are asked interactively if the are not set inside the script - and can interactively entered passwords can optionally be saved machine-encrypted after confirmation.
# wizard_SFTP_Sync
# open password logon (not recommended unless "sycntree -c" from other software)
roots = ['C:/www-edit', 'sftp://myself:directpassword@sftpserver.dom/subdir_from_myself']
# wizard_SFTP_Sync
# password will be asked interactively and optionally be stored machine-encrypted after confirmation
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom/subdir_from_myself']
# how to point to a absolute server folder (relative from the root folder / ):
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom//subdir_from_root']
# override the the default '~/.ssh/known_hosts' file
config.setSFTPKnownHostsFile('~/sshsx/known_hosts.txt')
# trigger private key logon (per server)
config.setSFTPPrivateKey('sftpserver.dom', '~/sshsx/private_RSA_key_sftpserver.txt')
config.setSFTPPrivateKey('otherserver.dom', '~/sshsx/private_DSA_key_otherserver.txt', 'funny_pwd')
# change connection timeout to 120 seconds (default: 60)
config.setSFTPConnTimeout(120.0)
# suppress interactive questions for keys, known hosts etc. / just raise errors
config.setInteractive(0) # like -p from commandline
roots = ['C:/www-edit', 'sftp://myself@sftpserver.dom:2277/subdir_from_myself']
The following default locations are searched for private SSH keys: ~/.ssh/id_rsa , ~/.ssh/id_dsa
Otherwise you can set a path to a private key (per server) explicitely by this function:
config.setSFTPPrivateKey(host, priv_file_or_key, password=None)
Note: in case of an encrypted private key, the key password can also be entered interactively - and optionally be stored machine-encrypted by confirmation for later auto-use.
Note: the priv_file_or_key can be an immediate RSA/DSA key string as well (instead of a path to a file).
A Known Hosts File different from default '~/.ssh/known_hosts' can be set like:
config.setSFTPKnownHostsFile(know_hosts_file)
Change the connection timeout (Time after which a new connection will be used):
config.setSFTPConnTimeout(120.0)
Notes: ~ is %USERPROFILE% on Windows. Thus '~/.ssh/known_hosts' on Windows for example expands to %USERPROFILE%/.ssh/known_hosts . Check the latter location by copying the folder path into the Windows Explorer address navigation. Typically this ends up in a folder like "C:\Documents and Settings\user\.ssh"
This is the DAV over HTTP(S) file system - also called "WebDAV" or WebFolders". WebDAV is a modern full-featured bi-directional WAN file transfer protocol extending the one-way Hyper Text Transfer Protocol (HTTP). Thus DAV file/folder URLs have the same layout than normal HTTP-URLs - e.g.: 'https://myself@davserver.com/davroot' . DAV is often considered to be a more reliable replacement for FTP. DAV is supported by current web servers like Apache (with mod_dav installed) and other DAV servers. A DAV server like Apaches mod_dav usually has to be configured (".htaccess", "http_conf") in order to allow (password protected) DAV read/write/property-change access to (certain) folders. If the web server of your ISP enables DAV, you should have documentation from your ISP about how to access and use DAV. DAV over HTTPS is excellent for secure encrypted WAN file exchange.
#wizard_Native_DAV_Sync
roots = ['C:/www-edit', 'https://myself@www.davserver.dom/davroot']
Proxy support for DAV (HTTP & HTTPS):
Extra Transfer File:
DAV over HTTPS: x509 Client Keys & Certificates:
x509 Client Keys & Certificates by adding this to the project script:
config.setDAVKeyCertFiles('www.davserver.dom',
key_file='/path/keyfile.x509',cert_file='/path/cerfile.x509')
(the host name must be the same as used in your DAV URL)
Change the connection timeout (Time after which a new connection will be used)
config.setDAVConnTimeout(20.0) # change timeout to 20 seconds
(Default is 10 seconds)
A special DAV File system, which stores file time stamps in extra files and not in DAV-properties. Suitable for simple DAV-Servers which do not support the DAV-Method "PROPPATCH" (for example Tomcat). Similar to XFTP vs. FTP
roots = ['C:/www-edit', 'xhttps://myself@davserver/davroot']
For special configuration see also (normal) DAV file system documentation.
Compresses and decompresses files individually on the fly on any other file system.
The (created) compressed files on the target file system have an extra extension '.sx.gz' attached by default.
roots = ['C:/www-edit', 'igz:http://myself@davserver/www-gz-backup']
roots = ['C:/www-edit', 'igz:sftp://myself@server/gz-backup']
Change the default file extension for compressed files.
config.SetIGZExtension('.gz')
Encrypts and decrypts or batch-converts files individually on the fly by using any command line stream coder (pipe), which is installed on the system (e.g. GPG). Works on top of any other SynchronEX file system
The encrypted files on the target file system have an extra extension '.sx.enc' added by default.
config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
)
roots = ['/home/myself/www-edit', 'ienc:http://myself@davserver/www-encrypted-backup']config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
key='_mygpgpassphrase_'
)
roots = ['C:/original', 'ienc:D:/encrypted']config.SetIENCPipe('ienc:',
'gpg -e -r USER1 --batch --always-trust',
'gpg -d -r USER1 --batch --always-trust --passphrase-fd 0', '.sx.gpg',
keyfile='C:/mygpgpassphrase_is_in_here.txt'
)
roots = ['C:/www-edit', 'ienc:igz:sftp://myself@server/encrypted-gz-backup']
.
The SynchronEX software is about incremental multi-root file&object-tree synchronization, backup, updating backup, FTP, incremental ZIP archive update, file tree mirroring, reliable copying of file trees, scheduling projects, finding file duplicate files, finding empty directories, advanced browsing and finding of files & other file tree related work.
SynchronEX = robust engine + GUI frontend. The engine can work independently and silently. E.g. can be called from batch jobs, schedules or other software.
SynchronEX internally uses a sophisticated extensible virtual file system (VFS)
technology. Thus SynchronEX bridges seamlessly into FTP, WEBDAV, ZIP and other
widely used file system protocols. Other VFS add special capabilities to local
file systems. E.g.: XVS,LimitedMediaVFS,TFS,NULL .... : see File Systems
Synchronization is
based on file time stamps.
A Simulation mode enables evolving arbitrary complex projects without risk.
SynchronEX can use a synchronization database file when advanced comparisons
have to be done, which cannot be accomplished simply by direct file/directory
comparison: A Y-collision detection feature (-y /syncdb=1) provides protection against
dangerous overwrites of concurrent file changes. Hard synchronization
including the synchronization of file/directory removals also makes use of the
synchronization database.
Usage of
SynchronEX is very simple for common tasks (GUI Wizards). On the other
hand you are able to schedule arbitrary complex patterns, or even
evolve your
project script into a full blown Python automated program.
SynchronEX uses human readable project scripts which can be set up with wizards
and can be evolved to more complex scripts/programs. It is recommended to place
projects files into one of the root folders of interest.
Exclude & Include:
There exist 5 channels of exclude/include masks: For files OR directories, relative OR
rooted (absolute) + a logical callback function. SynchronEX' mask policy is
"Take all minus excludes but take includes.". Rooted (absolute) masks have priority over relative masks.
In addition a logical callback function, which is called for each file or
directory, can bias exclusion 3-state: force exclude, force include, delegate
decision to upper masks.
Complex python scripts can be directly executed inside the project script.
An arbitrary number of synch-jobs can be executed programmatically (e.g.
loops; synctree.do() calls ) in a project script. ( If no such call is done explicitly, a
synctree.do() is automatically scheduled after parsing the project script.
)
Copy directions in synchronizations are scheduled based on file time
stamps. Exact 1-hour distances with no change in file size, are
scheduled as "IN SYNC" because ( common incompatible daylight savings
handling on different
computers ).
SynchronEX takes special mean to defend against synchronization
inconsistencies upon cable interrupts/OS crash. E.g. it uses special time stamps
during copy and block OS interrupt signals.
Commandline paramaters take priority over project file settings.
XVS enables auto-back-up of former versions of files which are changed or deleted during a synchronization run or backup run. This is the most save mode at all for file synchronizations and backup tasks: You can not anymore loose files or loose old versions of your files unintentionally. XVS can be used together with syncremove=1 option ( hard synch mode ) in order to set up extremely effective synchronizations.
The principle of XVS is simple: If any file on an XVS tagged file system (e.g. root "xvs:C:/texts" ) is overwritten or removed during a synchronization run, the old file is first moved to a XVS backup location. E.g. file C:/texts/mytext.txt is backed up as C:/texts/XVS/000.mytext.txt , then on next overwrite to C:/texts/XVS/001.mytext.txt, then C:/texts/XVS/002.mytext.txt and so on. The ./XVS sub-folder is generated automatically on the fly when necessary.
You can tag any local or network root path in SynchonEX for XVS processing. ( This does not alter anything in your real file system of that disk drive. It is just a virtual file system inside SynchronEX itself. )
Thus you can prepend a "xvs:" tag to a root directory in order to declare it as XVS virtual file system. In the GUI you can also simply use the XVS checkbox or the dedicated XVS wizard which activates the XVS for all roots by default. Example:
roots = [ "xvs:C/texts", "xvs://server/texts" ]
This way you get historic backups for all versions of the files. At any time you can remove XVS backup files, e.g. when they consume a lot of disk space. The GUI tool in Menu/Tools/XVS Cleanup can be used to quickly bulk sweep XVS backup files.
The default maximum of backup versions of a file is 10000. If more versions are to be generated, an error is thrown. You can change this number e.g. by
config.SetXVSLimit(100)
.. in a script.
XVS system can be applied to local/UNC filesystems only. The UXVS file system ("Universal XVS") is the same as the SXV file system, but works on _any_ other filesystem. UXVS works a little slower generally:
roots = [ "uxvs:sftp://user@server/texts", "xvs://server/texts" ]
"What has XVS to do with CVS or MS SourceSafe?"
Answer: Not much. XVS works completely automatic without needing to know
anything about repositories, tags, check-in/check-out. Is is probably the
most easy-to-use method for version tracking at all. It works without any
problems in any office or development environment where valuable documents of any nature are edited iteratively.
But of course it is not a total replacement for CVS with controlled version
tracking (
multi-year, branches etc. )
However even in source code development environments there are situations where it is reasonable to combine XVS and a version tracking system like CVS. E.g. XVS could take care of fast iterations, personal iterations and automatic daily versioned backups while CVS/SourceSafe does the long-term manual controlled version tracking.
Another special application with CVS repositories: SynchronEX & XVS are reported to successfully and consistently keep a CVS repository itself mirrored on 2 or more locations for certain applications, where there is no permanent network connetions (see also "collision detection" for consistency ).
Forcing synchronization upon collisions (syncdb=1) ?
First execute the project, overview the collisions and do all necessary manual
merge work.
In
order to really force synchronization solely based upon file times, simply erase
the .syncdb database file of that project and re-execute the
project.
Finding files duplicate in content but differing in filename ?
Synchronizing two computers with a mobile (e.g. ZIP-Drive) Disk in between ?
Also this case is no problem with SynchronEX and very save, valid and effective with
syncdb=1
/ syncremove=1
HOME-Computer <--->
ZIP-Disk <---> WORK-Computer
For example on your home computer you have this:
# c:/texts/config.syncpy |
... and on your work computer you have this
# d:/texts/config.syncpy |
You could also place both project files central on the ZIP-disk for example as home.syncpy & work.syncpy. With syncdb=1 provided, you could also carry a pice of a server 'away' and work on it and re-sync with valid collision detection.
" Warning Message: project / working directory not in roots ; consider check_cwd=0 "
" I have lots of collisions (syncdb=1). How do I get rid of them? "
" How to log on to a Windows folder on a remote machine automatically (with password)"
#force auto-logon on remote windows folder:
os.system(r"net use \\COMPUTERNAME\RemoteFolder /u:user pwd")
roots = ['c:/myfolder', r'\\COMPUTERNAME\RemoteFolder']
....
#force auto-logon on remote windows folder with drive letter mapping:More on Windows command "net use".
os.system(r"net use Z: \\COMPUTERNAME\RemoteFolder /u:user pwd")
roots = ['c:/myfolder', 'Z:/']
....
" I want to call a SynchronEX project from C, Delphi, Script ... with console window hidden "
" How do I get a detailed log of FTP commands issued? "
" SynchronEX won't create directories on my FTP server "
" Is it possible to compare two directories with SynchronEX but copy the changed files to a third directory? "
" A backup to ZIP archive always appends? "
" I have problems with SynchronEX on local area networsk & Windows XP firewall (XP Pro SP2) "
Control Panel -> Network Connections -> Local Area Network ->properties
On the General Tab:
Internet Protocol (TCP/IP) -> Properties -> Advanced -> WINS -> NetBIOS setting is either set to Default or Enable NetBios over TCP/IP
File and Printer Sharing for Microsoft Networks must be checked
On the Advanced Tab:
Enable Microsoft Firewall by checking the checkbox.
Press Settings push button to configure advanced firewall settings:
On the Services tab:
Uncheck all checkboxes to disable all default services.
If your computer hosts an FTP site, or web server, etc. you’ll have to leave the appropriate check box checked.
Press the Add push button to add Port 137 UDP to firewall
Press the Add push button to add Port 138 UDP to firewall
Press the Add push button to add Port 139 TCP to firewall
On the ICMP tab
Uncheck all checkboxes to disable all ICMP message support. You can check Echo (ping) if you want PING to work.
SynchronEX main page.
Stars:
Xellsoft's HarvEX
: eBay Automation Software
Xellsoft
Custom Software
Python scripting in SynchronEX
webmaster@xellsoft.com
- © 2003-2012 Xellsoft.com. All Rights Reserved
- Online Privacy - Affiliate
- Company
