mirror - copy file trees from FTP server to client

mirror [-FGLNPTdfmnrtv] [-C<config file>] [-U<log file>] [-g<site:path>] [-k<variable>=<value>] [-p<pattern>] [-R<pattern>] [-s<site name>] [-u<username>] [<path>]

mirror is a perl script which uses the local FTP client to make exact copies of directories or parts thereof on remote FTP servers. With version 2.9 the source documentation was converted to HTML. Please use your favourite web browser to read this in the documentation tree e.g.,

$ lynx /usr/share/doc/mirror/html/mirror.html The HTML documentation is the authoritive source.

-Cconfig_file

Load alternative config file.

-d

Turn on debugging - more -d's means more debugging.

-f

Same as "-kforce=true".

-F

Use files for assoc arrays (see also the variable use_files).

-gsite:path

Get all matching files on given site. If path matches .*/.+ then it is the name of the directory and the last part is the pattern of filenames to get. If path matches .*/ then it is the name of a directory and all its contents are retrieved. Otherwise path is the pattern to be used in '/'. By default local files are overwritten by newer but not older remote files of the same name and are not deleted for missing remote files.

-G

Same as "-kget_file=true -kinteractive=true".

-kvar=val

set variable to value.

-L

Generate a pretty list of what is being mirrored.

-m

Same as "-kmode_copy=true".

-n

Do nothing, just show what would be done.

-N

Don't load mirror.defaults.

-ppattern

Just do packages matching pattern.

-P

Same as "-kget_file=false -kinteractive=true".

-r

Same as "-krecursive=false". Stops downloading of directory tree listings to speed mirroring of a few files in the same directory. Sorry, this is the opposite of traditional option behaviour.

-Rpattern

Skip till the first package name matches pattern then do all. it and following packages.

-sSITENAME

Same as "-ksite=SITENAME".

-t

Same as "-ktext_mode=true".

-T

Dont transfer just force local timestamps to match remote.

-uusername

Same as "-kremote_user=username", prompts for remote_password.

-ULOGFILE

Set the upload log to LOGFILE - if none given uses the file $home/upload_log.$mday.$mon.$year

-v

Print version and exit.

path

Path to package file either full or relative to default directory. Package files may reset many variables from defaults. See HTML documentation listed below for details.

0

Successful program execution. Applies even when mirror package or remote server files were missing or there were network errors. Result details are written to stdout.

>0

Missing or corrupt perl executables or mirror source files.

On a Debian system, configuration and program files of the mirror packages are found in the following directories:

/etc/mirror/
/usr/share/doc/mirror/
/usr/lib/mirror/

The -g option can keep a single file or directory up to date and requires no configuration files.

Use -r to avoid downloading a whole directory listing if you just want a few files in the same directory.

Example configuration and package files are in subdirectories of /etc/mirror and /usr/share/doc/mirror/examples.

If mirror behaves strangely use -d to see what is happening. The more you use the more detail so redirect the output to a file.

When mirror fails to work for you, the default value of a parameter described in mirror.html may need setting or resetting e.g. if your machine connects through a strict firewall, you may need passive FTP. Set the variable,

t passive_ftp=true

on a line in your package file for passive FTP.

For postprocessing of a mirror run you can set

t mail_prog=pathname

where pathname is a script mirror will execute after completing a package. Mirror sends the logfile to pathname's stdin which pathname may read and process. You can set

t mail_subject=" $keyword ..."

where $keyword will be replaced by its value for the package and pathname can read this as an argument. mail_to can be set to ordinary arguments. If you still want email from this mirror package pathname must send it. Try all this only when no simpler way of using mirror works.

Use cron to deal with network errors by daily or weekly repetition at a time when the network near both server and client is idle.

A bash script can run mirror commands sequentially but mirror-master can start them simultaneously and/or sequentially for best use of core and network idle time.

You should not mirror the same site more than once in 24 hours unless you use ls-lR files to improve tracking and minimize server load. Some FTP servers encourage the use of ls-lR files by disabling recursive directory listings. See mirror-lslR.html for instructions.

mirror downloads each file into a temporary file prefixed .in. and renames on completion. If you or anything interrupt and then restore the net link, wait for mirror to either restart downloading or time itself out in about ten minutes. When relinked and after a time out reissue the command and mirror restarts from the end of the .in. file.

To stop mirror press Control-Z. After a restart with fg wait for mirror to resume downloading or time out. A restart after such time out, begins with the end of the .in. file. If you reboot, log off, interrupt with ^C or kill with ^U mirror downloads the .in. file again on restart.

For safe system use create a normal user say ftpadm. Use chown -R ftpadm. on the directories /var/log/mirror/, /var/lib/mirror/, /etc/mirror/ and /home/ftp/pub/ to allow ftpadm to operate mirror and mirror-master. Add ftpadm to ftpd's etc/group and etc/passwd using ftpd documentation. Servers with more than one archivist may have an ftpadm group.

The author of the mirror script is Lee McLoughlin and the source of the documentation and scripts is:

http://sunsite.org.uk/packages/mirror/

The copyright statement can be found in the file

/usr/share/doc/mirror/copyright.

HTML documentation at /usr/share/doc/mirror/html/mirror.html. mirror-quick.html, mirror-lslR.html, chmod(1), cron(8), crontab(1), do_unlinks(1), mirror-master(1), pkgs_to_mmin(1), mkls-lR(1), ftp(1), ftpd(8).