debmirror - Debian partial mirror script, with ftp, http, hftp or rsync and package pool support

debmirror [options] mirrordir

This program downloads and maintains a partial local Debian mirror. It can mirror any combination of architectures, distributions, and sections. Files are transferred by ftp, and package pools are fully supported. It also does locking and updates trace files.

To support package pools, this program mirrors in three steps.

1. download Packages and Sources files

First it downloads all Packages and Sources files for the subset of Debian it was instructed to get.

2. clean up unknown files

Any files and directories on the local mirror that are not in the list are removed.

3. download everything else

The Packages and Sources files are scanned, to build up a list of all the files they refer to. A few other miscellaneous files are added to the list. Then the program makes sure that each file in the list is present on the local mirror and is up-to-date, using file size (and optionally md5sum) checks. Any necessary files are downloaded.

mirrordir

This required parameter specifies where the local mirror directory is. If the directory does not exist, it will be created. Be careful; telling this program that your home directory is the mirrordir is guaranteed to replace your home directory with a Debian mirror!

--debug

Enables verbose debug output, including ftp protocol dump.

--progress -p

Displays progress bars as files are downloaded.

--verbose -v

Displays progress between file downloads.

--source

Include source in the mirror (default).

--nosource

Do not include source.

--md5sums -m

Use md5sums to determine if files on the local mirror that are the correct size actually have the correct content. Not enabled by default, because it is too paranoid, and too slow.

--passive

Download in passive mode.

--host=remotehost -h

Specify the remote host to mirror from. Defaults to ftp.debian.org, you are strongly encouraged to find a closer mirror.

--user=remoteusername -u

Specify the remote user name to use to log to the remote host. Helpful when dealing with brain damaged proxy servers. Defaults to anonymous.

--method=ftp|hftp|http|rsync -e

Specify the method to download files. Currently, supported methods are ftp, hftp (ftp over http proxy), http or rsync. To connect a rsync server, you need to put ':' prefix in the root directory (i.e. :debian, which means host::debian).

--proxy=http://user:pass@url:port/

Specifies the http proxy (like Squid) to use for http and hftp method.

--timeout=seconds -t

Specifies the timeout to use for network operations (either FTP or rsync). Set this to a higher value if you experience failed downloads. Defaults to 300 seconds.

--root=directory -r directory

Specifies the directory on the remote host that is the root of the Debian archive. Defaults to /debian, which will work for most mirrors. The root directory has a dists/ subdirectory.

--dist=foo[,bar,..] -d foo

Specify the distribution (woody, sarge, sid) of Debian to mirror. This switch may be used multiple times, and multiple distributions may be specified at once, separated by commas. Using the links (stable, testing, unstable) does not have the expected results but you may add those links manually. Defaults to mirroring sid.

--section=foo[,bar,..] -s foo

Specify the section of Debian to mirror. Defaults to main,contrib,non-free,main/debian-installer.

--arch=foo[,bar,..] -a foo

Specify the architectures to mirror. The default is --arch=i386. Specifying --arch=none will mirror no archs.

--cleanup

Do clean up any unknown files and directories on the local mirror (see step 2 above). On by default.

--nocleanup

Do not clean up the local mirror after mirroring is complete.

--postcleanup

Clean up the local mirror but only after mirroring is complete and only if there was no error.

--ignore=regex

Never delete any files whose filenames match the regex. May be used multiple times.

--exclude=regex

Never download any files whose filenames match the regex. May be used multiple times.

--exclude-deb-section=regex

Never download any files whose Debian Section (games, doc, oldlibs, science, ...) match the regex. May be used multiple times.

--limit-priority=regex

Limit download to files whose Debian Priority (required, extra, optional, ...) match the regex. May be used multiple times.

--include=regex

Don't exclude any files whose filenames match the regex. May be used multiple times.

--skippackages

Don't re-download Packages and Sources files. Useful if you know they are up-to-date.

--adddir directory

Also download Packages and Sources files from the specified directory on the remote host (the directory is relative to the root of the Debian archive). If you used this option for --adddir dists/proposed-updates in the past use --dist <dist>-proposed-updates now. This feature is now obsolete and will be removed soon.

--getcontents

Download Contents.arch.gz files.

--max-batch=number

Download at most max-batch number of files (and ignore rest).

--rsync-batch=number

Download at most number of files with each rsync call and then loop.

--ignore-missing-release

Don't fail if the Release file is missing.

--ignore-release-gpg

Don't fail if the Release.gpg file is missing.

--dry-run

Simulate a mirror run. This will still download the meta files to .temp but won't replace the old meta files, won't download debs and source files and only simulates cleanup.

--rsync-options=options

Specify alternative rsync options to be used. Default options are -aIL --partial. Care must be taken when specifying alternative options not to disrupt operations, it's best to only add to those options. The most likely option to add is --bwlimit=x to avoid saturating the bandwidth of your link.

--ignore-small-errors

Normally debmirror will report an error if any deb files or sources fail to download and refuse to update the meta data to an inconsistent mirror. Normally this is a good things as it indicates something went wrong during download and should be retried. But sometimes the upstream mirror actually is broken. Specifying --ignore-small-errors causes debmirror to ignore missing or broken deb and source files but still be pedantic about checking meta files.

--help

Display a usage summary.

Mirror size for a singe arch and binary only (in MiB):

         | sarge | etch |  sid  |  all
---------+-------+------+-------+-------
main     | 8816  | 9126 | 10777 | 20577
contrib  |  126  |  118 |   291 |   363
non-free |  282  |  345 |   464 |   666
d-i      |   44  |   28 |    31 |    78
all      | 9187  | 9536 | 11476 | 21502

Mirror size per arch (in MiB):

         | sarge | etch |  sid  |  all
---------+-------+------+-------+-------
source   |  9339 | 9419 | 11495 | 17521
all      |  4478 | 5047 |  6160 | 10459
alpha    |  4256 | 3906 |  4732 |  9708
amd64    |  3644 | 3635 |  4877 |  9152
arm      |  3445 | 3193 |  3933 |  7845
hppa     |  4112 | 3713 |  4541 |  9167
i386     |  4422 | 3979 |  5005 | 10477
ia64     |  4709 | 4489 |  5316 | 11043
m68k     |  3372 | 3072 |  3664 |  7139
mips     |  3631 | 3364 |  4099 |  8237
mipsel   |  3560 | 3319 |  4049 |  8106
powerpc  |  4208 | 3967 |  4742 |  9915
s390     |  3673 | 3452 |  4144 |  8489
sparc    |  3761 | 3585 |  4390 |  8893

All numbers reflect the state of 2005 Dec 9th and do not include any filesystem overhead (which adds 1-2GB for ext2/3).

 debmirror /mirror/debian

Simply make a mirror in /mirror/debian, using all defaults.

 debmirror /mirror/debian --ignore=non-US/
 debmirror /mirror/debian/non-US -h non-us.debian.org -r /debian-non-US \
           -d sid/non-US -s main,contrib,non-free

Make one full mirror, and supplement it with a mirror of non-US, in a directory inside.

 debmirror -a i386,sparc -s main -h ftp.kernel.org \
           -d sid -d sarge /home/me/debian/mirror --nosource \
           --progress

Make a mirror of i386 and sparc binaries, main only, and include both unstable and testing versions of Debian. Download from ftp.kernel.org.

 debmirror -e rsync -r :debian /home/me/debian/mirror

Make a mirror using rsync. rsync server is ftp.debian.org::debian.

  ~/.gnupg/pubring.gpg

    Debmirror uses gpg to verify Release and Release.gpg using the
    default keying ~/.gnupg/pubring.gpg. This can be changed by
    exporting GNUPGHOME resulting in $GNUPGHOME/pubring.gpg being used.

    To add the right key to this keyring you can import it from the
    debian keyring (in case of the debian archive) using:

      gpg --keyring /usr/share/keyrings/debian-role-keys.gpg --export \
          "Debian Archive Automatic Signing Key (2005)" | gpg --import

    or download the key from a keyserver:

      gpg --keyserver keyring.debian.org --recv-keys <key ID>

    <key ID> for debian and 2005 is 4F368D5D and can be found in the
    gpg error message from debmirror -v output.

This program is copyright 2001 by Joey Hess <joeyh@debian.org>, under the terms of the GNU GPL, copyright 2003 by Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de>.

The author disclaims any responsibility for any mangling of your system, unexpected bandwidth usage bills, meltdown of the Debian mirror network, etc, that this script may cause. See NO WARRANTY section of GPL.

Current: Goswin von Brederlow <brederlo@informatik.uni-tuebingen.de> Previous authors: Joey Hess <joeyh@debian.org> Joerg Wendland <joergland@debian.org>

Waste bandwith put a partial mirror on your laptop today!