Creating a partial Debian mirror

When installing multiple Debian Linux systems (ok, GNU/Linux) with something like FAI, a mirror for some debian packages is more or less a requirement. It's possible to mirror a whole Debian repository, but I'm tight on diskspace and it wastes bandwidth. Also I want the mirroring tool to be able to run on other distributions and operating systems than Debian.

Existing solutions

• Partial debian mirror uses a chroot environment and uses apt-get in there so only packages that are actually needed are fetched. A repository is then created using apt-move. This is would be an option, but it needs root to work and it might need some work for non-linux environments.
• Repository proxies. You point your sources.list to a proxy that caches on demand. Nice but it's only possible to use http in the client's sources.list; using file doesn't need a big /var partition on the client when using FAI and it saves some time without an explicit download step.
  • apt-proxy
  • apt-cacher
  • aptcached
• Mirror tools for whole repositories.
  • debmirror mirrors whole repositories.
  • DpartialMirror is an attempt to optimize the synchronizing of a remote distribution with a local one. Synchronizing is done through the script distsync. A helper script dsc2man converts descriptions of the synched distribution to man pages.
  • apt-mirror
  • Setting up a Debian archive mirror

Debpartmirror

W. Borgert's partial mirror scripts didn't exist at the time and would have been a good starting point. My solution is as follows:

1. Use apt-get to fetch the packages and their dependencies.
   apt.conf enables you to use apt-get completely separate from the system. This way apt-get can function independently from the system state. apt-get first refreshes the package lists that the fai clients use (only one source.list for all clients is supported). For each file in the pkglists/ directory, packages and dependencies are fetched. This allows conflicting packages to be fetched when put in separate files.
2. Use apt-move to generate a repository from the fetched packages.
   apt-move gets all packages in apt-get's local cache directory and puts them in a nice repository that is apt-get-able.

Debpartmirror is packed in an archive that should run out-of-the-box, download it here. It contains binaries so it can be run on any linux distribution with libc6. Run ./move.sh in the tools subdirectory. On first run, it will create an empty repository. It will print a help message when run without arguments. move.sh basically is a wrapper script for bin/apt-move that adds some functionality. There is not much documentation, but to start you should do the following (in the tools/ directory).

1. Make your own sources.list
2. Edit the package lists in pkglists/. When using FAI, you can symlink your existing package-list-directory, adding the file _BOOTSTRAP_ (found in debpartmirror's pkglists dir).
3. Run ./move.sh faimirror, and the repository will be created. You can then nfsmount the ../debian/ directory on your client boxes and add sources.list entries. See move.sh's command filesources.

Apt on FreeBSD

A final note about portability. To compile apt on FreeBSD, a small patch is needed. For your convenience, you can also download already compiled apt tools for FreeBSD4. Put apt-get, gzip (not included) and the directory methods/ in debpartmirror's tools/bin directory.

&copyleft; 2004 by Willem van Engen