Creating a Local Ubuntu Mirror

June 28, 2006 by Isaac

Since part of my job entails system administration I get to spend a lot of time configuring, fixing, setting up, and tweaking computers. I'm currently setting up several servers and since my favorite Linux distribution of all time is Ubuntu, guess what I'm installing?

Ubuntu is a Debian derivative, and as such uses apt to install packages. Normally, you run apt-get (or some front-end such as Synaptic) to install/remove package. It then resolves all the dependencies, downloads, and installs the software for you. So, for example, you could run apt-get install kde and after downloading a couple hundred megs of software in dozens of packages you would be up and running. Sure beats doing it by hand.

For a home user that's great, run apt-get and let 'er rip. However, for corporate users it's not so nice. Bandwidth is a premium and if there are multiple computers running Ubuntu (or Debian or whatever) then it would save tons of bandwidth to have a local package repository. Fortunately, there is a package called apt-mirror that does just such a thing for us. The apt-mirror package works for not only Debian-based systems, but it will also work for RPM based systems. I will cover Ubuntu specifically, but the apt-mirror website covers other systems.

For Ubuntu, make sure that the 'universe' repository is added to your list of package sources. Then, type:

apt-get install apt-mirror

Once apt-mirror is installed, edit /etc/apt/mirror.list. For reference, I've listed my mirror.list source below. In the blog entry some of the lines were wrapped, but in reality they are one line. I believe that if you cut and past that it will all come out ok.

# apt-mirror configuration file
set base_path /ubuntu/ubuntu-mirror
set mirror_path $base_path/mirror
set skel_path $base_path/skel
set var_path $base_path/var
#
set nthreads 5
#
#

#
# Sources
#

# Dapper
deb http://us.archive.ubuntu.com/ubuntu/ dapper main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ dapper main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ dapper-updates main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ dapper-updates main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ dapper-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ dapper-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu dapper-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu dapper-security main restricted universe multiverse

# Breezy
deb http://us.archive.ubuntu.com/ubuntu/ breezy main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ breezy main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ breezy-updates main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ breezy-updates main restricted universe multiverse
deb http://us.archive.ubuntu.com/ubuntu/ breezy-backports main restricted universe multiverse
deb-src http://us.archive.ubuntu.com/ubuntu/ breezy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu breezy-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu breezy-security main restricted universe multiverse

Looks just like sources.list doesn't it? That's by design and I am so glad they retained compatibility.

You'll notice from the above configuration file that I keep my mirrored repositories in /ubuntu/ubuntu-mirror. You should of course change these to suit your own setup. Now that our mirror.list file is complete we need to create the directories.

mkdir /ubuntu/ubuntu-mirror
mkdir /ubuntu/ubuntu-mirror/mirror
mkdir /ubuntu/ubuntu-mirror/skel
mkdir /ubuntu/ubuntu-mirror/var

Finally, we are ready to run apt-mirror. Make sure you have plenty of disk space and TONS of time, as depending on what you put into your config file this will take a very long time.

After an initial download of ***MANY*** hours you will have a local copy of around 100GB for the entire Ubuntu repository. Of course you don't have to download the whole thing. If you know for sure that no one in your organization uses anything besides the i386 version then you can stick to that repository only and get away with around 30GB of downloads.

From here you need to configure all your machines to use the new local repository, instead of downloading the packages from the internet. Your new Ubuntu mirror server needs to be configured to allow other computers to access the packages. Personally, I set up an Apache webserver and through some symbolic links and an alias directive in the Apache configuration I allowed access to my repository via http://my.machine/ubuntu.

In my installation that made sense, as I also needed a webserver. However, depending on your requirements a FTP server or NFS mount might be more appropriate. The specifics of these configurations are beyond the scope of this article. In another blog entry I do cover some basic vsftpd configuration. However, if you're actually doing this then I assume you have a fair bit of system administration under your belt. In the end your local repository will be accessed over HTTP, FTP, or NFS.

Of course my mirror server also runs Ubuntu, so the sources.list file on my mirror machine needs to point somewhere. From that machine, here is my sources.list file:

deb file:///ubuntu/www/archive/ dapper main restricted universe multiverse
deb-src file:///ubuntu/www/archive/ dapper main restricted universe multiverse
deb file:///ubuntu/www/archive/ dapper-updates main restricted universe multiverse
deb-src file:///ubuntu/www/archive/ dapper-updates main restricted universe multiverse
# deb file:///ubuntu/www/archive/ dapper-backports main restricted universe multiverse
# deb-src file:///ubuntu/www/archive/ dapper-backports main restricted universe multiverse
deb file:///ubuntu/www/security/ dapper-security main restricted universe multiverse
deb-src file:///ubuntu/www/security/ dapper-security main restricted universe multiverse

This particular computer hosts the mirror, but is running Ubuntu, so I use the file:// protocol specifier. All of the other computers on the network use http to access the repository. Also note that my mirror is installed under /ubuntu/ubuntu-mirror. However, in the sources.list file that's not where I tell it to get the files. Remember my mention of symbolic links? This is where that comes into play.

Once the sources.list file is set up correctly on all your corporate computers, run apt-get update to resync the list of packages and you are in business.

Now, in doing this I ran into a frustrating problem. The packages downloaded just fine, and my sources were configured properly. However, when I ran apt-get update I would get the following error:

W: GPG error: file: dapper-updates Release: The following signatures were invalid: BADSIG 40976EAF437D05B5 Ubuntu Archive Automatic Signing Key <ftpmaster {at} ubuntu(.)com>ftpmaster {at} ubuntu(.)com> W: You may want to run apt-get update to correct these problems

After MANY frustrating hours searching the web I found some solutions. Basically, the description given was that for whatever reason the gpg key used to sign the Release file was not on my keyring. However, I did notice a funny thing. After running apt-mirror a couple days after my initial download this problem went away. Perhaps it was some glitch in the repositories online? In any case, if you continue to have this problem then you may need to import the gpg key. To do this run the following (as root):

gpg --keyserver subkeys.pgp.net --recv KEY
gpg --export --armor KEY | apt-key add -

Where you substitute the signature key number from the error message for KEY. In my case the key was 40976EAF437D05B5. Of course if that doesn't work then there are other options. I've found that deleting the offending Release.gpg file is one way (although not the best) to get rid of this problem. The other way is to add this to your apt.conf file:

APT::Get::AllowUnauthenticated 1 ;

Neither of those methods are suggested, but if downloading the key doesn't work for you then you may be stuck with one of those.

In the end this new mirror server worked great and updating the Ubuntu computers behind our corporate firewall is SO much faster.

--

Updated: 17 July 2006
I added some more details and clarified a few points.

Posted in Linux