Menu
Index

Contact
LinkedIn
GitHub
Atom Feed
Comments Atom Feed



Tweet

Recent Articles

23/04/2017 14:21
Raspberry Pi SD Card Test
07/04/2017 10:54
DNS Firewall (blackhole malicious, like Pi-hole) with bind9
28/03/2017 13:07
Kubernetes to learn Part 4
23/03/2017 16:09
Kubernetes to learn Part 3
21/03/2017 15:18
Kubernetes to learn Part 2

Glen Pitt-Pladdy :: Blog

Home Lab Project: apt-cacher-ng with CentOS

With running a Home Lab with lots of different VMs which all need updating, but all have mostly the same packages, it makes a lot of sense to cache packages.

With CentOS, out the box it likes to balance between mirrors by looking up from a Mirror List which means that each time your packages may come from a different place. You could disable this, but then you risk creating hot-spots on mirrors which is neither good for CentOS nor your time.

This article shows how to solve that with a bit of extra config with apt-cacher-ng on Debian Jessie. This can handle caching for yum as well as apt, so don't let the name put you off.

Client config

To make yum use apt-cacher-proxy add the following line in your /etc/yum.conf:

proxy=http://address.of.your.cache:3142

Easy as that!

Mirror Lists

Out the box apt-cacher-ng will not pass mirror list lookups and log them as forbidden. To solve that we need to add an extra VfilePattern in /etc/apt-cacher-ng/acng.conf to match the request pattern:

VfilePatternEx: ^/\?release=[0-9]+&arch=

Using VfilePattern(Ex) allow Volatile (short caching) of results.

Other Files

Update 20160702 - I hit a problem when the GPG key couldn't be retrieved via the proxy. You can extend this pattern to include things like keys or any other things that get thrown in the mix. Check apt-cacher.log for "403 Forbidden" and add further matching into the pattern. For an example key the pattern becomes:

VfilePatternEx: ^(/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY-examplevendor)$

Merging Mirrors

It's rather a waste to have all the different mirrors caching differently when they should all have the same, interchangeable content. To solve this we need to tell apt-cacher-ng about all the mirrors involved so it can group them up. For this add a Remap line to /etc/apt-cacher-ng/acng.conf like this:

Remap-centos: file:centos_mirrors /centos

Then you will need to provide a mirror list in /etc/apt-cacher-ng/centos_mirrors which can be done by fetching it from CentOS and extracting the http cells with a one-liner:

# curl https://www.centos.org/download/full-mirrorlist.csv | sed 's/^.*"http:/http:/' | sed 's/".*$//' | grep ^http >/etc/apt-cacher-ng/centos_mirrors

SSL Passthrough

Update 20160702 - One snag you may encounter is that there's quite a few repos around these days that have some aspect as SSL which apt-cacher-ng doesn't handle out the box. One example is if you install the "epel-release" package in CentOS 7 this looks up mirrors with SSL and fails.

The way this can be handled is to put in a PassThroughPattern in /etc/apt-cacher-ng/acng.conf which could look like something like this:

PassThroughPattern: (mirrors\.fedoraproject\.org|some\.other\.repo|yet\.another\.repo):443

This basically enables CONNECT requests to pass through for requests matching these three patterns. Keep in mind that there is no caching in this case. Particularly for large infrastructures or Lab setups (my case) you might want to consider using non-SSL repos to use the cache and save bandwidth. There is still a level of safety without SSL (relying on GPG signatures alone), but you should weigh up the security requirements for your particular use case.

Making it work

To bring the config live you need to restart restarted apt-cacher-ng with:

# systemctl restart apt-cacher-ng.service

If that fails then checking the logs is normally a good place to start. Chances are there's a typo or something that's crept in.

Then on your CentOS box you can force yum to clear caches and re-try:

# yum clean expire-cache
# yum update

With the configuration in place, on your cache machine you should see files being fetched via apt-cacher-ng and a "centos" directory where mirrors are merged in /var/cache/apt-cacher-ng/ should appear.

Comments:

Glenn Wood Image  26/04/2016 02:10 :: Glenn Wood

Excellent! Worked first time!

Anwar Yagoub Image  01/06/2016 11:21 :: Anwar Yagoub

very helpful

Emmanuel Lacour Image  14/06/2016 16:48 :: Emmanuel Lacour

Thanks, on centos 7 at least, "yum clear expire-cache" is an unknown option, "yum clean expire-cache" seems ok

Glen Pitt-Pladdy Image  15/06/2016 15:14 :: Glen Pitt-Pladdy

Quite right! Well spotted - I'll fix that typo

James Burnett Image  13/07/2016 14:55 :: James Burnett

Fantastic, just what I wanted. Thanks!

James Burnett Image  13/07/2016 15:32 :: James Burnett

I feel I should also point out that you refer to editing /etc/acng.conf a couple of times, but it's actually /etc/apt-cacher-ng/acng.conf (as you refer to in the SSL passthrough section).

Cheers!

Glen Pitt-Pladdy Image  13/07/2016 15:37 :: Glen Pitt-Pladdy

Well spotted - thanks for the feedback. I'll fix that shortly.

I also realise that I didn't say anything about restarting it to bring the extra config live which might not be so obvious to people that aren't familiar with this stuff.




Are you human? (reduces spam)
Note: Identity details will be stored in a cookie. Posts may not appear immediately