!!! WARNING - Don't do this !!!

This is an extremely dangerous thing to do and can result in severe data loss, corruption and an awful lot of stuff I haven't even thought of. This article deals with one particular configuration (scenario) and different systems will behave differently.

This is not for the feint of heart or inexperienced system administrators and should only be attempted by ninja sysadmins. It's highly likely that at some point you will find stuff that doesn't behave as expected and you will need to work things out on your own.

Do not attempt this if you do not fully understand the risks and what is involved, or do not have front-line experience getting seriously broken systems back to life.

Ensure that you have comprehensive backups and are able to restore from those backups. The important bit being that you can restore the backups. A lot of people have backups, but when it comes to restoring things don't always work out the way they expected. The important thing is not that you have a backup but that you can restore it.

A number things have architecture specific file formats (eg. some databases) so ensure you know what is needed on your systems.

This process is widely considered impossible which should give you an idea what you are playing with here. Don't do this unless you are happy loosing everything.

Background & Credits

This process is really an updated version with my personal experience of what Robin Lee Powell already did before.

I upgraded my home server to a faster dual-core 64-bit capable processor to help it cope with the hammering I give it and wanted leave the legacy stuff behind and get the performance benefits of 64-bit. Nope, it's not twice as fast or anything like that, maybe up to 20-30% depending on what you are doing. Some stuff may even be slower on 64-bit, and if the bottle neck is disk thrashing then it's going to make negligible difference.

In my case the server takes care of a lot of stuff:

• MythTV
• OpenVz
• MySQL (for several things)
• Apache (several websites)
• Cacti (monitoring)
• Lots of other small stuff

Post-mortem

I put this first as it's really stuff that needs considering first, but I only find out about by completing the conversion.

Consider what would happen if your box got totally bricked by this. Would you have internet access? Would you be able to download extra packages you needed? What would the implications be? It's worth making sure you can download all the packages you may need if you run into trouble.

It's also worth making sure you can pick individual files off your backups if you get into trouble. Occasionally one missing library could see you with a broken package manager or similar. Also it's worth having both 32-bit and 64-bit boot CDs / USB devices to hand if you need to fix things.

Beware of custom built packages - if you have any then re-build them for the new architecture before you start. That will save a load of grief. In my case my MythTV is built from Ubuntu source package so that it matches what is run on Ubuntu clients.

Make sure you have plenty of time, disk space and bandwidth. This process will download everything you have installed again.

I would also suggest doing this on a test machine that you can learn on first and break. A VM is ideal here as you can snapshot it and roll it back to where you made what since turned out to be a mistake. There will still be things you don't expect, but I did this and it was definitely worthwhile. In my case I have a VM I used to build the 32-bit MythTV packages and that was a good candidate to try everything out and rebuild the packages again first. Many breakages turn out to have been due to something done much earlier in the process and can be difficult to work out.

Stuff that breaks

I might have been really lucky, but the only thing that gave me any real problems was the .rrd files used by Cacti. Many other monitoring / graphing systems also depend on RRDs so pay attention here. The .rrd format is architecture dependant so the files need to be dumped and restored before and after.

As the .xml dumps are pretty big and I monitor loads of stuff (several hundred RRDs), I gzipped them on the go. In /var/lib/cacti/rra I did:

# for f in *.rrd; do echo $f; rrdtool dump $f | gzip --fast >$f.xml.gz; done

Also take a note of the permissions of the .rrd files else you could get into trouble later.

OpenVz was another problem with the quota files, but these are easily recreated so no major problem there.

The only other breakage was that my snmp user UID got changed. That just meant the cache files for my Apache stats needed the owner changing, but otherwise all was fine.

Getting ready

You will need a stack of amd64 packages for:

• 32-bit compatibility libraries
• 64-bit kernel
• 64-bit libraries for dpkg & amt
• various other 64-bit packages
• it's also useful to have 64-bit packages for your favourite editor, in my case vim

The packages I needed where....

32-bit compatibility

You need ia32-libs and dependencies. The full list I used was:

• ia32-libs_20111001_amd64.deb
• lib32ncurses5_5.7+20100313-5_amd64.deb
• libc6-i386_2.11.2-10_amd64.deb
• lib32asound2_1.0.23-2.1_amd64.deb
• lib32stdc++6_4.4.5-8_amd64.deb
• libv4l-0_0.8.0-1_amd64.deb
• lib32bz2-1.0_1.0.5-6_amd64.deb
• lib32v4l-0_0.8.0-1_amd64.deb
• lib32gcc1_1%3a4.4.5-8_amd64.deb
• lib32z1_1%3a1.2.3.4.dfsg-3_amd64.deb

64-bit kernel

As I am running OpenVz (whicn incidentally won't work with a 64-bit kernel and 32-bit vzctl / vzquota), I got that kernel:

• linux-image-2.6.32-5-openvz-amd64_2.6.32-39_amd64.deb

64-bit libraries for dpkg & apt

These are needed to get the package management up again once in 64-bit mode:

• libbz2-1.0_1.0.5-6_amd64.deb
• libgcc1_1%3a4.4.5-8_amd64.deb
• libstdc++6_4.4.5-8_amd64.deb
• libc6_2.11.2-10_amd64.deb
• libselinux1_2.0.96-1_amd64.deb
• zlib1g_1%3a1.2.3.4.dfsg-3_amd64.deb

And of course dpkg and apt themselves:

• apt_0.8.10.3+squeeze1_amd64.deb
• dpkg_1.15.8.11_amd64.deb

Other 64-bit packages & Editor

In my case:

• sysvinit_2.88dsf-13.1_amd64.debb
• perl-base_5.10.1-17squeeze2_amd64.deb
• vim_7.2.445+hg~cb94c42c0e1a-1_amd64.deb
• vim-common_7.2.445+hg~cb94c42c0e1a-1_amd64.deb

There is always a chance that you may need more on your system so ensure you have a way of downloading and getting the packages onto the target system.

Initial steps

First off we are going to seriously mess up the package management by doing this so we need to get an idea of the state the system is in. I would also suggest cleaning up any old cruft before starting and making sure the system is in a nice consistent state before starting. Tidy systems are good normally, but here they are critical.

To dump the current package list:

# dpkg --get-selections >list_i386

Keep that file somewhere safe, in fact keep several copies in safe places - we will need it later.

Although I didn't need it, I would also suggest as a safety net having the staticly linked version of busybox on hand. Install busybox-static and then on at least one terminal get a root instance running in case you need to get out of trouble:

# busybox ash

The rest of the process seems to work via ssh or in X, but I would suggest you have a few terminals (including on the console) open with root shells running to get you out of trouble.

During the process the system will be extremely fragile and won't tolerate a reboot so ensuring you have a dependable power supply is also important.

On with the show - 64-bit Kernel

Install the new kernel and reboot onto it to get the 64-bit kernel layer. It still supports most 32-bit stuff:

# dpkg --force-architecture -i linux-image-2.6.32-5-openvz-amd64_2.6.32-39_amd64.deb

After reboot we should see a 64-bit kernel:

# uname -a
Linux boni 2.6.32-5-openvz-amd64 #1 SMP Thu Nov 3 04:22:10 UTC 2011 x86_64 GNU/Linux

32-bit Compatibility

Get the compatibility libraries in first - these will allow the existing (some of them anyway) binaries to continue running:

# dpkg --force-architecture --force-overwrite -i libc6-i386_2.11.2-10_amd64.deb
# dpkg --force-architecture -i ia32-libs_20111001_amd64.deb  lib32bz2-1.0_1.0.5-6_amd64.deb  lib32asound2_1.0.23-2.1_amd64.deb lib32gcc1_1%3a4.4.5-8_amd64.deb lib32ncurses5_5.7+20100313-5_amd64.deb lib32stdc++6_4.4.5-8_amd64.deb lib32v4l-0_0.8.0-1_amd64.deb lib32z1_1%3a1.2.3.4.dfsg-3_amd64.deb
# dpkg --force-architecture -i libv4l-0_0.8.0-1_amd64.deb

If anything has failed to configure then give it a kick:

# dpkg --configure -a

The other thing is that we will need /lib/ld-linux.so.2 symlinked to /lib32/ld-linux.so.2:

# ln -sf /lib32/ld-linux.so.2 /lib/ld-linux.so.2

If everything is still running and the installs completed successfully then we are doing well.

dpkg and the 64-bit leap

Once the 64-bit dpkg is in, 64-bit packages will be used as native, but if this step is done wrong then you can end up with no package management. Make sure you take a backup copy of the existing dpkg and dpkg-deb which will get you out of most trouble if you replace them with the 32-bit ones again.

libc6 is the first thing that we need to sort as virtually everything depends on it:

# dpkg --force-architecture -i libc6_2.11.2-10_amd64.deb

Then before we make the leap to 64-bit dpkg we need to install some dependencies:

# dpkg --force-architecture -i zlib1g_1%3a1.2.3.4.dfsg-3_amd64.deb libbz2-1.0_1.0.5-6_amd64.deb libselinux1_2.0.96-1_amd64.deb libstdc++6_4.4.5-8_amd64.deb

Then dpkg and apt can be installed:

# dpkg --force-architecture -i dpkg_1.15.8.11_amd64.deb apt_0.8.10.3+squeeze1_amd64.deb

At that point dpkg should be working still, and now on amd64:

# dpkg --print-architecture
amd64

An extra package was needed for apt to work:

# dpkg --force-architecture -i libgcc1_1%3a4.4.5-8_amd64.deb

You should be able to run:

# apt-get update

If that works then we are getting to the exciting bit. Don't proceed unless you are sure the package management is solid.

Replacing 32-bit with 64-bit

Next we are going to let apt-get do it's magic to replace all the 32-bit packages with 64-bit. Care is needed as if you have a crufty system or 32-bit packages you built yourself (ie. apt-get can't download them again) then it could get messy. Check that the action to be taken is sensible before proceeding:

# apt-get -f install

In my case I had to install the 64-bit builds of MythTV and related packages before it would work without wanting to remove critical system components. During this run I also got a breakage with telinit not working... hence having sysvinit downloaded and on hand. Install the new sysvinit manually:

# dpkg -i sysvinit_2.88dsf-13.1_amd64.deb

Configure everything so far:

# dpkg --configure -a

Now we are ready for a few more rounds:

# apt-get -f install

.... repeating that as needed.

Again I ran into a breakage with perl and needed to install a package manually forcing an overwrite:

# dpkg --force-overwrite -i /var/cache/apt/archives/perl-base_5.10.1-17squeeze2_amd64.deb

Now we are ready for a few more rounds:

# apt-get -f install

.... repeating that as needed.

I also got a breakage with kbd. In my case I was using console-tools which was conflicting. The aim is to get back to the original state so:

# apt-get --reinstall install console-tools libconsole

And then do some more rounds of apt-geting. Once there is nothing else to fix then we can move on to picking up the missing bits.

Picking up the pieces

Right now the fundamental stuff to keep the system up is in place but there are an awful lot of 32-bit packages still present and ready to break the system should you reboot, blink etc. We need to find these and we will start from the package list we created earlier:

# for package in `grep '[^e]install$' list_i386 | awk '{ print $1 }'`; do dpkg -s $package | grep -q 'Architecture: i386' && echo $package; done >list_still_i386

Then run apt-get to reinstall these packages:

# apt-get --reinstall install `cat list_still_i386`

If apt-get complains, chances are there are some packages you have built yourself, packages that are i386 specific (eg. old kernels) or other cruft on the system. Sort those manually and then set it off again. In some cases you may need to edit the file (that's where having your editor packages on hand is useful) to remove packages that can't be downloaded but we don't want to touch just yet like libc6-i686.

Once that's run then get the list again and see if there is anything else in it that is viable.

Next we will do another check which may give the same results but is worth doing none the less:

# for package in `dpkg -l  |  grep '^i' | awk '{ print $2 }'`; do dpkg -s $package | grep -q 'Architecture: i386' && echo $package; done >list_still_i386

This is really the same thing, just using dpkg's status rather than the stored list from earlier. Same story here - fix up what is viable. In many cases the remaining packages can just be removed.

The acid test

All going to plan the system should be fairly safe now. It's time to test that and reboot!

With a bit of luck you are back at a mostly working machine. We now need to get things back to as they where. Time for that package list again:

# dpkg --get-selections | diff list_i386 -

Everything with a > is something new that wasn't on the machine before, and everything with a < is something that has gone away. In some cases these will be things like the old 32-bit kernel or other cruft. Fix them up as appropriate for your system and get the system back to as it was.

You may also want to run the previous checks for 32-bit stuff again just to make sure you got it all.

Restoring RRDs

At this point we can restore the .rrd files for Cacti in /var/lib/cacti/rra with:

# for f in *.rrd.xml.gz; do echo $f; d=`echo $f | sed 's/.xml.gz$//'`; zcat $f >tmp.xml; rrdtool restore tmp.xml $d -f ; done

Make sure they all work then remove the dumps and tmp.xml. If needed change the permissions to match the originals.

Resuscitating OpenVz

OpenVz seems to have architecture dependant quota files. It also get's it's self stuck with quota on which means you can't really do much. I found the easiest thing here was just to remove the files in /var/lib/vzquota and reboot which started from a clean state and re-creates the quota files.

Everything else

Almost everything just works for me. The only thing I had to do was fix the ownership for the cache files for my Apache snmpd extension script as the snmp UID changed.

All seems well but it's still important to watch things closely for breakages. Things I was concerned with like MySQL seem to have worked flawlessly.

Considering how extreme this process is, it's actually worked out far easier than I expected. In total it took me about 4-hours to have a fully converted system including the time with apt-get downloading and installing all the replacement packages.