On ports vs pkg_add — a digression into keeping your FreeBSD programs up to date with the latest patches and software

Something that confused the hell out of me when I first started using FreeBSD was ports vs pkg_add. It was worth getting used to ports, I think, because these days I use nothing else.

What’s the difference?

The initial difference that struck me was speed. pkg_add installs things a lot faster than ports. Recently, I had to install TeXMacs, a program I hope dies a painful death in a pit full of wildebeest droppings somewhere, and I had to install it fast. To do that, I started building the port, installed the binary package with pkg_add and once my port was finished being made I deinstalled the pkg and installed the port. It took about ten seconds to install the pkg, and about an hour and a half to install the port.

Given that MASSIVE speed difference, why the hell would anyone use ports? The answer is that they’re magically delicious in every single way.

No, but seriously. Once you start using ports rather than pkges, I doubt you’ll ever ever ever go back to the pkg.

Yeah, but what’s the sodding difference?

Okay. A pkg is a prebuilt binary. A port is building the same binary, but doing it from source on the local machine.

So, a pkg is built in a generic kind of way. It runs on most machines. A prebuilt binary is how windows, Mac OS, most linuxes, all distribute their stuff. My linux of choice is Ubuntu. When you do an add/remove programs install, a synaptic install, or (for me) an apt-get install, you’re installing prebuilt binaries. They’re convenient, as they generally have a lot of the fiddly bits they need prepackaged with them. When a pkg just works, it’s a beautiful and fast thing.

The problem is when they don’t. That’s when ports shine.

A port is to a pkg as gold is to brass. Brass is nice, but once you sell a pound of brass and a pound of gold then you’ll never want brass again.

The best way to get familiar with ports is by using them. Do you have a ports tree? If so, and you’ve never used it, it’ll be out of date.

If not, that sucks and you should have one.

It is necessary, absolutely necessary, to have your ports tree always up to date if you’re running on ports. I discovered this in a painful way, by mixing ports and pkges.

A Brief Digression
Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time.

When I came back to it, I had no idea what to do. Fortunately, a friend of mine is a FreeBSD guru. So I asked him. And he talked me through solving all my issues.

First, he said, update your ports tree. He left me to work out how to do it. I read things about cvsup, I read things about new tools that had just been developed, I had no idea. I stumbled through the most excellent FreeBSD diary until I discovered the solution, and I updated my ports tree.

Next, he said, you’re fortunate to have installed portupgrade a long time ago. Run this command:

portupgrade -ay

This took about three or four days to run. A long long time. Especially as I did not know what screen was back then. Ouch!

There were options screens occasionally, which I answered as best I could, and eventually it was done. And my god, the system worked! I had to know more!

Digression Digressed

First, become root. This assumes you’re root. If you don’t have a /usr/ports directory, do a:

mkdir /usr/ports

If you don’t have a /usr directory, you are in a world of pain and I pity you.

FreeBSD contains a tool that brings a ports tree up to date in a relatively painless way. This is portsnap. To update your ports tree using portsnap, run:

portsnap fetch

This takes a little time to run. Be patient, brew a cup of tea and drink it.

Now run:

portsnap extract

This takes time to run, too. Maybe five, ten minutes depending on your computer’s speed. Notice what’s being spread across your screen: why, it’s pathnames with packagenames inside!

Once it’s done, execute:

cd /usr/ports

And do an ls.

You’ll notice names of types of packages. Here’s mine right now:

.cvsignore INDEX-6.bz2 Templates audio deskutils ftp java net ports-mgmt textproc x11-fonts
.portsnap.INDEX INDEX-6.db Tools benchmarks devel games korean net-im portuguese ukrainian x11-servers
CHANGES KNOBS UIDs biology distfiles german lang net-mgmt print vietnamese x11-themes
COPYRIGHT LEGAL UPDATING cad dns graphics mail net-p2p russian www x11-toolkits
GIDs MOVED accessibility chinese editors hebrew math news science x11 x11-wm
INDEX Makefile arabic comms emulators hungarian mbone packages security x11-clocks
INDEX-5 Mk archivers converters finance irc misc palm shells x11-drivers
INDEX-6 README astro databases french japanese multimedia polish sysutils x11-fm

Lots of packages. Some indexes, a README, a file named UPDATING and so on.

Good stuff. Damn good stuff.

cd into something! How about this:

cd ports-mgmt

An ls will show a bunch of package names. They’re all tools that manage ports. This is the ports management section of the ports tree.

The first thing I always install is portupgrade.

cd portupgrade

Before going any further, do an ls. You’ll see some neat files. A pkg-descr, a Makefile, a pkg-message, a pkg-plist and distinfo. What are all of these?

Well, it took me a long time to discover, but I know these days.

The Makefile describes everything FreeBSD needs to know to make a binary of this package. It tells FreeBSD where to download the source code from, then what to do with it. It tells FreeBSD what options are available for this port, so that it builds differently, and anything else FreeBSD needs to know to build this file so it will work on your computer.

Do a less on the Makefile and give it a read. You’ll understand some stuff, not others. It’s all shell script, and you’ll eventually get very very very used to reading it if you administer a FreeBSD system.

distinfo contains information for FreeBSD to know that when you download the source code (which FreeBSD will do automatically for you at the right time) it is not corrupt. It contains an md5sum and a SHA256 sum and a size for the package. If you don’t know how these work, it’s interesting reading around and finding out what they do. For now, though, just understand that they GUARANTEE that your package will be built properly.

The pkg-descr is exactly what it almost sounds like. It’s a description of the contents of the package. For this package, the most useful thing to note iniside it is that you’re going to get several cool extra tools, like pkg_deinstall and pkgdb. My god, if I could count the number of times pkgdb has saved my ass, wow.

pkg-mesg is a neat message that’ll pop onto the screen when the port is installed, giving VERY USEFUL information about how to configure it properly. You often have to scroll the screen up a little to see its contents, so bear that in mind. The pkg-mesg for moinmoin, for example, tells you where and how your moin install is and what to do to make it work. Ports are friendly that way, they often tells you what you need to do to make your install work.

pkg-plist contains a list of packages installed by this port, so that when deinstall happens everything is deinstalled properly.

Some of these files may not exist, some different files might be there. The canonical reference is, as ever, the handbook.

So anyway, to install portupgrade from this port, type “make install clean”. You’ll see a bunch of downloading occur, a bunch of making of files, maybe a bunch that you weren’t aware you need to have installed. For example, it’s not uncommon to see ruby on rails be installed by ports managment stuff, a lot of it seems to be written in ruby, or for moinmoin you’ll see python installed because that’s the language python is written in.

The neat thing is for web related stuff. Ports are so damn smart that if you’ve got apache installed and you install moin, it’ll install mod_python for you automatically so that you’re ready to go. If you hadn’t already installed apache, you’ll have to do some manual fiddling, though.

So now you’ve got portupgrade installed, the best thing to do is decide whether you’re going for a ports based system or a pkg based system. YOU SHOULD NOT MIX. PICK ONE AND STICK WITH IT. HELL WILL HAPPEN IF YOU DON’T.

Why Hell?

pkges are built for you at specific version numbers. Ports, on the other hand, you build from the ports tree you download with portsnap. The Makefiles are often at different version numbers than the pkges. But pkg_add will clobber your currently installed ports! So you can install an old version of a package over a new version.

Remember above, I said ports download recursively, satisfying their dependencies? Well, if your version of moin is dependent on python 2.5, and the current pkg was only at 2.4, then when you pkg_add python and destroy your current version, moin will stop working!

Now, stop panicking quite so much, FreeBSD is smarter than this overly simple example. But the principle is true. Minor revisions are actually important numbers.

You will mire your entire operating system in a world of shit by mixing the two different systems.

How To Solve This Issue?

We actually just installed the tool to solve this issue, portupgrade. Run:

“portupgrade -ay”

It will discover all your installed ports OR packages, test their version number against the version in the ports tree and make sure you’re at the appropriate version. Portupgrade is discussed in the handbook, but be aware that it’s simply a completely badass tool.

There are other tools available for keeping your installed packages up to date with the latest packages and patches and so on, but these are the ones I use.

Isn’t There a Catch?

There are a couple of catches to using ports.

First, pkgdb, the database it uses to keep track of everything, occasionally notices conflicts that must be fixed. There is one command to solve this:

pkgdb -F

Last weekend, for whatever reason I had to run this four times in a row. But that’s all I had to do to solve the issue, run that command. That’s ease of use for you, as far as I’m concerned. If you try to install anything with a conflict, the install will fail telling you to run pkgdb -F.

The other is UPDATING. In /usr/ports, there’s a text file called UPDATING. This contains useful information on known security issues, problems with certain installs. Recently, FreeBSD moved to xorg 7.2, and there’s a whole list in UPDATING on how to update to 7.2 using ports. It’s a pain having to follow the steps, but that’s all you have to do. Just follow the UPDATING steps and you’ll be fine. FOLLOW ALL OF THEM.

And that’s that.

This upgrades all your installed ports to be the latest and greatest. It’s the tools that automate everything that make ports so amazing to me. That and knowing that every package is there specifically for my machine.

But I Need It In A Hurry

This was the issue with TeXMacs. I needed it fast. So, what did I do? I cded into /usr/ports/editors/texmacs && make

This started making the binary in the background.

Then I crossed my fingers, prayed, and did a pkg_add -r texmacs.

This installed hte binary which, fortunately, didn’t clash with my system. I used the binary until the port was built (ports are usually fast, incidentally, just not in the case of larger installs) and then did a:

cd /usr/ports/editors/texmacs && make deinstall && make install && make clean

So I deinstalled the pkg, which ports is smart enough to do, then installed my already built binary and cleaned up afterwards.

I’ve ran into versioning issues with pkg_add, but never with ports. And it’s never hard to keep up to date with ports. To find a port to install, simply:

cd /usr/ports

make search name=pkgname

So to locate the latest version of python:

make search name=python | less

Bunches of stuff flies past the screen, so pipe it to less so you can read at your leisure. You’ll find it.

Ports is very very comprehensive. It seems to have, for example, the entire CPAN library for Perl. It doesn’t have all the python packages I’d like, but it has a lot of them. If you find something that ports doesn’t have, consider making a port and submitting it: http://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/ is the documentation for how to make a port.

Now that you’ve solved your package maintenance issues, in the future we’ll consider how to make your base distribution up to snuff with the current patches.

P.S I’m Security Paranoid

For those amongst us who TRUST NOTHING, consider the port portaudit port. It’s in /usr/ports/ports-mgmt — go ahead and cd into the portaudit directory and make install clean.

It will email your administrator email address with the results of a portaudit scan against the latest portaudit database every night, telling you about known security issues with anything and how to resolve those issues if any resolution is available. Damn useful! For more information, read the handbook page on portaudit.