oscartheduckin’ around

July 25, 2007

On ports vs pkg_add — a digression into keeping your FreeBSD programs up to date with the latest patches and software

Filed under: freebsd, how to — oscartheduck @ 1:57 pm

Something that confused the hell out of me when I first started using FreeBSD was ports vs pkg_add. It was worth getting used to ports, I think, because these days I use nothing else.

What’s the difference?

The initial difference that struck me was speed. pkg_add installs things a lot faster than ports. Recently, I had to install TeXMacs, a program I hope dies a painful death in a pit full of wildebeest droppings somewhere, and I had to install it fast. To do that, I started building the port, installed the binary package with pkg_add and once my port was finished being made I deinstalled the pkg and installed the port. It took about ten seconds to install the pkg, and about an hour and a half to install the port.

Given that MASSIVE speed difference, why the hell would anyone use ports? The answer is that they’re magically delicious in every single way.

No, but seriously. Once you start using ports rather than pkges, I doubt you’ll ever ever ever go back to the pkg.

Yeah, but what’s the sodding difference?

Okay. A pkg is a prebuilt binary. A port is building the same binary, but doing it from source on the local machine.

So, a pkg is built in a generic kind of way. It runs on most machines. A prebuilt binary is how windows, Mac OS, most linuxes, all distribute their stuff. My linux of choice is Ubuntu. When you do an add/remove programs install, a synaptic install, or (for me) an apt-get install, you’re installing prebuilt binaries. They’re convenient, as they generally have a lot of the fiddly bits they need prepackaged with them. When a pkg just works, it’s a beautiful and fast thing.

The problem is when they don’t. That’s when ports shine.

A port is to a pkg as gold is to brass. Brass is nice, but once you sell a pound of brass and a pound of gold then you’ll never want brass again.

The best way to get familiar with ports is by using them. Do you have a ports tree? If so, and you’ve never used it, it’ll be out of date.

If not, that sucks and you should have one.

It is necessary, absolutely necessary, to have your ports tree always up to date if you’re running on ports. I discovered this in a painful way, by mixing ports and pkges.

A Brief Digression
Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time.

When I came back to it, I had no idea what to do. Fortunately, a friend of mine is a FreeBSD guru. So I asked him. And he talked me through solving all my issues.

First, he said, update your ports tree. He left me to work out how to do it. I read things about cvsup, I read things about new tools that had just been developed, I had no idea. I stumbled through the most excellent FreeBSD diary until I discovered the solution, and I updated my ports tree.

Next, he said, you’re fortunate to have installed portupgrade a long time ago. Run this command:

portupgrade -ay

This took about three or four days to run. A long long time. Especially as I did not know what screen was back then. Ouch!

There were options screens occasionally, which I answered as best I could, and eventually it was done. And my god, the system worked! I had to know more!

Digression Digressed

First, become root. This assumes you’re root. If you don’t have a /usr/ports directory, do a:

mkdir /usr/ports

If you don’t have a /usr directory, you are in a world of pain and I pity you.

FreeBSD contains a tool that brings a ports tree up to date in a relatively painless way. This is portsnap. To update your ports tree using portsnap, run:

portsnap fetch

This takes a little time to run. Be patient, brew a cup of tea and drink it.

Now run:

portsnap extract

This takes time to run, too. Maybe five, ten minutes depending on your computer’s speed. Notice what’s being spread across your screen: why, it’s pathnames with packagenames inside!

Once it’s done, execute:

cd /usr/ports

And do an ls.

You’ll notice names of types of packages. Here’s mine right now:

.cvsignore INDEX-6.bz2 Templates audio deskutils ftp java net ports-mgmt textproc x11-fonts
.portsnap.INDEX INDEX-6.db Tools benchmarks devel games korean net-im portuguese ukrainian x11-servers
CHANGES KNOBS UIDs biology distfiles german lang net-mgmt print vietnamese x11-themes
COPYRIGHT LEGAL UPDATING cad dns graphics mail net-p2p russian www x11-toolkits
GIDs MOVED accessibility chinese editors hebrew math news science x11 x11-wm
INDEX Makefile arabic comms emulators hungarian mbone packages security x11-clocks
INDEX-5 Mk archivers converters finance irc misc palm shells x11-drivers
INDEX-6 README astro databases french japanese multimedia polish sysutils x11-fm

Lots of packages. Some indexes, a README, a file named UPDATING and so on.

Good stuff. Damn good stuff.

cd into something! How about this:

cd ports-mgmt

An ls will show a bunch of package names. They’re all tools that manage ports. This is the ports management section of the ports tree.

The first thing I always install is portupgrade.

cd portupgrade

Before going any further, do an ls. You’ll see some neat files. A pkg-descr, a Makefile, a pkg-message, a pkg-plist and distinfo. What are all of these?

Well, it took me a long time to discover, but I know these days.

The Makefile describes everything FreeBSD needs to know to make a binary of this package. It tells FreeBSD where to download the source code from, then what to do with it. It tells FreeBSD what options are available for this port, so that it builds differently, and anything else FreeBSD needs to know to build this file so it will work on your computer.

Do a less on the Makefile and give it a read. You’ll understand some stuff, not others. It’s all shell script, and you’ll eventually get very very very used to reading it if you administer a FreeBSD system.

distinfo contains information for FreeBSD to know that when you download the source code (which FreeBSD will do automatically for you at the right time) it is not corrupt. It contains an md5sum and a SHA256 sum and a size for the package. If you don’t know how these work, it’s interesting reading around and finding out what they do. For now, though, just understand that they GUARANTEE that your package will be built properly.

The pkg-descr is exactly what it almost sounds like. It’s a description of the contents of the package. For this package, the most useful thing to note iniside it is that you’re going to get several cool extra tools, like pkg_deinstall and pkgdb. My god, if I could count the number of times pkgdb has saved my ass, wow.

pkg-mesg is a neat message that’ll pop onto the screen when the port is installed, giving VERY USEFUL information about how to configure it properly. You often have to scroll the screen up a little to see its contents, so bear that in mind. The pkg-mesg for moinmoin, for example, tells you where and how your moin install is and what to do to make it work. Ports are friendly that way, they often tells you what you need to do to make your install work.

pkg-plist contains a list of packages installed by this port, so that when deinstall happens everything is deinstalled properly.

Some of these files may not exist, some different files might be there. The canonical reference is, as ever, the handbook.

So anyway, to install portupgrade from this port, type “make install clean”. You’ll see a bunch of downloading occur, a bunch of making of files, maybe a bunch that you weren’t aware you need to have installed. For example, it’s not uncommon to see ruby on rails be installed by ports managment stuff, a lot of it seems to be written in ruby, or for moinmoin you’ll see python installed because that’s the language python is written in.

The neat thing is for web related stuff. Ports are so damn smart that if you’ve got apache installed and you install moin, it’ll install mod_python for you automatically so that you’re ready to go. If you hadn’t already installed apache, you’ll have to do some manual fiddling, though.

So now you’ve got portupgrade installed, the best thing to do is decide whether you’re going for a ports based system or a pkg based system. YOU SHOULD NOT MIX. PICK ONE AND STICK WITH IT. HELL WILL HAPPEN IF YOU DON’T.

Why Hell?

pkges are built for you at specific version numbers. Ports, on the other hand, you build from the ports tree you download with portsnap. The Makefiles are often at different version numbers than the pkges. But pkg_add will clobber your currently installed ports! So you can install an old version of a package over a new version.

Remember above, I said ports download recursively, satisfying their dependencies? Well, if your version of moin is dependent on python 2.5, and the current pkg was only at 2.4, then when you pkg_add python and destroy your current version, moin will stop working!

Now, stop panicking quite so much, FreeBSD is smarter than this overly simple example. But the principle is true. Minor revisions are actually important numbers.

You will mire your entire operating system in a world of shit by mixing the two different systems.

How To Solve This Issue?

We actually just installed the tool to solve this issue, portupgrade. Run:

“portupgrade -ay”

It will discover all your installed ports OR packages, test their version number against the version in the ports tree and make sure you’re at the appropriate version. Portupgrade is discussed in the handbook, but be aware that it’s simply a completely badass tool.

There are other tools available for keeping your installed packages up to date with the latest packages and patches and so on, but these are the ones I use.

Isn’t There a Catch?

There are a couple of catches to using ports.

First, pkgdb, the database it uses to keep track of everything, occasionally notices conflicts that must be fixed. There is one command to solve this:

pkgdb -F

Last weekend, for whatever reason I had to run this four times in a row. But that’s all I had to do to solve the issue, run that command. That’s ease of use for you, as far as I’m concerned. If you try to install anything with a conflict, the install will fail telling you to run pkgdb -F.

The other is UPDATING. In /usr/ports, there’s a text file called UPDATING. This contains useful information on known security issues, problems with certain installs. Recently, FreeBSD moved to xorg 7.2, and there’s a whole list in UPDATING on how to update to 7.2 using ports. It’s a pain having to follow the steps, but that’s all you have to do. Just follow the UPDATING steps and you’ll be fine. FOLLOW ALL OF THEM.

And that’s that.

This upgrades all your installed ports to be the latest and greatest. It’s the tools that automate everything that make ports so amazing to me. That and knowing that every package is there specifically for my machine.

But I Need It In A Hurry

This was the issue with TeXMacs. I needed it fast. So, what did I do? I cded into /usr/ports/editors/texmacs && make

This started making the binary in the background.

Then I crossed my fingers, prayed, and did a pkg_add -r texmacs.

This installed hte binary which, fortunately, didn’t clash with my system. I used the binary until the port was built (ports are usually fast, incidentally, just not in the case of larger installs) and then did a:

cd /usr/ports/editors/texmacs && make deinstall && make install && make clean

So I deinstalled the pkg, which ports is smart enough to do, then installed my already built binary and cleaned up afterwards.

I’ve ran into versioning issues with pkg_add, but never with ports. And it’s never hard to keep up to date with ports. To find a port to install, simply:

cd /usr/ports

make search name=pkgname

So to locate the latest version of python:

make search name=python | less

Bunches of stuff flies past the screen, so pipe it to less so you can read at your leisure. You’ll find it.

Ports is very very comprehensive. It seems to have, for example, the entire CPAN library for Perl. It doesn’t have all the python packages I’d like, but it has a lot of them. If you find something that ports doesn’t have, consider making a port and submitting it: http://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/ is the documentation for how to make a port.

Now that you’ve solved your package maintenance issues, in the future we’ll consider how to make your base distribution up to snuff with the current patches.

P.S I’m Security Paranoid

For those amongst us who TRUST NOTHING, consider the port portaudit port. It’s in /usr/ports/ports-mgmt — go ahead and cd into the portaudit directory and make install clean.

It will email your administrator email address with the results of a portaudit scan against the latest portaudit database every night, telling you about known security issues with anything and how to resolve those issues if any resolution is available. Damn useful! For more information, read the handbook page on portaudit.

Advertisements

15 Comments »

  1. Good explanation but I just need to make one point.

    The FreeBSD project needs to figure out how to separate anything requiring X11 and anything not requiring X11. I love compiling pine, mutt, apache, python, perl, etc. from source. I don’t like compiling gnome, Xorg, or openoffice at all. It would be nice if I can just update the GUI tools via binary updates (who cares how they are compiled or what is compiled with them) while being able to specify how I want Postfix or Apache to be compiled.

    Comment by Don T. Bothers — July 26, 2007 @ 9:29 pm

  2. The other option would be to use the ‘make package’ build option. This is useful if you have several machines similar machines: build on one, and do binary installations on the rest.

    There’s also ‘portupgrade’ which allows you to take an already installed port and create a package out of it. This is useful if you’re upgrading from one version to another, and want to keep a copy of the old one around Just in Case(tm).

    Comment by David Magda — July 26, 2007 @ 9:50 pm

  3. That’s the interesting, especially then you could just open Handbook, chapter 4: “Installing Applications: Packages and Ports”. No needs in Guru. No needs in Linux. Just Read This Fucking Manual.

    BTW, I recommend you use port-managment-tools, after installing it you may install any port by command “portinstall PORTNAME”.

    Comment by Dennis Yusupoff — July 27, 2007 @ 9:34 am

  4. Very well described. I find that the biggest hurdle in intrudocing FreeBSD to techs is ports vs. pkgsrc.

    Comment by Christopher — July 27, 2007 @ 1:30 pm

  5. About disabling X11 ports use WITHOUT_X11 KNOBS.

    Comment by Janos Mohacsi — July 27, 2007 @ 1:33 pm

  6. […] that the one accessed via pkg_add. If you want to know more about the pro’s and con’s this article makes some interesting […]

    Pingback by Ruminations on the Digital Realm » Archive » PC-BSD Day 4: Making a fresh start — September 9, 2007 @ 12:05 am

  7. This guide has clarified all sorts of stuff I didn’t understand about FreeBSD. 😀

    Comment by pstatic — January 23, 2009 @ 1:26 am

  8. Great guide, no wonder my old system crashed when I periodically switched between the two.

    Comment by Jengajam2 — July 25, 2009 @ 2:38 am

  9. Appreciated

    Comment by Beginner — August 20, 2009 @ 3:39 pm

  10. Hi! I was surfing and found your blog post… nice! I love your blog. 🙂 Cheers! Sandra. R.

    Comment by sandrar — September 10, 2009 @ 3:07 pm

  11. Great article. Thanks!

    Comment by Andras Horvath — December 15, 2009 @ 3:45 pm

  12. Thank you for the article, it summarizes everything I managed to understand about FreeBSD after a day of tinkering.

    It makes sense now.

    Comment by Alex — August 26, 2011 @ 2:56 pm

  13. I really like your writing style, great information, appreciate it for posting :D.

    Comment by http://dream-analysis.org — March 20, 2012 @ 2:46 am

  14. You’ve failed to say what makes ports good though ‘they’re magically delicious in every single way’ is just a sales pitch… You list the disadvantages… 4 days to update ports, couple of hours to install a package.. but is there any advantage at all? Given that these days optimisation differences are irrelevant (since there aren’t 10 intel variants like there were a few years ago) binaries are likely to be bit for bit identical with what you’d compile anyway.. so what is the payoff that justifies it taking several days to install a base set of packages vs. an of hour or two on any other OS?

    Comment by Tony — May 6, 2012 @ 1:47 pm

  15. No choice, because some ports such as flash player can only be install in ports only. If you mixed packages with ports such as X11/Xorg, then the X11 packages will likely be backward and forward incompatible with the ports you need because packages is not available for download. I don’t want to install gnome 2 via ports because there are not many changes in it because it is unmaintained by linux. I prefer packages for the non-bleeding edge software. The fact that packages are not up to date because ports are bleeding edge and consume a lot of time to compile for other people to download. If I am looking for non-bleeding edge software, then packages will do a good job for me as it is not expected to change fast like a wild fire. In my opinion, minor security bug fixed should be backward and forward compatible with the other software that share the same dependency/library, particularly the random low level fragmentation, which is small but not modular as too much dependent affect the modularity. Perhaps a work around approach is to make the library more interdependency instead of dependency? Example of modularity is that if you want to change a tire for your car, you don’t have to change the entire car, but simply the tire, regardless of the versioning of the tire. You have the latest spare tire you want to try? You don’t have to mess up with your car engine.

    Comment by anonymous — October 4, 2012 @ 6:15 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: