oscartheduckin’ around

October 13, 2008

remote port building issue #1

Filed under: freebsd — Tags: , — oscartheduck @ 4:28 pm

So, I have some ruby and some shell. I think the shell is doing better than the ruby, but I can also see where teh ruby is coming in.

I have this vague design in mind:

1. locally, you state that you want to perform installation of port X.
2. a list is built which is rsynced over to the remote machine
3. the remote machine has a daemon listening which can build the packages — yeah, there’s a good chance of a security hole here, but that’s recognised.
4. build the software as packages remotely
5. notify the machine that its build is ready — some mechanism to acknowledge this and allow for the reverse rsync automagically will be nice.
6. install the packages

Now, see the problem? Libraries. Ports are nefarious beasts; the machines need to perfectly in sync to do this easily, which is doable with Jails, but.

I don’t want jails to have to be the solution. Building the packages statically would help out, but inflate things.

I need to work out what changes to the ports infrastructure would make this easier. Any ports ninjas reading this?

Advertisements

October 12, 2008

decisions, decisions, decisions

Filed under: freebsd, ruby — Tags: , , , — oscartheduck @ 8:04 pm

I’ve been thinking about writing a new ports installation tool which solves a few problems I’ve always thought would be neat to solve. Thus:

1. decision 1 — write a new ports tool or extend an existing one?

This is really hard. I’d write a new ports tool in ruby or shell. Guess what the two existing main ports tools are written in? That’s right, ruby and shell! So I just sit here thinking about the wisdom of potentially rewriting a lot of functionality. Especially with portupgrade being written by a hacker who’s ten times the programmer I am, and portmaster being, well, fucking fantastic.

Here’s some of my aims:

1. The ability to instruct a remote machine to build every port needed
for a local upgrade of ports as a packaged to be pkg_added.
2. The ability to cross compile everything. This means that a local
arm board can upgrade its onboard dns server by having a remote
xeon based computer build the port.
3. The ability to run the command without super user privileges. I
want to have a directory I have write access to that my ports
can build to and be able to do an upgrade of those ports
automatically.
4. The ability to synchronise ports across machines. This should be
architecture independent, and should take into account that
ports can with this tool be installed into home.

Which leads to decision 2:

2. How do I technologically best accomplish this?

Which is the same as decision one, really.

There are so many decisions. How do you search for a port when someone wants to install it? The other two tools I usually use for this stuff answer this question by saying “we don’t”.

So, accepting my premise that I’m not as good as these other folks at solving this issue, I won’t either.

A second: how do you determine if a port is to be upgraded? Well, there are two options I see:
1. roll my own method of doing so
2. use someone else’s method of doing so

I prefer the idea of number 2. Software is something that should be built modularly, so that each piece can be upgraded independently. And if I just use someone else’s tool, then I inherit their improvements.

portupgrade looks like the tool I shoould be thinking about, but I don’t want to follow their design decisions. I instinctively think portmaster has the right idea with not using an external database; maintaining pkgdb is annoying, irritating, and a whole host of other words which mean the same damn thing. There are a host of poorly documented prompts when repairing the database; stale dependency, eh? Great! What does it MEAN? I’ve always _assumed_ it was a reference to the idea that a dependency listed in the database is no longer needed, or somesuch thing; googling around can tell you, I’m sure, but the man page for pkgdb doesn’t, and if it ain’t in the man page then it’s undocumented. And that means that yes, I do consider man man to be inadequate because it tells you there a re other sections of the manual but doesn’t tell you _what_they_are_. I know what they are, they’re historically significant and everyone should know them, but man man should list them too.

I’m getting dangerously close to ranting, when I’m actually trying to document something. The aim here is to give people who’ve never written this sort of tool an idea of how the thought process goes. It’s all decisions, and no decision is correct.

June 30, 2008

Eleven. Megabytes. Suckers.

Filed under: freebsd — oscartheduck @ 1:43 am

I emailed my mentor yesterday night with the news that I now have FreeBSD available in 11meg.

And I can shrink it further.

There’s really a simple objection to this statement, though. It is implied by this: “James, have you tested that thing yet?” and the answer is, “OHGOTOHELL”. Evidence of the cruftiness of this thing is available in the fact that this tiny 11 meg of nothing sits in a whopping great big 493 meg disk image that I haven’t shrunk down yet.

I imagine I’ve screwed /etc/rc up, too. But that thing’s tiny, really. It’ll be simple enough.

Where am I going from here:

1. verify the functionality. Yeah, I have to do this still. Once I can boot to the command prompt, I’ll be happy.

2. once I have that functionality, evaluate needed binaries and crunchgen ’em. I think of this in a vague way as returning TinyBSD to its PicoBSD roots, though this shouldn’t be read as my hubristically stating I’m as awesome as the TinyBSD developers or PicoBSD.

3. cross compile for ARM. The aim for this has been from the start to build it for non-x86 chips. ARM especially is so huge right now it makes sense as a target.

In a moment of blog type reminiscence, one of my friends was a huge Acorn fan, so working with ARM feels like looking back on my past somehow.

I told my mentor that I felt peculiar about this project because it’s so clearly not the same as many other projects, and as he has many times already he gave me the right advice. This isn’t a development down-on-the-ground-pointing-pointers kind of project, it’s a 10 000 foot view project. I feel daft because it’s not as down in the works as I’d like sometimes, but I feel good because I look at what I’m doing and it’s basically using these fantastic tools that other developers have spent their time making.

Thus far, TinyBSD is the main tool I’ve been working with. Those folks have making a small OS down to such a simple tool, it’s hard for me to imagine doing this without it.

June 7, 2008

booting to pxe

Filed under: freebsd — oscartheduck @ 6:52 pm

My initial plan was to only use compact flash, but I have to buy a card. So for your viewing pleasure, the initial boot screen. PXE server appearing on my laptop in a couple of hours.

U-Boot 1.1.5 (Nov  2 2006 – 10:31:07)

DRAM:  64 MB
NAND:  NAND device: Manufacturer ID: 0xec, Chip ID: 0xda (<NULL> NAND 256MiB 3,3V 8-bit)
256 MiB
DataFlash:AT45DB642
Nb pages:   8192
Page Size:   1056
Size= 8650752 bytes
Logical address: 0xD0000000
Area 0:x09D0000000 to D0003FFF (RO)
Area 1:x09D0004000 to D0007FFF
Area 2:x09D0008000 to D0037FFF (RO)
Area 3:x09D0038000 to D083FFFF
In:    serial
Out:   serial
Err:   serial
DM9161A PHY Detected
No link
MAC: error during RMII initialization
Hit any key to stop autoboot:  3 x08x08x08 2 x08x08x08 1 x08x08x08 0
DM9161A PHY Detected
No link
MAC: error during RMII initialization
BOOTP broadcast 1

June 1, 2008

Where’s the damn GSOC Code, dude?

Filed under: Uncategorized — Tags: , — oscartheduck @ 11:51 pm

I was talking to my mentor the other day about having committing code because, let’s face it, FreeBSD has some of the world’s greatest coders writing for it. And I’m terrified and nervous. Because I write crappy code, I know I write crappy code, I’m comfortable writing crappy code, because I never distribute my crappy code.

This last part changed with GSOC. I have to have people able to read my code, I have to distribute the code, all that stuff. And the official SCM is Perforce, but I’m not a huge fan so far. I much prefer git.

So this is the git URL for the code:

git://github.com/jamesh/shinybsd.git

It’s called shinybsd; feel free to track it. If you want to actually commit to the repo, let me know and I’ll help you get set up. Be aware that you’ll need an account on github.com

I’ll be syncing this back up semi-regularly with perforce, so no worries if that’s what you want to watch.

May 9, 2008

code -> pdf

Filed under: how to — Tags: , , — oscartheduck @ 2:59 pm

I’m on my way out right now, but I’m going to write up a script tonight called code2pdf, and wanted to write down the history of it before it leaves my head.

I have been taking a CS class this semester, and have been writing code as a consequence. The requirements for submitting this code are twofold:

1. pop all code and a scriptfile proving the code works on a USB stick

2. print out copies of all code and the scriptfile and submit with above.

Now, most of the folks are using some kind of windows kludge to write their code, so they were taking screenshots of command windows and handing that in as a script file. I just used script, because I figured that’s what it’s there for. The problems were twofold:

1. I don’t have a printer in my house.

2. The printers at school are all connected to windows boxes.

So when I come in with a nice raw text file and attempt to print it, it looks _awful_. Same goes for code; my text files looked horrible.

But my instructor is a unix geek, so she didn’t mind at first. However, the driver being used to run the printer has a quirk in it; it has its own definition of what a tab is. I couldn’t believe it; my nice sensible “two spaces to a tab” code was running and running and running over line after line of blank space. Which didn’t worry me, but it made trying to read the code a horrible task for the instructor. True to her form, she mildly pointed this out to me and asked me to work on the issue, no fuss just a pleasant request. Which is the best way to get results.

So, I started thinking about it. First things first, I stopped using tabs and started using spaces. The printouts looked a lot better from that alone.

But I decided not to stop there; my instructor had been polite, and I have the power of FreeBSD available. I installed OpenOffice from the port (which took ages), after which my hard drive killed itself. So I installed anew and installed OpenOffice again, which took forever again. And I copied my code into OpenOffice and output it as a .pdf.

.pdf rules. I format things nicely, go through the file and make sure everything’s pretty and I’m happy as a clam at high tide. The script file was more difficult than the code, funnily enough, as openoffice is smart enough to read the .txt extension, scan the interior and then kindly offer to format everything as a spreadsheet.

Remove the extension and open the file again, and a different preprocessor asks for advice. I told it to use luxi mono as the font, and everything looks pretty. From there to .pdf is as simple as using the filter.

But why stop there? With the ports system at my fingertips, I can do anything! And I don’t want to start up openoffice just to spew out a .pdf file, more importantly. After a couple of creative searches, I found  a beautiful program called highlight, which takes code in and spews out one of several forms, HTML, rtf, TeX, LaTeX, XML, a few others.

I export everything to .rtf and then open it in openoffice, just to take a look. It looks great; colour code highlighting, the works. Export to pdf, to make sure I have it in time for my deadline, and then get back to work.

I export everything as TeX, then use pdftex and it comes out as a gorgeous looking pdf, but only in greyscale. I wanted to hand in colour, so I didn’t worry about this and instead used openoffice, but tonight I’ll be looking for a .rtf to .pdf filter on the command line so I can deal with everything and get colour output.

Whether I find it or not, I’ll write up a quick script to take in code and output a greyscale pdf file. Then I’ll investigate script files, which I imagine will be as simple as txt2pdf or something named similarly. Then I’ll submit it as a port; it’ll be a front end to other tools, but I think the ability to ensure my code looked gorgeous from any operating system and printed out correctly was sufficiently useful that other students would like the same thing. And lo! It will be called something witty.

I’m thinking it should be a simple thing that offers flags to output colour or greyscale pdfs (eventually), and also can handle script files. Pretty code all day long.

May 3, 2008

ca-ca-catching up

Filed under: Uncategorized — Tags: , , — oscartheduck @ 4:39 am

I did something today that I’ve never done before. I decided that I was going to see how far that FreeBSD and gnome integrate with each other. Whenever I install FreeBSD, I usually run it as a CLI os. I install a window manager, but it’s not there as anything really other than support for multiple terminals. I use ubuntu for a nice pretty OS with integration between all the disparate elements, and FreeBSD for real work.

So I started following the instructions from the FreeBSD Gnome folks for auto-mounting this and pleasant integration that. In the process, amazing things happened. Screensavers started working brilliantly, icons that were missing showed up out of nowhere, and I started to see the truth of a friend of mine’s observation to me two years ago that FreeBSD and Gnome are exceptionally well integrated with each other.

I almost feel like writing a port that’s just a shell script to execute the relevant bits and pieces to make FreeBSD shine with Gnome; it’s amazing that all this time I could have had this on the desktop and instead was using linux.

I privately noted an interesting thing today. I started using the _excellent_ portmaster tool to solve ports issues instead of portupgrade recently, and out of nowhere the amount of circular dependencies registered in pkgb skyrocketed. It’s probably merely a coincidence, but still.

trap just became my new best friend: http://docs.hp.com/en/B2355-90046/ch23s07.html

I’ve been following the code for NanoBSD of late; it’s extremely clearly written and provides excellent examples of shell code. Well worth a look.

I’ve also been following my interests down the path to the source code of common utilities. What’s neat is the little things I hadn’t thought of. For example, from boot0cfg, in the case statement that reads argv, Robert Nordier sets a variable to equal 1:

case 'f':
fpath = optarg;
break;

Then,

/* save the existing MBR if we are asked to do so */
if (fpath)
write_mbr(fpath, O_CREAT | O_TRUNC, mbr, mbr_size);

Robert directly tests for truth on the fpath variable. I’ve always called my variables something like fpath_test to remind myself that they’re essentially booleans; however, looking at this I’m suddenly hit by the “Oh, there’s no real need to do what I do, and it might sacrifice a little clarity”.

This suddenly makes me wonder: would there be any performance impact of using boolean types here? I can’t imagine there would be, as I assume the boole types are little more than static constants.

I noticed this little gem in the code for boot0cfg, too:

opttbl[] = {
{"packet", 0},
{"update", 1},
{"setdrv", 0}
};

Is it just me, or is that sucker a dictionary/hash?

I need to include the following to satisfy the license:

/*
* Copyright (c) 1999 Robert Nordier
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS“AS IS” AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS
* BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
* OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
* OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
* EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

April 30, 2008

Reading code

Filed under: freebsd, Uncategorized — oscartheduck @ 1:58 am

Today, I finally got to really start a beginning on my embedded BSD proposal; I started reading NanoBSD and TinyBSD, to see what they’re doing.

I love reading other coders’ code, because it’s always a learning experience. Poul-Henning Kamp, for example, writes very nice shell script. Very nice indeed.

As I am about to redistribute his code a little, let me include this :

<—-snip—->

# Copyright (c) 2005 Poul-Henning Kamp.
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS “AS IS” AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
# $FreeBSD: src/tools/tools/nanobsd/nanobsd.sh,v 1.30 2008/01/12 22:58:06 simon Exp $

<—-/snip—->

Now that that’s over with, a little code is in order:

<—-snip—->

# Name of this NanoBSD build. (Used to construct workdir names)
NANO_NAME=full

# Source tree directory
NANO_SRC=/usr/src

# Where nanobsd additional files live under the source tree
NANO_TOOLS=tools/tools/nanobsd

# Where cust_pkg() finds packages to install
NANO_PACKAGE_DIR=${NANO_SRC}/${NANO_TOOLS}/Pkg

# Object tree directory
# default is subdir of /usr/obj
# XXX: MAKEOBJDIRPREFIX handling… ?
#NANO_OBJ=””

# The directory to put the final images
# default is ${NANO_OBJ}
#NANO_DISKIMGDIR=””

<—-/snip—->

See what he did there? He’s declaring variables. Every one of them is prefixed with a comment that says “This next variable means this”. I swear, I started reading this and admired the time put in simply to make the code easier to follow.

I was a little worried, though. It looks wordy. But I got down to the code:

<—-snip—->
###########################################
#
# The functions which do the real work.
# Can be overridden from the config file(s)
#
#######################################################################

clean_build ( ) (
echo “## Clean and create object directory (${MAKEOBJDIRPREFIX})”

if rm -rf ${MAKEOBJDIRPREFIX} > /dev/null 2>&1 ; then
true
else
chflags -R noschg ${MAKEOBJDIRPREFIX}
rm -rf ${MAKEOBJDIRPREFIX}
fi
mkdir -p ${MAKEOBJDIRPREFIX}
printenv > ${MAKEOBJDIRPREFIX}/_.env
)
make_conf_build ( ) (
echo “## Construct build make.conf ($NANO_MAKE_CONF)”

echo “${CONF_WORLD}” > ${NANO_MAKE_CONF}
echo “${CONF_BUILD}” >> ${NANO_MAKE_CONF}
)

build_world ( ) (
echo “## run buildworld”
echo “### log: ${MAKEOBJDIRPREFIX}/_.bw”

cd ${NANO_SRC}
${NANO_PMAKE} __MAKE_CONF=${NANO_MAKE_CONF} buildworld \
> ${MAKEOBJDIRPREFIX}/_.bw 2>&1
)
<—-/snip—->

A note which tells us “variables are over now, busters”, then straight into code.

It’s a good example of how to write a shell script. There’s enough information to get one oriented, but after that you’re on your own. And let’s face it, if you can’t work this stuff out reading it then no number of comments are going to help.

In reading it, I ran across a couple of programs I’ve never used. I figured, though, what better time to get familiar with them than by also reading their source code? So I gleefully started wandering through my source tree, discovering what there is to discover there. And as ever, I was impressed by how simple a lot of programs are.

chflags, for example, is basically two case statements.

I love usage statements. This code is covered under the same copyright as above, though I cannot see its author listed in the file. Author, please let me know who you are!

void
usage(void)
{
(void)fprintf(stderr,
“usage: chflags [-h] [-R [-H | -L | -P]] flags file …\n”);
exit(1);
}

My favourite program in the source tree right now is hostname; I remember the first time I saw it being struck by how well the various pieces of the operating system fall together to form a cohesive whole. It features a case statement that becomes a cascading case statement. The logic of the program is tiny: In substance it reads:

if (*argv) {
if (sethostname(*argv, (int)strlen(*argv)))
err(1, “sethostname”);
} else {
if (gethostname(hostname, (int)sizeof(hostname)))
err(1, “gethostname”);
if (sflag) {
p = strchr(hostname, ‘.’);
if (p != NULL)
*p = ”;
}
(void)printf(“%s\n”, hostname);
}
I love it for several reasons. It’s simple, it taught me a lot about unix, it’s just plain awesome.

For example, sethostname? Where is that function? Well, it’s declared as: extern int sethostname(char *, int);

Where? Well, use unix to tell you! cd /usr/src && grep -R sethostname *. /usr/src/contrib/gcc/sys-protos.h

Ach, what’m I talkin’ about, anyway. I just like greppin’ code and seeing what there is to be found.

April 21, 2008

GSOC

Filed under: Uncategorized — oscartheduck @ 10:28 pm

http://code.google.com/soc/2008/freebsd/appinfo.html?csaid=AB2F0224F1C4AD72

Yeah. That’s me.

Awesome.

April 9, 2008

person.glow(oscar)

Filed under: Uncategorized — oscartheduck @ 2:10 am

I’ve been writing lots of OOP code of late, after a long stint writing functional programming code, and I’m in this weird space.

On the one hand, it’s interesting writing the code and seeing all kinds of magic appear to happen with almost nothing done. On the other hand, I don’t trust objects to keep track of themselves.

What’s a person to do in this situation? It’s especially bad with java; I don’t expect it to be able to do anything as a language because of its uniformly weird implementation. Having default types which override the types that a programmer enters in the code is _just_plain_wrong_. But java’s also where the employment is right now.

I much prefer ruby. Open classes, isThisBoolean?, uniformly object oriented. It’s plain old gorgeousness on stilts.

For those of you out there clustering away, consider this: to reimage a bunch of nodes without too much hassle, unplug your dhcp server from a switch, plug in a freshly configured SystemImager master box and copy of the dhcpd.conf file, bob’s your uncle you’ve got instant reconfiguration of all nodes. Plug the old dhcp server back in and everything’s ready to rock and roll.

« Newer PostsOlder Posts »

Blog at WordPress.com.