Original audio available at bsdtalk.blogspot.com. This document is available in other formats at http://derek.trideja.com/bsdtalk/.
BSDTalk: Hello and welcome to BSDTalk, number 48. It's Thursday, May 25th, 2006. In the news, DTrace is up and running on FreeBSD, and speaking of FreeBSD, here's an interview.
Well, today, on BSDTalk, we're speaking with Poul-Henning Kemp, and he's a FreeBSD developer, so welcome to the podcast.
PHK: Thank you.
BSDTalk: So, today, I just wanted to cover some basic questions about BSD and what you do with BSD, so perhaps you could start by describing how you're involved in the BSD projects.
PHK: My main involvement in FreeBSD is as a kernel developer, and this basically came about because I needed FreeBSD to work for various jobs I had back in the nineties. Today I work as an independent consultant and, more often than not, the work involves some kind of FreeBSD installation, or uses FreeBSD as a tool to achieve whatever goal the customer has in mind. I'm doing quite a lot of work on embedded systems, where FreeBSD goes in as merely a component of some monitoring system or server system, time server system, or something like that
I've almost spent six years on the core team of the FreeBSD project, from 1994 to 2000, and I'm happy to be out of that part of it now because I'm not really a governing body kind of person. So, today, I'm more of a developer than a politician.
BSDTalk: When did you get started using BSD or Unix in general?
PHK: Uh, Unix in general was back in the eighties, kind of in bits and pieces here and there. I worked for Commodore Computers at one point and they had a prototype computer called the Commodore 900, which was a Zilog 8000-based Unix computer with a graphical screen, black and white. Unfortunately, they had the Amiga coming up at the same time and didn't have the production and general capacity to deal with both projects, so they chose the Amiga instead of the 900, and that's why Commodore never entered the Unix business again. But that was a very interesting machine, actually.
I can see on the Internet there's a couple of them still alive here and there, but very few people know about them. Then, throughout the eighties, I worked in various companies doing Unix stuff. We had an oil company in Denmark that swapped their IBM mainframes out for Unix computers, and that was a major challenge. Unix computers are not very geared for administrative data processing, printing out 30,000 invoices kind of stuff. So, we had a lot of fun with that.
Then in the early nineties, I was mostly doing networking stuff, and when 386BSD came out, I saw that as a very convenient platform for putting up servers to do menial jobs like controlling printers or recording data from a telephone switch, and stuff like that. And from there, I ended up in the FreeBSD project with the various regroupings that happened after 386BSD came out.
BSDTalk: And why do you think BSD makes a nice platform, or why does it make a good version of Unix?
PHK: It works! [laughs] Much of my reason to go active into the development of FreeBSD in the early days was basically that the kind of commercial Unixes we had to deal with were, by today's standards, pretty crappy. For instance, many of the American Unix vendors did not realize that daylight savings time in Europe [was] different, so you couldn't use the daylight savings features of the operating systemyou'd get wrong timestamps and stuff like that.
Also, reporting bugs and getting them fixed from over in Europe was kind of like, pretty much impossible because you had to go through the local office of whatever companyUnisys, NCR, whoever wasyou know, they would have to understand the problem and they really had no idea what you were talking about, and then they would have to report the problem to some people on the other end, and more often than not, it would end up in the marketing department because that was the only telephone number they had... and, you know, three months later, they come back and tell you they can't reproduce the problem. And you're back in square one. And that was a very frustrating experience as responsible for running actual production on Unix systems. So, the feeling that "I can do better than that," was certainly a major part of it. I think we've come a long way with FreeBSD. FreeBSD now, more or less, does just work.
There's always a problem with new hardware device drivers and stuff like that, but more or less, you can take FreeBSD now as a tool for building something and not have to worry about the operating system. I have one box right now that has a 500 day uptime and there's no reason why it shouldn't continue. That's the kind of reliability and predictability I like from an operating system, and I think FreeBSD is doing really well in that area.
BSDTalk: And what projects are you currently working on within FreeBSD?
PHK: Right now I'm targeted to do the TTY system SMPng conversion, and that's a bit of a nasty piece of code because the TTY system hooks into the consoles and hooks into session management and stuff like that. So, I'm slowly beating my way through that. I still have a lot of weird ideas of what to do to the filesystem and disk I/O area, but unfortunately my time is a bit on the short side for that right now, so that's not very actually going on right now. I hope later in the year to get more time for FreeBSD and be able to do something about that [...] also.
BSDTalk: And you were saying this is TTY related to SMP?
PHK: Yeah, basically, SMP, once you have more than one CPU, you need to be very careful about not blocking the system down where only one CPU can do work. Fortunately, TTYs are not as important as they used to be. It used to be you had a VAX and you had serial terminals, and you had 200 VT120 terminals standing around or something, and these days, people will use SSH to login to the system and they'll use the console to login to the system and that's about it. But the code is still there and it needs to be updated to work on multiprocessor systems.
You can't have a system slowing down to only one CPU just because somebody logs into it. So, there's a lot of ancient code used to optimize the interrupt load on VAXes and stuff like that that can really be done without now because we have so much more CPU power and so much less TTY traffic that it could be done much simpler. I'm not sure that [...] For now, I think I'll simply concentrate on getting the locking right so that we'll have more than one CPU in the TTY code at one time.
BSDTalk: Yeah, I guess a while ago there was a big push within FreeBSD to work on SMP, and I guess some decisions to go about that lead to some people in one direction with DragonFly BSD, and the FreeBSD going in a different direction. How do you think the SMP work is going FreeBSD? How mature is it? How well does it scale, and perhaps, where does it need to improve?
PHK: I think it's going extremely well. We saw some numbers presented at BSDCan which, quite frankly, I'm very happy with. Kris Kennaway showed some filesystem scalability numbers which were very, very impressive, and the networking work is going forward really well also. Not quite finished, but it already has pretty good performance for a multiprocessing system, and as far as I know, pretty much beats the 4.x branch of FreeBSD now in most crucial benchmarks. So I'm very happy with it. It's taken five years longer than we thought it would, but there was this dotcom thing that came across us. But I'm very happy with the result, and I think the overall philosophy we chose has proven to come out right.
That's not to say it can't be done in other ways. I think the basic idea that Matt Dillon has proposed for the DragonFly project is very intriguing in a world where we'll see more and more CPU cores, but I also think it has some significant technical hurdles to cross. So, I'm sort of looking forward to seeing if he can do that. If he can square that circle, he certainly has something very interesting, but I'm not sure I would bet the FreeBSD project on that idea, so I think it's a great idea; he's taking his idea and starting a new project and can work on it and we can continue our work this way. And if it comes out, two or five years now, that he chose out to be right, so much the greater for the open source environment.
BSDTalk: And what versions of FreeBSD are you running?
PHK: [laughs] I always run the same version of FreeBSD, I run CURRENT. I'm a firm believer in eating my own dog food, so I run CURRENT and my home directory is encrypted with GBDE, and I have all the debugging features enabled in the malloc() implementation and so on. It's sort of a principle for me because if it doesn't work well enough that I can run it on my laptop, clearly something needs to be fixed and needs to be fixed fast. That's sort of my ground truth: does my laptop work? And right now it does, apart from suspend and resume. That's sort of a hit-and-miss thing. It works in some months, and then another month it doesn't work, and then it starts working again. I've never looked into ACPI and I hope I can avoid it. It doesn't look as the most stable technology to me.
BSDTalk: And are you running primarily i386 or are you running on other architectures?
PHK: To the extent I've been able to afford it, I've been trying to have all architectures represented in my lab. And that means I actually do have an Alpha machine in my lab. I have a Sparc64 machine and an AMD64 machine. I'm going to try to lay my hands on some ARM and maybe PowerPC maybe MIPS .. will go on those platforms, because when you do foundation work, infrastructure work, like the buffer cache and disk I/O or TTY code, it's important to be able to tell that it actually works on all the different platforms. Just fixing the buffer cache on 386 is not an option, it has to work on all platforms. So I like to be able to have them all in my lab and have them all running. And of course, there's also a little of just being a geek about that. Having different architectures is fun.
BSDTalk: You mentioned you did some embedded work. What architectures is that on?
PHK: That's mostly on the 386 at this point in time. I'm pretty much in love with the Soekris computers, which is a small, sufficiently PC-like [system] that you can pretty much do anything you can do on a PC, but it doesn't have graphics, it doesn't have a keyboard, and all that stuff. So, it's just a small box that uses not very much power, and you can nail it to the wall somewhere and forget it because there's no rotating parts in it. So that's my platform of choice for embedded work. I think my first embedded platform was actually a transistor-built computer many years ago.
BSDTalk: So what features of the next release of FreeBSD are you most looking forward to?
PHK: That's a good question. Because I run CURRENT, I don't think I see the releases as major things to look forward to. I was a release engineer for a couple of FreeBSD releases and pretty much all the 2.x series was my problem, so in some sense I probably still have an emotional hangup about release engineering. And because I run CURRENT, it's more of a gradual process for me; it's not like I... so now we try the new release and we get the new features. Features sort of dribble in along the road in CURRENT, so I don't know if there's anything I particularly look forward to in the next release.
There are certainly some things I look forward to getting to work better, for example, the TTY system and the VFS buffer cache system and that sort of stuff. But whether that happens in the exact next release or the next one again is pretty much not something I have very much control of. It depends on how much time is available and how hard the problem is, and whether or not you get the right idea to fix it and so on. So in that sense I'm probably a bit atypical by not really thinking about releases very much anymore.
BSDTalk: Perhaps another way to ask the question would be, if you do consult for customers, do you run your customers on CURRENT, and if you don't, are your customers perhaps looking forward to some features or fixes in future releases?
PHK: Well, in the market where I mostly work, I think what people are interested in is getting their problem solved, and the Unix or FreeBSD is just part of one of the components that goes into the solution, so I don't think they really have much awareness of it existing. For all they care, there's a box running here, it's looking after their transmitters or sending out NTP timestamps for them or something, and what goes on in the box is pretty much irrelevant as long as it fulfills the specifications and is cheap, stable, and all this stuff. So in that sense, many of my customers are not really cognizant of the fact that there's a FreeBSD machine in their solution. I mean, they know it and they get all the source code on a CD-ROM and all that stuff, but I think the place where the feature set makes most difference is where you use FreeBSD as a workstationas in, can you get your graphics and sound to work, or when you use it as a high performance server where the raw performance and network driver availability and stuff like that makes a big difference.
One of my current customers is actually an open source projectgoing to be an open source project, but right now, we're a little bit closed about itwhich is a web cache daemon for server side. Many content management systems are very slow: they have to pick the article out of the database and put ads and put this and put that in that frame and so on around it, so they're terribly slow in actually producing the content. And people will put some kind of cache in front of it like an Apache or a Squid or something like that, or maybe buy one of these horrendously expensive boxes to do it.
Basically what we're trying to do here is to write an open source, open cache daemon that is going to be as incredibly fast as we can do it, and in that case, of course, the raw network performance and the raw disk performance is very interesting. But again, in this case, this is not being targeted solely at FreeBSD, people will be able to put it on pretty much any decently Unix-like operating system if they want. So, in that case, FreeBSD is just one of the competitors as the underlying platform. But needless to say, I'm developing it on FreeBSD because that's what runs on my laptop.
BSDTalk: Will it be released under the BSD license?
PHK: I believe the license is BSD, yes, and I hope we will be able to pull in some early testers and developers over summer here. The concept is very interesting and when I talked about it at BSDCan and a lot of people took notice because there's certainly some novelty in the concept. Hopefully we'll hear a lot about this. The name of the project is Varnish, and maybe it will take off and maybe it won't. Time will tell.
BSDTalk: I'm sorry, what was the name of the project again?
PHK: Varnish. As in giving a coat of varnish.
BSDTalk: Gotcha. So, speaking of your laptop, maybe you could describe a little bit what it looks like when it turns on. What are you running for an environmentare you at the console or are you running a graphical environment?
PHK: This is going to be terribly disappointing for you. [laughing] I run X11 because it gives me the ability to have multiple terminal windows open at the same time on the same screen. I admit to running a Firefox and an IRC client, but my window manager is the old and proven CTWM. I'm not really good at running KDE and GNOME kind of things. I never know what to click on or where to click on to find things. I'm pretty much a command line kind of person, so basically for multiplexing terminals, I'm using a graphical environment.
BSDTalk: Yeah, I know, a lot of the developers that I talk to do seem to work in a similar environment where it's a minimal window manager, just so they can have xterms open...
PHK: You know what? I think it's an age thing, actually. I mean, I grew up working on 9600 terminals, and I guess I never quite got out of the habit. I tried working with an integrated development environment for an embedded platform recentlyyou know, one of these totally thing where you have the editor and the compiler and the debugger and everything on the same screen? And it was an annoying experience for me because, I mean, I had this small window with my source code, probably around 24 lines by 80, and then I had buttons and sliders and god knows what all around it on the screen. And it's like, the main thing for me is being able to see my source code, and all these buttons, I had no idea what half of them did. It was like, "How do I compile and how do I load the code" ... and forget about the rest of them. So in that sense, I guess I'm an sort of an old-fashioned developer. Of course, I would use the older-wiser correlation to claim that this is a better way to do it, but I probably have very little statistical backing for it in the real world.
BSDTalk: So you're in Denmark, it'd be interesting to hear about BSD in Europe or Denmark. Are there Denmark BSD users groups?
PHK: We have a BSD users group in Denmark, and it's to a large extent, a social thing where people will gather around a bonfire during summer and burn some burgers, or in some suitably-selected pub during winter to get some beer, and every so often we'll have a meeting with a couple of talks and stuff like that. But it's, it's a nice group and we're having a lot of fun. Next year we're doing the EuroBSDCon 2007 conference, September 14th and 15th, 2007 in Copenhagen. We'll be trying to make that a really good event to pull in people from all over Europe and some Americans, too, if we can. So that's going to take up quite a lot of activity on our part now, over the next year.
BSDTalk: And are you going to be participating or be part of the organization of that?
PHK: I think we had our first meeting about it yesterday, actually, and I think I ended up, my name ended up in three points on the to-do list, so I'll be very much involved in actually doing it. Whether I'll be doing any presentations is any good question right now. Depends whether I'll have time to have anything interesting to talking about.
BSDTalk: Are you planning to go to the EuroBSD in Italy this year?
PHK: I plan to go to all the EuroBSDCon conferences, and I also go to the BSDCan conference every year in Ottawa, and that's partly because as a senior developer in FreeBSD, I need to be able to communicate with the other developers during the developer summits, but also to talk to people about what they're using FreeBSD for and see if we're moving in the right direction. For instance, this year at BSDCan in Ottawa, there was a lot of talk about embedded systems suddenly. That's like water on my mill, but suddenly there's a lot of people saying, "Yeah, you know, we put FreeBSD into these little boxes, we put FreeBSD into these access points, we do... things we can't talk about with things we can't talk about, but the reason why we're here is FreeBSD," and stuff like that.
So there seems to be a lot of momentum building in that area now. Getting in touch with the users that way is very valuable to me as a developer to see where we should focus, what do we need to work on, what are we not explaining well enough. A lot of people come up and say, "Is there a way to do this?", and you say, "Oh yeah, we've had that for five years," and they say, "Oh." So clearly our documentation and general advocacy of features is a very important thing.
If you search the 'net, you'll practically find nothing about using FreeBSD in embedded environments and we'll have to do something about that, and explain more to people that this is actually one of the things it does really well.
BSDTalk: I haven't looked in a long time, but last time I checked there hadn't been activity in a while, but I used to mess around PicoBSD, this single-floppy BSD distribution.
PHK: Yeah, unfortunately, floppy disks have gotten rather small these days. I spent a lot of the nineties fighting to be able to install FreeBSD from a single floppy disk and then taking the rest off the 'net or something like that. I think the single floppy disk is simply now too small a target. If you need to have the ACPI code in to support your motherboard or whatever you're running on, then that pretty much takes up a megabyte on its own. So I think PicoBSD sort of fell off the radar for people as floppy disks simply became too small to do things.
The happening place these days is the Italian FreeSBIE project, which is more aimed at the running-off-a-CD-ROM kind of thing. It started out as this kind of run-only CD-ROM where you could stick it in a Windows machine, boot off the CD-ROM and you had a FreeBSD environment. It's gradually transforming into a much more general and configurable tool, so now you can build all sorts of images for all sorts of environments using the FreeSBIE toolkit. And the NanoBSD thing which I've been doing is probably, at some point, using the FreeSBIE build platform because there's no reason to have two different ones if their platform can do it.
So that's where we'll see a lot of the activity, but it still leaves a hole at the bottom of the scale where people work on various small systems, on cheap-like systems where you have, like, four megabytes of flash and two megabytes of RAM and that's it. For that kind of system, you need to have a build approach where you say, okay, I want /bin/ls, I want /bin/sh, but I don't want any of the rest of that stuff, and where you simply pick and choose the very few programs you will be able to need to run, say, a wireless access point or a firewall and stuff like that. We have a hole in that end of our build area, and that's one of the things that came up at Ottawa BSDCan. Hopefully we'll be able to do something about that.
BSDTalk: Are there any other topics you want to talk about today?
PHK: I guess if there's one thing I want to say, it's that one of the things we're horribly bad at in the FreeBSD project is actually involving our users. It takes a lot of insistence on the part of our users to be able to help us. This is probably just the accumulated bad habits of a bunch of computer code nerds who prefer to look at their source code to anything else. So, I think if there's any last things to say to our users, it's to keep nagging us. We will eventually listen to you. And, support the FreeBSD foundation if you can squeeze $1,000 out of your boss or something like that. That also helps FreeBSD a lot.
BSDTalk: Well, thank you very much for speaking with me today...
PHK: You're welcome.
BSDTalk: ...and I just want to wish you luck on your work, and thank you very much for all the effort that you've been giving to the FreeBSD project.
BSDTalk: ... if you'd like to leave comments on the website, you can reach it at bsdtalk.blogspot.com, or if you'd like to send me an e-mail, you can reach me at firstname.lastname@example.org.
[transcribed 11-Aug-2006 by Derek Warren - email@example.com - http://derek.trideja.com/]