BSDTalk Podcast #55: Interview with Greg 'groggy' Lehey

Original audio available at bsdtalk.blogspot.com. This document is available in other formats at http://derek.trideja.com/bsdtalk/.


BSDTalk: Hello and welcome to BSDTalk, number 55. It's Tuesday, July 18, 2006. Now onto today's interview.

Tonight on BSDTalk, we're speaking with Greg Lehey, also known as 'groggy'. So I want to welcome you to the podcast.

 

GL: Thank you.

 

BSDTalk: So to get started, why don't you introduce yourself and maybe describe some of the work that you do on the BSDs.

 

GL: Okay, that's a big question to ask. I've been around for a long time and just heading up to a third of a century in the industry, and I started off, obviously, a third of a century ago, working with mainframe computers from UNIVAC. I did a lot of moving around in the first ten years. I did some interesting stuff like satellite data processing with the German equivalent of NASA, and I started with Tandem Computers in about 1982, spent ten years with Tandem, first with their proprietary operating system, and then later on with UNIX. And around, about 1992, I decided that big companies didn't seem to know where they were going, so I became a consultant which turned out to give me plenty of time to write books. I spent most of my time in Germany. I'm originally from Australia, but I moved to Germany in 1972, and didn't come back to Australia until 1997. When I left Tandem, I'd had access to the System V source code as part of my job. One of the things I really missed about Tandem was the source code access. So I went around looking, "Where can I get a cheap copy of Unix system source code?" Well, System V wasn't cheap. I think we paid $50,000 for our single copy we had in Germany. And then I found this company who was selling BSD Unix source code for only $1,000. I thought, "This can't be too bad," so I went and bought one. And that's how I got involved first of all with BSDi, BSD386, which later became BSD/OS.

Around about that time, I was also doing some work for Novell, who wanted free software ported for System V for their UnixWare product. And I came out with a CD of about 150 different applications. You could think of it as something like a precursor to the Ports Collection. I went out looking for somebody to publish this thing in the public domain and met up with Jack Velte and Bob Bruce of Walnut Creek CD-ROM. Now, they were interesting people, and they were sponsoring this project, rather like a free version of BSD/OS which they called FreeBSD. And I looked at that, and it had this guy, Jordan Hubbard, working for them, and he sent me a tape with FreeBSD 0.9 on it. And I tried this out and I said, "Hm, that's not too bad. It's almost as good as BSD/386." But I had BSD/386, of course; I didn't change.

Now, as I mentioned earlier, being a consultant gave me plenty of time to write books. And because I'd been doing this work on the CDs, I thought, why not write a book about porting Unix software? So I approached O'Reilly and said, "How about it?" And Tim O'Reilly said, "Well, I'll give it a couple of years of good sales and then nobody will care anymore." So I wrote the book, which took me much longer than anybody except my editor expected, and finally published in late 1995. Around about this time, Jack Velte and a couple of FreeBSD people were in Germany to go the—some book fair, I forget which one it was, it might have been the Frankfurt book fair, I have this feeling it has something to do with Hannover... anyways, I showed them the drafts of my book and they said, "Hey, that's just what we want! Why don't you write a little fifty page book about how to install FreeBSD?" And I said, "Yeah, I can do that." So, we went and hacked up some stuff after dinner and they were quite happy with that.

I got the book out pretty quickly. Of course, it wasn't fifty pages, it was 200 pages by the time I was done. But that was the beginning of what would later become The Complete FreeBSD. Now, there's quite a bit of information on the website. In any case, The Complete FreeBSD turned out to be quite a success. It's been through five editions now, but I suppose the real thing is my own time is changing. I don't have time to update the book anymore, so I thought it was probably a good idea, rather than just let it die, to at least put it out in the public domain where people can get hold of it. My big hope is that people will maintain it. We need some kind of team for that, and we haven't really set that up properly yet.

That's how I got involved with the FreeBSD project. Gradually, BSDi came and kept coming back and asking more and more money for updates, and at the same time, because of the book, I decided to run FreeBSD. That's the only real reason; I would've otherwise stuck with BSD/OS. And I thought, "You know, this is crazy. What I've seen of FreeBSD—it's a pretty reliable operating system. Why should I pay BSDi when I can get the stuff for free from the FreeBSD Project?" And so I did that and moved, I suppose more or less full time—and, I did a gradual move from BSD/OS to FreeBSD in the timeframe of about 1996-1998. And by 1998, I thought there was no point in buying any more updates for BSD/OS. So I became... only a BSD... only a FreeBSD user.

Around about that time also, I got involved, I was doing some work with teaching in China for Tandem. We came across this product the Veritas Volume Manager, which I thought was quite nice. A little bit later, I got a call from somebody in California, Nicole Harrington, who was working for a company whose name I forget... and she said, told me that they'd had a disk crash, and they couldn't get the system back up and running. It was running a product called ccd, which I think it is still available in FreeBSD—the Concatenated Disk Driver. And it wasn't very good at the time, and it was really no reflection on Nicole that she couldn't get the thing up and running because basically it wasn't really designed to handle disk failures very well.

So I thought, what we really need here is something like Veritas for FreeBSD. So I set to writing the thing which—I used the Latin pun to call it 'vinum' which means wine. And round about that time also, a company called Cybernet came looking for a RAID5 implementation, so they paid me a certain amount of money to do that, and the result was the Vinum volume manager, which was introduced into the FreeBSD source tree in late 1998. Now, Vinum has since been replaced by various GEOM classes, but I think in some cases, GEOM still hasn't quite got all the features that Vinum had even five years ago. It'd be nice to see people incorporate some of them into the more traditional GEOM classes.

Now, after Vinum—in parallel with the work I'm always doing on the FreeBSD project, of course I have to earn money, and FreeBSD's not really a very good way to earn money. Some people managed to do it but I wasn't one of them. But in about late 1999, a guy called Dan Shearer who's not really well-known—unjustifiably, I think; Dan is a real fountain of ideas—Dan Shearer got a whole lot of Australian open source hackers together to form the Australian branch of a company called LinuxCare. LinuxCare, I think, still is in San Francisco. At the time they were living on venture capital in the middle of the dot-com boom. That's probably one of the better examples of how to ride the wave, I suppose. The wave came down and we all moved on, but at that time, I was working for LinuxCare as "the BSD person". LinuxCare really didn't have any idea what to do with BSD, but they knew that some companies were using it, and that I was well-known, and so I should be the BSD person.

So I spent the best part of a year working for LinuxCare without really doing any work. And about the same time as all of this happened, we were still recoiling from the reaction to the Mindcraft benchmarks in 1999, where Microsoft showed that they could indeed outperform Linux in some networking cases. And we looked at that and thought, "If that could happen, BSD wouldn't put on a very good picture there either," so we kept quite quiet. About a year later, we finally got together to wonder what to do about the issue. The real problem was the way both Linux and the BSDs handled multiprocessors. So we had a meeting in the Yahoo! complex in June 2000 where we discussed what we decided to call SMPng, the new generation SMP support.

I got involved in that from the beginning and spent about six months working on the interrupt handlers, and that was then taken over mainly by John Baldwin. The SMP project has now officially been declared closed, only five years after it started. We expected it to take a long time—about two years—but you know how these things work out.

So after LinuxCare deflated and the Australian branch closed down, most of us moved on to IBM where I was working on Linux. And IBM's a great company to work for if you fit into the IBM mold. And I enjoyed working for IBM, but it was fairly clear that IBM's way of working was not for me, and in fact, it started from the fact that I was working from home, which is not really a thing that IBM's very much into. So I moved on from there and did some more consulting before working for a local Adelaide company which is in storage systems. One of the things I didn't like about that company—they're nice people—but it's very much proprietary technology, so I can't tell you anything about it.

 

BSDTalk: [laughs]

 

GL: And so about a year ago, I left them and moved onto MySQL, who need no introduction, and it's definitely not proprietary technology.

 

BSDTalk: So currently you are working at MySQL. Are you working on MySQL and FreeBSD or just specifically on MySQL?

 

GL: My job title, I suppose, is senior software engineer for online backup, but MySQL, of course, runs on FreeBSD and a lot of my work is involved in insuring that MySQL runs well on FreeBSD. Not only FreeBSD, by the way. I'm involved mainly with the FreeBSD project but I'm also a developer on the NetBSD project even if I don't really get very much work done. One of my interests is to insure that the BSDs don't fragment too much. Or I try to insure that. Getting that actually done it more difficult even than it sounds, but we should be promoting all of the BSDs, not just FreeBSD, not just NetBSD, not just MyBSD. So in general, if there's any BSD question within MySQL, I usually get involved with it.

 

BSDTalk: Do you think that there is more fragmentation going on in recent years, or do you think that it's holding steady?

 

GL: I think it's getting better. It may not be getting very much better, but it's getting better. We have more communication, I think, between the individual groups than we used to. There seem to be fewer personal differences, also, than there used to be.

BSDTalk: So are there any other current projects within FreeBSD that you're working on or is it mostly just the MySQL work?

 

GL: At the moment, MySQL is keeping me very busy, so I stay in touch with the FreeBSD project, I do minor changes from time to time, but I don't really have any time to get involved in major projects in that environment.

 

BSDTalk: Do you get to play with or benchmark any large installations of MySQL on FreeBSD?

 

GL: I don't do any benchmarking. I do do a lot of bug chasing, and we have a number of bugs which seem to happen only on very large installations. I mentioned Nicole Harrington earlier in connection with a crash on ccd. It turns out that she's also involved with one bug which I've had open now for over six months. It's a very large database, very busy database with image data, so also large rows. And from time to time, they have crashes which look to me like some kind of race condition, and we really can't reproduce this thing at all in our environment, and the company, of course, is somewhat reluctant to allows us to test in their environment because it only really happens on a live database. And we're really scratching our heads about this one.

If anybody who's listening thinks this might sound familiar, they should look at MySQL bug #12187. And if it's anything like what you're experiencing, please contact me. We're really a little bit at a loss with this particular one. In general, though, from my own work with developing software rather than doing performance testing, so, databases we work with are relatively small.

 

BSDTalk: Now, besides developing software as a career, do you also use FreeBSD in your personal life and on your personal workstations?

 

GL: Yes. I haven't, well, I do have a machine here, 'round behind my left shoulder which runs Microsoft [Windows]. That's more out of protest because there are so many people who send me Microsoft-encoded mail, and I have to be able to read it somehow. But I've been using BSD on the desktop since 1992. As I mentioned earlier, it started off being BSD/OS and gradually migrated to being FreeBSD. I currently on my desk have one, two, three... computers running FreeBSD, and my wife in the next room has another machine running FreeBSD. I have a machine running the TV projector, also running FreeBSD. I have another machine running my beer brewing which also runs FreeBSD. I also have a laptop for work which runs Linux. Both the Linux machine and the Microsoft machine, both laptops are dual boot with FreeBSD.

 

BSDTalk: And do you run all on the i386 architecture or do you use other architectures?

 

GL: Currently I'm running only on i386. I did get given a Sun Enterprise 4500 8 CPU SPARC machine a couple of months ago, but that's currently waiting for me in Silicon Valley and I'm trying to find a way of getting it to Australia without paying an arm and a leg for it. I've also in the past run on other SPARC architectures. I've had a couple of PPC machines but I didn't run FreeBSD on them.

 

BSDTalk: And what about your desktop environment? Are you all console or do you run a graphical environment?

 

GL: No, I run X. I've been running X right, since from the very beginning, when I first got BSD/OS 12 years ago, 14 years ago. One of the first things I did was to insure that it ran X properly. I've currently got in front of me six monitors. One machine is running three of them, another machine is running two of them. The sixth monitor I put in yesterday, and is also on the desk behind there ... a graphics card which will go into the second machine. So when I finish here on this interview, I'm going to go take that, of course get another monitor, and by this evening I hope I shall have seven monitors spread around me. They're all running X. The two machines are joined together with a little program called x2x—that's digit two. So the entire array of monitors is controlled from one mouse and one keyboard. I've got a web page on this on my website, but that of course is not completely up to date, and it will certainly be out of date by this evening.

 

BSDTalk: And what window manager do you use?

 

GL: I'm currently using FVWM2. I started off in the early days—I forgot what window manager, it might have been FVWM actually—but I purchased MWM, the Motif Window Manager, sometime in about 1993. I used that basically until the object file formats changed enough that it was really becoming impractical. And I found that FVWM2 gave me the closest approximation to MWM. I've been interested in finding other window managers, but there are very few which really handle multiple screens well. In particular, on my Linux box, I'm currently running Fedora Core—is it 5 that's the latest one?—anyways, the most recent version of Fedora, and with the GNOME desktop. And although I can see a lot of very good things in GNOME, I found it very restrictive in the way it's used. It takes up too much space on the desktop, tries to make me do things its way instead of helping me do the things I want. So for the foreseeable future, I'll probably continue with FVWM2, although I'm very much open to suggestions what else I might use.

 

BSDTalk: Yeah, I think that's a common complaint among power users of the GNOME desktop.

 

GL: That's interesting to know because I, I know quite a large number, Linux people of course, from my time at LinuxCare, and most of them used either GNOME or KDE. There were very few who used a window manager as opposed to a desktop environment.

 

BSDTalk: Another question that I always like to ask is your text editor of choice.

 

GL: [laughs] There can only be one!

 

BSDTalk: [laughs]

 

GL: Uhm, I've been using emacs since 1980, and I didn't find out about vi until some years after I'd been, uh, had emacs wired into my fingers. And I'm currently, actually spending quite a bit of time reading about all the things that have happened to emacs since I wrote my original macros about 12 years ago, and found that about 8, about 10 years ago, I got a book called 'GNU EMacs Extensions' from O'Reilly, written by Bob Glickstein, and I find it's a wonderful book to read before going to sleep, but it doesn't completely put me to sleep In fact I find it easier to read than a number of novels. A very useful book, and it shows there's still a lot of power left in emacs that I haven't even found out about yet.

 

BSDTalk: Unfortunately, my mind is too simple for emacs, so, it, uh...

 

GL: Your mind isn't too simple, it's just a question of what you're used to, I think. Dr. Dobb's did this famous article on baby ducks... must be twenty years ago now, where, I was a subscriber to Dr. Dobb's at the time, and they had this picture of a freshly-hatched duckling, and like many birds, when a duckling hatches, the first thing it looks at, first moving thing it looks at is treated as a parent. So, if it's a human, then you have a duckling following it around for the rest of its life. And the idea there was that the, that editors are like that. You start off with an editor, and the first editor which you can really use becomes your editor of choice and it's almost impossible to change. Now, in my case, it had ... I'd been using numerous editors before emacs came along, but emacs blew the rest of them away.

There are some really funny things about emacs that turn people off, like Stallman's insistence on backspace brings up a help screen...

 

BSDTalk: [laughs]

 

GL: ...and that is just really bizarre. But I've, I've put complete beginners at an emacs screen after fixing the backspace problem, and they found it quite easy to use. Whether the, uh, cursor keys all work the way you'd expect, everything basically works the way you'd expect, the thing has menus... the only thing they don't know is the power which they're missing by not knowing the details.

 

BSDTalk: Plus the added benefit of the Towers of Hanoi.

 

GL: Oh yes, well, actually, the Towers of Hanoi are not nearly as interesting as Doctor.

 

BSDTalk: [laughs] Yes... among all these questions, if we kind of circle back to an earlier part of the interview regarding books and placing them into the public domain, now that you've put those works out to the world, are you planning on writing on any new pieces of documentation or books?

 

GL: At one stage, I was going to say, "I think my book writing days are over," and that may very well be the case. But I've also been very careful about saying I won't do this anymore, or I won't do that anymore. There is a very good chance that I will write another book, I just don't have any plans at the moment. As I mentioned earlier, I would really like to get a team together to maintain the complete FreeBSD as a reference book or as an introductory book on FreeBSD. And if anybody's listening that would, has the necessary skills, and the will to maintain a book written entirely in groff, then please contact me.

 

BSDTalk: Well, are there any other topics that you want to speak about tonight? Or, I guess, for you, this morning...

 

GL: Yes, this morning indeed... uhm... no, I don't think there's anything... I could rant on numerous subjects, of course, but I don't think that there's anything that the audience would like to here.

 

BSDTalk: [laughs] I'm sure the audience will have to forgive me because there are probably plenty of questions I wish I had asked. But I guess we'll end it there and I just want to thank you for being generous with your time.

 

GL: You're welcome, thank you for the call.

 

BSDTalk: If you'd like to leave comments on the website or reach the show archives, you can find them at bsdtalk.blogspot.com, or if you'd like to send me an e-mail, you can reach me at bitgeist@yahoo.com. Also, I'd like to, once again, thank the BSD team for the use of the opening music. Thank you for listening and this has been BSDTalk number 55.


[transcribed 03-Sep-2006 by Derek Warren - derek@trideja.com - http://derek.trideja.com/]