Scott Rosenberg (scottros) Sat 24 Feb 07 08:08
I'd never heard of David Allen or his "Getting Things Done" methodology before it came up in the course of my observation of the work at OSAF. Mitch Kapor had some acquaintance with Allen, I think through his interest in personal productivity software, and he thought that some of Allen's ideas could be useful in Chandler's design. Around the same time, coincidentally (or maybe not, in a synchronicity-of-good-ideas kind of way), Merlin Mann's 43 Folders blog started up -- Mann showed up at OSAF and gave a talk about GTD, and later on Allen met with the OSAF design team. In brief, GTD suggests that you organize your life by creating a "system" -- it can be on paper or in a computer, doesn't matter -- for tracking *everything* in your life, from the smallest to-do to the biggest life-goal. In this it resembles other personal productivity systems, but it has a few methodological wrinkles and some unique philosophical traits derived from Allen's martial-arts background. You divide things up along several different axes -- different projects on the one hand, and different "contexts" (basically, locations where you can Get Things Done). And you break vague, general tasks like "start work on new Web site" down into very specific "next actions", like "call Web hosting company to compare prices" and "post ad for site designer." The "next action" concept is extremely valuable for people who get the hang of it; it routinizes the "journey of a thousand miles begins with a single step" notion into a form that can actually help you. The most useful aspect of GTD to me is the idea of clearing your head of the nagging "what am I forgetting?" feeling by letting you trust the system you've built to be your sort of outboard brain. Some other insights include stuff like, be sure you have a reliable means to defer things without losing track of them (i.e., "this becomes important on the 21st of next month but I don't want to think of it till then") and also to not lose sight of things you've delegated and are "waiting on." There's this paradox at the heart of GTD, which people like Mann are the first to admit: The whole technique is about being productive, and yet the effort to implement the technique offers a nearly infinite set of opportunities to procrastinate by tweaking your "system." This temptation proves particularly seductive to the many programmers who have flocked to the GTD method and who are constantly trading tips about the latest cool software tool for managing GTD systems. In a sense GTD is all about applying a programming systems approach to one's own work and life; programmers are used to bug trackers and trouble-ticket systems, and GTD is sort of an extension of that mindset into all realms of life. Some people find that incredibly helpful. I've fooled around with GTD myself with only partial success. I use Ecco, this ancient, semi-defunct PIM program, and it adapts beautifully to the GTD approach, but I'm not very good at "gardening" the system, and I find myself much happier just Doing Things than figuring out how to Get Things Done. It might be that I am simply a naturally productive person, or it might be that I'm slovenly and ill-disciplined! GTD turned out to be only a small piece of the Chandler saga, but I think that as Chandler adds features (the most recent version of the program shows a Dashboard view of calendar events and tasks that embodies some GTD-style ideas) the hordes of GTD devotees may latch onto it in ways that could help give it what it most needs -- a community of early adopters.
Brian Slesinsky (bslesins) Sat 24 Feb 07 09:30
That sounds a little similar to dividing up a project into "stories" in Extreme Programming (which I've found very useful) with the difference that it's individual-oriented rather than team-oriented and extends to all sorts of tasks. How has the rise in popularity of Agile methods affected your story?
Paul Bissex (biscuit) Sat 24 Feb 07 21:01
From my reading, the team sure could have used an injection of short-iteration development religion. It was painful in parts to read about them spinning their wheels over design decisions with no runnable code being produced.
Paul Bissex (biscuit) Sat 24 Feb 07 21:08
Thinking about that for a bit gave me another question to put in the queue: Scott, was it hard, especially given your experience with the Salon CMS project, being around when the Chandler team seemed utterly stalled out? Did you ever have the urge to step in and "save" them (even if you weren't exactly sure how you'd do it)?
Scott Rosenberg (scottros) Sun 25 Feb 07 22:28
Brian asked about the rise of agile methods. While I'm certainly finding that word used more regularly in the field, I rarely encounter people who have actually paid close attention to the specifics of the Agile Manifesto or the actual programs espoused by the various flavors of Agile and Extreme. Maybe that's a comment on the circles I travel in! At OSAF, certainly, though there were individuals who had exposure to or interest in various methodologies, the project never made a formal commitment to any of them, other than its broad and formative commitment to the open source approach itself. The Agile Manifesto itself is, I think, fascinating, and it points in more than one helpful direction. But you could recite the list of things that it values ("working software over comprehensive documentation," "responding to change over following a plan" and so on) and find that the Chandler team actually shared much or most of those values. "Responding to change over following a plan"? That's pretty much what they were doing, a lot of the time. The criticism of agile from the "heavyweight" types is that it becomes an excuse for not planning at all, but OSAF certainly had plenty of plans. Paul, I'm glad to be able to report that I never had the urge to "step in and save" the Chandler team! I would be very poorly qualified. There were a handful of occasions where I was in the room during a small-group meeting and someone might turn to me and ask what I thought, but they were pretty trivial. It was important to me to minimize "observer effects" (distortions caused by the act of observation that popular parlance tends to mislabel as "Heisenberg effects"), and mostly my goal -- during actual meetings as opposed to interviews -- was to vanish into the background. But I'm happy you asked because last week I read the review of my book in Harvard Magazine (by Prof. Harry Lewis), which accused me of a "lack of journalistic objectivity about the source of the woes [the book] depicts," and that bugged me a bit. Lewis makes some good points and fair criticisms; but I don't see where or how his observations at all justify that sort of charge. Of course, I've spent my entire writing career arguing that objectivity is a myth. But in a situation like this the word is being used, I think, as a way of saying that I went native on my subjects, I hung out with them for so long that I identified too closely with them, or something like that. Software-project Stockholm syndrome! And that is so far from my experience, and, I would argue, the truth, that it's funny. It would have been extremely easy to tell the Chandler story as slash-and-burn satire, a la, say, BURN RATE, but that's neither fair nor my style. In the end, I think a lot of people in the field -- Lewis is a professor of computer science who argues that "conventional management methods" would readily have rescued Chandler -- are a little too ready to hold the story of DREAMING IN CODE at arm's length. If we say that Chandler's saga is the exception to the rule, and that most of the rest of the field has licked these problems of scheduling and project estimation and collaboration and group vocabulary and so on, then we've got nothing to worry about. I'm not nearly so confident. If the problems are really so manageable, why are we still surrounded by train wrecks and disasters? Big projects are still hard, at every scale of team -- from our very small team here at Salon, to the mid-size group at OSAF, to the armies that labored over Windows Vista for twice as long as they were supposed to. So when I hear people like Lewis saying that Chandler is too "idiosyncratic" to generalize from, it sounds to me like a way of dismissed my broader conclusion -- that developing innovative software has some inherent and unique difficulties -- without actually grappling with it. PS URL of Harvard review is http://www.harvardmagazine.com/2007/03/talented-eccentrics.html
Cupido, Ergo Denego (robertflink) Mon 26 Feb 07 06:40
>I'm not nearly so confident. If the problems are really so manageable, why are we still surrounded by train wrecks and disasters? Big projects are still hard, at every scale of team -- from our very small team here at Salon, to the mid-size group at OSAF, to the armies that labored over Windows Vista for twice as long as they were supposed to.< I'm with you. Several scales are at work. Even with a small, select team, I would think a large system in any medium would be hard to manage; with a large team, even harder. (The recent Airbus problems had the added burden of work going on in diffused sites.) How about considering scale (in all dimensions) and overtly planning for the related problems? (Sort of "the problem we need to manage is management".) I tend to also fault the tendency to people in groups to value commitment as a style over analysis and, of course, the time honored tendency to throw good money after bad. There is a silver lining, however. In a world free of such problems, we might have one political party in power forever, a disgusting thought even if things were done in a better manner. BTW, plenty of complexity goes on fairly well (e.g. biological systems) with little human control suggesting that the self-ordering aspects of the universe are vastly superior than anything humans have demonstrated. (I say this at the risk of being accused of giving succor to the intelligent design freaks).
Michael Zentner (mz) Mon 26 Feb 07 07:34
>>> and that most of the rest of the field has licked these problems of scheduling and project estimation and collaboration and group vocabulary and so on My experience is just the opposite.
Elaine Sweeney (sweeney) Mon 26 Feb 07 10:03
Great to find this discussion here. Scott, I really enjoyed finding your book and reading it too. I forget why I picked it up to browse but when I saw the _Mythical Man-Month_ references I was sold. When Computer Literacy still had their big bookstore open on Trimble, I went in and saw a banner "Classics of Software Engineering". Immediately I started weaving my way through the store back to it, thinking of the titles that were probably in that nice boxed set I could see in stacks. Alas, they were some of the Microsoft Press authors collected together - books published in the last 5 years or so, and nice books I had enjoyed but not exactly the 'classics' I expected. It was one of those 'no hindsight' moments that seem to be so common in high tech. So it was wonderful to see you pulling in true classics and examining them in the light of the Chandler project and recent trends. I was surprised you did not touch on Brook's "second system" theory, that there are special gremlins in the planning of the successive development, as the Chandler project did seem to have the shadow of Kapor's first PIM hanging over it quite a bit. It was probably one of those interesting avenues that just did not fit in smoothly, though.
Mike Godwin (mnemonic) Mon 26 Feb 07 19:18
Hi, Scott. I still haven't read the book yet -- I've been tied up finishing a reading project for the WELL's sf conference -- but as you can imagine I have a kind of personal interest in the subject matter, since my own professional career began as the first employee of a Mitch Kapor startup. At the very beginning of EFF, we (that is, Mitch and I) officed with On Technology, and I learned a lot about the software business just being around the On folks. I'm interested in what you might have to say about how the On experience informed Mitch's direction of Chandler. (For those who don't remember, On Technology produced software for the Mac that did under Mac OS 6 and 7 what Spotlight does under Mac OS X -- only faster and better, and a decade and a half earlier. I immediately fell in love with the product, even before I had the chance to work for Mitch; it was only after coming to Cambridge, however, that I learned a bit of what Mitch had been trying to do with On Tech at the outset, and how the goals had shifted.) Mitch Kapor is, by the way, one of the best people I've ever worked with and among the best I've ever known -- his flaws, which it sounds like the book touches on, are so greatly outweighed by his virtues that it sometimes seems like carping even to mention them at all. I'm guessing lots of the Chandler folks have similarly high opinions of Mitch.
Scott Rosenberg (scottros) Mon 26 Feb 07 21:36
Robert, there's a long quest for software that behaves in a more "biological" manner -- or at least programs that handle faults in a more self-healing way, systems that are more homeostatic in behavior. I don't go into it at great length in DREAMING IN CODE but I do touch on it in the "Engineers and Artists" chapter, which surveys some of the more radical critiques of conventional software development (like those of Charles Simonyi, Alan Kay and Jaron Lanier, who speaks most directly to the biological metaphor). Elaine, thanks for the kind words. I am a total believer in Brooks' "second system" theory -- the idea that your second system is the most dangerous one of all because you will think it's a chance to fix all the crap you got wrong the first time around. I'm a believer because I lived it myself; our software disaster at Salon in 2000 was itself a "second system", an attempt to "do right" what we hadn't been able to do thoroughly when we rushed a quickie content-management system in place in spring of 1999 just in time for our IPO. Be very afraid when you find yourself thinking like that... Mike, good to see you here! I think the On Technology experience has always hung pretty heavily over Mitch. In a sense, *that* was his "second system," I think, in the Brooksian sense; his "second company," anyway. The On saga was chronicled on the front page of the Wall Street Journal in 1990 in a piece headlined "Painful Birth: Creating New Software Was Agonizing Task For Mitch Kapor Firm --- Despite Expert's Experience, Job Repeatedly Overran Time and Cost Forecasts." The subject of On Technology didn't come up often at OSAF; but there was one meeting in 2004 -- I quote this in the book -- where Kapor told his team, "The good news is, the last time I tried something like this, which was at On Technology, I had shut it down before we got this far. This time, I'm committed to launching." I do think Kapor's perseverance, his willingness to carry on in the face of wave after wave of delay and setback, comes at least in part from a determination not to repeat what happened with On. But I don't know that for a fact.
Mike Godwin (mnemonic) Tue 27 Feb 07 16:34
Here's a speech by Mitch from the On Technology days: <http://www.findarticles.com/p/articles/mi_m0REL/is_nDIRECT_v90/ai_8547178>
Paul Bissex (biscuit) Tue 27 Feb 07 17:17
Scott, if you were starting this book today, do you have any idea what kind of project you'd be looking to cover? For example, you've said that among your reasons for choosing Chandler in 2002/2003 was the fact that it was open source. I imagine that open source would still be on your list, but what other criteria might you apply today if you were looking for a subject for your book?
Scott Rosenberg (scottros) Tue 27 Feb 07 21:33
Mike, thanks for that link! DREAMING IN CODE includes an extensive discussion of Kapor's formal "Software Design Manifesto," but I hadn't found that transcript, which presents his ideas in a more informal manner. It's interesting to hear his (to me) sharp and accurate critique of the hierarchical file system. He understood in 1990 that it was going to be a disaster. Here it is, 2007 -- Microsoft tried once in the '90s (with "Cairo") and once more this decade (WinFS) to solve that problem and seems to have given up. OS X has some nifty flourishes but it's basically the same problem. Chandler -- under its broadest vision, where the notion was you'd be able to use it to organize all your data -- was another attempt to help us out of that mess. It's just a hard problem. Now here's the Web, growing insanely from out of nowhere, and -- funny thing: no hierarchy. (I'm looking forward to David Weinberger's forthcoming book EVERYTHING IS MISCELLANEOUS to shed further light on this.) Paul, it's almost impossible for me to think of what sort of project I'd be looking for today. When I chose Chandler, I was originally thinking of it merely as the first in a whole group of projects I'd write about. It was pure coincidence that, right at the moment I was conceiving my book project, Chandler popped into the news. I didn't actually have a checklist of criteria. At various times I idly imagined trying to follow a Microsoft project for one kind of contrast (OneNote looked fascinating), a lone-coder Web app for another (maybe something like del.icio.us), and so on. BitTorrent -- there's a story providing a lot of contrast to Chandler. Five or six projects! That would have been a great book to write, had I been able to clone myself several times. Today, obviously, Web apps are the rage, but it almost feels too easy to assume that would make the best story to tell. Web apps have one thing in common; they all need a browser. So Firefox itself would probably make an amazing epic. Really, I take what I have to describe, unabashedly, as a zen or John Cage-y approach to this stuff. Everything is interesting if you look at it closely enough. Every software project has a story to tell, and most of the stories are probably both fascinating and illuminating, if told well enough.
Mike Godwin (mnemonic) Tue 27 Feb 07 22:52
Well, it does help to have some great personalities to go along with the story you're telling (as you did here).
Earl Crabb (esoft) Wed 28 Feb 07 10:27
I'm midway through the book, trying to catch up as fast as possible, but early on, I was struck by the amount of attention paid to the inability to estimate project size. Having been the "system architect" on several projects, we were able to quite accurately estimate manpower needed for coding and testing hunks of those projects. (The largest was a 15 man-year project that produced 250,000 lines of 370 assembler code that is still in use today, after 30+ years.) That was back in the '70's and '80's, though, and I wondered why it is so much more difficult today. Do you think it is because of the massive operating systems of today, and the proliferation of languages? When we code now, we have to make so many assumptions that the OS and interpreter/compiler will do what we think they'll do, that much of our time debugging is trying to figure out why the OS or interpreter/compiler isn't doing what we expected? Are we building systems so complicated that we're bound to fail?
Gail Williams (gail) Wed 28 Feb 07 13:28
Fascinating. For some reason that reminds me of this rather famous article, http://www.wholeearthmag.com/ArticleBin/111-3.pdf It's the famous Vernor Vinge essay on the technological singularity, or the end of human-dominated history, when Artificial Intelligence gets really smart and becomes self-aware. Recently he's been saying that this might not happen. Acccording to the notes at: http://discuss.longnow.org/viewtopic.php?t=401 "The overall non-Singularity condition he called "The Age of Failed Dreams." The main driver is that software simply continues failing to keep pace with hardware improvements. One after another, enormous billion-dollar software projects simply do not run, as has already happened at the FBI, air traffic control, IRS, and many others. Some large automation projects fail catastrophically, with planes running into each. So hardware development eventually lags, and materials research lags, and no strong AI develops. " I'm not sure I have a question, just interesting to see what extraordinary complexity brings when coordination between human-created parts is required.
Paul Bissex (biscuit) Wed 28 Feb 07 14:33
I'm pretty sure that Scott addresses in the book somewhere (I don't have it here at the moment) that there's a pretty big difference between a software project tackling a well-mapped space and a project that is setting out *expressly* to map a completely new space. And if he didn't say it in the book, he said it in one of the interviews I've listened to. And if not there, then somebody else said it but I only read them because I was fired up from reading Scott's book! I'm don't know whether this is the difference between Earl's experience and OSAF's, but I'm guessing it's a factor. Also, it seemed pretty clear to me that hard ship-dates were not actually a very high priority at OSAF. Certainly Kapor could have made a hard requirement at the outset that a usable rev be shipped every, say, three months. But he didn't. That's a legitimate choice IMO.
Elaine Sweeney (sweeney) Wed 28 Feb 07 14:42
There was something I thought was a very good point - one of the team saying that his estimates were poor because his assumptions interlaced with all the other poor estimates. That's one of the best descriptions I've heard of why solid teams can be wildly off. I think also by the time you get an end-to-end schedule estimate from everyone on a team, that they are all 'raring to go off and slay the beast. It's extremely difficult to get to the part of identifying the risks and mitigating them then.
Sharon Lynne Fisher (slf) Wed 28 Feb 07 18:19
(Coincidentally, with the Tracy Kidder mention above, I'm sitting in a lecture by Tracy Kidder right now. Anything I should ask him?)
Scott Rosenberg (scottros) Wed 28 Feb 07 20:07
I think I probably missed the window for Tracy Kidder Q&A, but I really don't have anything to ask him. Just general admiration for his work! Earl, I think there is definitely a trait that some programmers and managers have (and more don't) of being able to come up with good estimates for schedules, but it appears, from what I can tell, to be an art rather than a science, and it depends on experience as much as anything else. I do think that the complexity of today's systems makes things harder. The programmer for the most part no longer has to worry too much about the many lower layers of abstraction that his Wizzy Cool Web App depends on. But when there is a particularly elusive bug, she or he will face an even more bewildering maze of possibilities in trying to isolate that bug. This is what Joel Spolsky so eloquently described as "The Law of Leaky Abstractions." And it's why, each time we add a new layer of abstraction to make it easier for programmers to leverage new capabilities -- a fine thing in itself -- life in some ways gets harder, not easier, for the poor coder. In the book I talked about this "ladder of abstractions" and said, "No matter how high they climb on that ladder, they will always have to run up and down it more than they'd like -- and the taller it becomes, the longer the trip." Gail, the whole issue of the Singularity enters my tale only at the very end -- where I describe Mitch Kapor's "Long Bet" with Ray Kurzweil about the likelihood of "strong" AI arriving by 2029. Kurzweil thinks it's inevitable, as a precursor step to the biggie Singularity he's sure is headed our way; Kapor takes the humanist view -- that there are aspects of human intelligence that relate to the union of our bodies and minds in a single organism, and that duplicating that in logic and silicon is a lot further off than the AI true believers think. Thanks for the pointer to the notes on Vinge's talk -- I have been to several of those Long Now events but missed this one. I love his freehand graphs! -------------- Let me turn the tables temporarily in this interview, since we're at half-time on the fortnight, as I understand it, and ask this extremely clued-in group a question of my own. I've had a certain set of experiences that led me to feel that "software is hard." I went out and found confirmation of that both in the literature (the original formulation of that phrase is Knuth's) and in the field. The result was "Dreaming in Code." But, you know, am I just wrong? Is software really not that hard, and have I just been staring at one end of the spectrum? I'm asking partly in devil's advocate mode, but I hope I'm also open-minded enough to be able to consider the argument, and weigh at least the possibility that I've just been off-base, or overly pessimistic. That is what I'm hearing from some quarters of the development world, and I'm interested in what you all think.
Earl Crabb (esoft) Wed 28 Feb 07 22:54
Well, I think it all depends! Just like "is building a building hard?" If you're building a corn crib, no, it's not hard. If it's a skyscraper, yeah, it'a a bunch harder. A piece of software is pretty much different from any other piece of software (which you point out in the book, even). To code any given algorithm, might be easy. Some algorithms are easier than others, e.g., adaptive Huffman coding is easier than simple Huffman coding, and bubble sort is easier than heapsort. However, when you start putting external requirements on the coding, the complexity can increase rapidly. In c, for example, the algorithms mentioned above are much easier to code than they would be if you were required to use Fortran or COBOL. Even so, this softwre is not all that hard. But then, these kinds of algorithms are analagous to the corn crib, not the skyscraper. They depend on few environmental fators, they rely on the formulas and the features of the coding language. This software is not hard. But now, take a larger project, say the 250k line project I mentioned earlier. Lots of algorithms, a couple compilers, some graphics output, some report writers, and some very fancy data manipulations were in that package. A huge project to us back then, but still, one that could be partitioned into fairly easy chunks. Assembler language leaves no guessing what the language will do at any point, and the operating system (in this case a variant of the precursor of IBM's VM/CMS) was primitive enough that we knew what it would do, pretty much. So, still, not hard. Tedious, but not all that hard. But now, take even a simpler project, such as Mitch's project... the underlying idea is simple, anyway...but given that it has to work on multiple machines, multiple operating systems, multiple display and other I/O devices, multiple data storage devices, and appear to be the same to the user, now it becomes hard, very hard. Each of the environmental variables has it's own interface, and each of those has a different way of talking to your software. Each has it's own share of bugs, as well, which you don't find out about until too late, and you have to do a workaround while waiting for the manufacturer of that component to either fix it or not. That's what makes software hard, in my opinion. Don Knuth would probably have a very different opinion of why it's hard, though!
Michael Zentner (mz) Thu 1 Mar 07 08:07
>>> But, you know, am I just wrong? Is software really not that hard, and have I just been staring at one end of the spectrum? Actually, software is easy. Too easy. Which is the problem. It's almost trivially easy to write software that does some simple task. It requires no controls, no oversight, no review, no nothing except a small amount of knowledge. Think about the canonical "Hello World" program. The problem comes when a large complex project is approached on that level.
Christian Crumlish (xian) Thu 1 Mar 07 13:21
Well, sure it's hard, but reading yoyr book I often found myself frustrated about the infinite regresses the Chandler/OSAF team let themselves get into. We can't do X until we do Y. Over and over and over. But why not? This is all bootstrapping. They were/are trying to make something that hadn't existed before. Inevitably they were going to have to kick away the ladder and maybe even later realize they had used the wrong foundation and build it over etc., sort of like how Mozilla was a rewrite of Netscape Navigastor Instead they seem to have gotten caught up in a strict sense of dependencies that led them often into paralysis, instead of toward running code.
Ted Leung (ted-leung) Thu 1 Mar 07 16:27
Hi folks, My name is Ted Leung, and I work at OSAF and appear in the book. Scott and Salon asked if someone would participate in the conference, and I volunteered. I'm not sure what the best/most useful way is for me to participate, but I just wanted to let you know that I am here. We can see where it goes from there...
Get Shorty (esau) Thu 1 Mar 07 16:34
Hey, I knew a Ted Leung once, a long time ago.
Members: Enter the conference to participate