Inkwell: Authors and Artists

    
I'd never heard of David Allen or his "Getting Things Done" methodology 
before it came up in the course of my observation of the work at OSAF. 
Mitch Kapor had some acquaintance with Allen, I think through his interest 
in personal productivity software, and he thought that some of Allen's 
ideas could be useful in Chandler's design. Around the same time, 
coincidentally (or maybe not, in a synchronicity-of-good-ideas kind of 
way), Merlin Mann's 43 Folders blog started up -- Mann showed up at OSAF 
and gave a talk about GTD, and later on Allen met with the OSAF design 
team.

In brief, GTD suggests that you organize your life by creating a "system" 
-- it can be on paper or in a computer, doesn't matter -- for tracking 
*everything* in your life, from the smallest to-do to the biggest 
life-goal. In this it resembles other personal productivity systems, but 
it has a few methodological wrinkles and some unique philosophical traits 
derived from Allen's martial-arts background. You divide things up along 
several different axes -- different projects on the one hand, and 
different "contexts" (basically, locations where you can Get Things Done). 
And you break vague, general tasks like "start work on new Web site" down 
into very specific "next actions", like "call Web hosting company to 
compare prices" and "post ad for site designer." The "next action" concept 
is extremely valuable for people who get the hang of it; it routinizes the 
"journey of a thousand miles begins with a single step" notion into a form 
that can actually help you. 

The most useful aspect of GTD to me is the idea of clearing your head of 
the nagging "what am I forgetting?" feeling by letting you trust the 
system you've built to be your sort of outboard brain. Some other insights 
include stuff like, be sure you have a reliable means to defer things 
without losing track of them (i.e., "this becomes important on the 21st of 
next month but I don't want to think of it till then") and also to not 
lose sight of things you've delegated and are "waiting on." 

There's this paradox at the heart of GTD, which people like Mann are the 
first to admit: The whole technique is about being productive, and yet the 
effort to implement the technique offers a nearly infinite set of 
opportunities to procrastinate by tweaking your "system." This temptation 
proves particularly seductive to the many programmers who have flocked to 
the GTD method and who are constantly trading tips about the latest cool 
software tool for managing GTD systems. 

In a sense GTD is all about applying a programming systems approach to 
one's own work and life; programmers are used to bug trackers and 
trouble-ticket systems, and GTD is sort of an extension of that mindset 
into all realms of life. Some people find that incredibly helpful. I've 
fooled around with GTD myself with only partial success. I use Ecco, this 
ancient, semi-defunct PIM program, and it adapts beautifully to the GTD 
approach, but I'm not very good at "gardening" the system, and I find 
myself much happier just Doing Things than figuring out how to Get Things 
Done. It might be that I am simply a naturally productive person, or it 
might be that I'm slovenly and ill-disciplined!

GTD turned out to be only a small piece of the Chandler saga, but I think 
that as Chandler adds features (the most recent version of the program 
shows a Dashboard view of calendar events and tasks that embodies some 
GTD-style ideas) the hordes of GTD devotees may latch onto it in ways that 
could help give it what it most needs -- a community of early adopters.
  

    
That sounds a little similar to dividing up a project into "stories"
in Extreme Programming (which I've found very useful) with the
difference that it's individual-oriented rather than team-oriented and
extends to all sorts of tasks.

How has the rise in popularity of Agile methods affected your story?
  

    
From my reading, the team sure could have used an injection of
short-iteration development religion. It was painful in parts to read
about them spinning their wheels over design decisions with no
runnable code being produced.
  

    
Thinking about that for a bit gave me another question to put in the
queue: Scott, was it hard, especially given your experience with the
Salon CMS project, being around when the Chandler team seemed utterly
stalled out? Did you ever have the urge to step in and "save" them
(even if you weren't exactly sure how you'd do it)?
  

    
Brian asked about the rise of agile methods. While I'm certainly finding 
that word used more regularly in the field, I rarely encounter people who 
have actually paid close attention to the specifics of the Agile Manifesto 
or the actual programs espoused by the various flavors of Agile and 
Extreme. Maybe that's a comment on the circles I travel in! At OSAF, 
certainly, though there were individuals who had exposure to or interest 
in various methodologies, the project never made a formal commitment to 
any of them, other than its broad and formative commitment to the open 
source approach itself.

The Agile Manifesto itself is, I think, fascinating, and it points in more 
than one helpful direction. But you could recite the list of things that 
it values ("working software over comprehensive documentation," 
"responding to change over following a plan" and so on) and find that the 
Chandler team actually shared much or most of those values. "Responding to 
change over following a plan"? That's pretty much what they were doing, a 
lot of the time. The criticism of agile from the "heavyweight" types is 
that it becomes an excuse for not planning at all, but OSAF certainly had 
plenty of plans.

Paul, I'm glad to be able to report that I never had the urge to "step in 
and save" the Chandler team! I would be very poorly qualified. There were 
a handful of occasions where I was in the room during a small-group 
meeting and someone might turn to me and ask what I thought, but they were 
pretty trivial. It was important to me to minimize "observer effects" 
(distortions caused by the act of observation that popular parlance tends 
to mislabel as "Heisenberg effects"), and mostly my goal -- during actual 
meetings as opposed to interviews -- was to vanish into the background.

But I'm happy you asked because last week I read the review of my book in 
Harvard Magazine (by Prof. Harry Lewis), which accused me of a "lack of 
journalistic objectivity about the source of the woes [the book] depicts," 
and that bugged me a bit. Lewis makes some good points and fair 
criticisms; but I don't see where or how his observations at all justify 
that sort of charge. Of course, I've spent my entire writing career 
arguing that objectivity is a myth. But in a situation like this the word 
is being used, I think, as a way of saying that I went native on my 
subjects, I hung out with them for so long that I identified too closely 
with them, or something like that. Software-project Stockholm syndrome! 
And that is so far from my experience, and, I would argue, the truth, that 
it's funny.

It would have been extremely easy to tell the Chandler story as 
slash-and-burn satire, a la, say, BURN RATE, but that's neither fair nor 
my style. In the end, I think a lot of people in the field -- Lewis is a 
professor of computer science who argues that "conventional management 
methods" would readily have rescued Chandler -- are a little too ready to 
hold the story of DREAMING IN CODE at arm's length. If we say that 
Chandler's saga is the exception to the rule, and that most of the rest of 
the field has licked these problems of scheduling and project estimation 
and collaboration and group vocabulary and so on, then we've got nothing 
to worry about. 

I'm not nearly so confident. If the problems are really so manageable, why 
are we still surrounded by train wrecks and disasters? Big projects are 
still hard, at every scale of team -- from our very small team here at 
Salon, to the mid-size group at OSAF, to the armies that labored over 
Windows Vista for twice as long as they were supposed to. So when I hear 
people like Lewis saying that Chandler is too "idiosyncratic" to 
generalize from, it sounds to me like a way of dismissed my broader 
conclusion -- that developing innovative software has some inherent and 
unique difficulties -- without actually grappling with it. 

PS URL of Harvard review is
http://www.harvardmagazine.com/2007/03/talented-eccentrics.html
  

    
>I'm not nearly so confident. If the problems are really so
manageable, why are we still surrounded by train wrecks and disasters?
Big projects are still hard, at every scale of team -- from our very
small team here at Salon, to the mid-size group at OSAF, to the armies
that labored over Windows Vista for twice as long as they were supposed
to.<

I'm with you.

Several scales are at work. Even with a small, select team, I would
think a large system in any medium would be hard to manage; with a
large team, even harder. (The recent Airbus problems had the added
burden of work going on in diffused sites.) How about considering scale
(in all dimensions) and overtly planning for the related problems?
(Sort of "the problem we need to manage is management".)

I tend to also fault the tendency to people in groups to value
commitment as a style over analysis and, of course, the time honored
tendency to throw good money after bad. 

There is a silver lining, however. In a world free of such problems,
we might have one political party in power forever, a disgusting
thought even if things were done in a better manner. 

BTW, plenty of complexity goes on fairly well (e.g. biological
systems) with little human control suggesting that the self-ordering
aspects of the universe are vastly superior than anything humans have
demonstrated.

(I say this at the risk of being accused of giving succor to the
intelligent design freaks). 
  

    
>>> and that most of the rest of 
the field has licked these problems of scheduling and project
estimation 
and collaboration and group vocabulary and so on

My experience is just the opposite.
  

    
Great to find this discussion here.  Scott, I really enjoyed finding
your book and reading it too.  I forget why I picked it up to browse
but when I saw the _Mythical Man-Month_ references I was sold.

When Computer Literacy still had their big bookstore open on Trimble,
I went in and saw a banner "Classics of Software Engineering". 
Immediately I started weaving my way through the store back to it,
thinking of the titles that were probably in that nice boxed set I
could see in stacks.  Alas, they were some of the Microsoft Press
authors collected together - books published in the last 5 years or so,
and nice books I had enjoyed but not exactly the 'classics' I
expected.  It was one of those 'no hindsight' moments that seem to be
so common in high tech.

So it was wonderful to see you pulling in true classics and examining
them in the light of the Chandler project and recent trends.

I was surprised you did not touch on Brook's "second system" theory,
that there are special gremlins in the planning of the successive
development, as the Chandler project did seem to have the shadow of
Kapor's first PIM hanging over it quite a bit.  It was probably one of
those interesting avenues that just did not fit in smoothly, though.
  

    

Hi, Scott. I still haven't read the book yet -- I've been tied up finishing
a reading project for the WELL's sf conference -- but as you can imagine
I have a kind of personal interest in the subject matter, since my own
professional career began as the first employee of a Mitch Kapor startup.

At the very beginning of EFF, we (that is, Mitch and I) officed with On
Technology, and I learned a lot about the software business just being
around the On folks. I'm interested in what you might have to say about
how the On experience informed Mitch's direction of Chandler.

(For those who don't remember, On Technology produced software for the Mac
that did under Mac OS 6 and 7 what Spotlight does under Mac OS X -- only
faster and better, and a decade and a half earlier. I immediately fell in
love with the product, even before I had the chance to work for Mitch; it
was only after coming to Cambridge, however, that I learned a bit of what
Mitch had been trying to do with On Tech at the outset, and how the goals
had shifted.)

Mitch Kapor is, by the way, one of the best people I've ever worked with and
among the best I've ever known -- his flaws, which it sounds like the book
touches on, are so greatly outweighed by his virtues that it sometimes
seems like carping even to mention them at all. I'm guessing lots of the
Chandler folks have similarly high opinions of Mitch.
  

    
Robert, there's a long quest for software that behaves in a more 
"biological" manner -- or at least programs that handle faults in a more 
self-healing way, systems that are more homeostatic in behavior. I don't 
go into it at great length in DREAMING IN CODE but I do touch on it in the 
"Engineers and Artists" chapter, which surveys some of the more radical 
critiques of conventional software development (like those of Charles 
Simonyi, Alan Kay and Jaron Lanier, who speaks most directly to the 
biological metaphor). 
 
 Elaine, thanks for the kind words. I am a total believer in Brooks' 
"second system" theory -- the idea that your second system is the most 
dangerous one of all because you will think it's a chance to fix all the 
crap you got wrong the first time around. I'm a believer because I lived 
it myself; our software disaster at Salon in 2000 was itself a "second 
system", an attempt to "do right" what we hadn't been able to do 
thoroughly when we rushed a quickie content-management system in place in 
spring of 1999 just in time for our IPO. Be very afraid when you find 
yourself thinking like that...
 
Mike, good to see you here! I think the On Technology experience has 
always hung pretty heavily over Mitch. In a sense, *that* was his "second 
system," I think, in the Brooksian sense; his "second company," anyway. 
The On saga was chronicled on the front page of the Wall Street Journal in 
1990 in a piece headlined "Painful Birth: Creating New Software Was 
Agonizing Task For Mitch Kapor Firm --- Despite Expert's Experience, Job 
Repeatedly Overran Time and Cost Forecasts." The subject of On Technology 
didn't come up often at OSAF; but there was one meeting in 2004 -- I quote 
this in the book -- where Kapor told his team, "The good news is, the last 
time I tried something like this, which was at On Technology, I had shut 
it down before we got this far. This time, I'm committed to launching." I 
do think Kapor's perseverance, his willingness to  carry on in the face of 
wave after wave of delay and setback, comes at least in part from a 
determination not to repeat what happened with On. But I don't know that 
for a fact. 
  

    

Here's a speech by Mitch from the On Technology days:
<http://www.findarticles.com/p/articles/mi_m0REL/is_nDIRECT_v90/ai_8547178>
  

    
Scott, if you were starting this book today, do you have any idea what
kind of project you'd be looking to cover? For example, you've said
that among your reasons for choosing Chandler in 2002/2003 was the
fact that it was open source. I imagine that open source would still
be on your list, but what other criteria might you apply today if you
were looking for a subject for your book?
  

    
Mike, thanks for that link! DREAMING IN CODE includes an extensive 
discussion of Kapor's formal "Software Design Manifesto," but I hadn't 
found that transcript, which presents his ideas in a more informal manner. 
It's interesting to hear his (to me) sharp and accurate critique of the 
hierarchical file system. He understood in 1990 that it was going to be a 
disaster. Here it is, 2007 -- Microsoft tried once in the '90s (with 
"Cairo") and once more this decade (WinFS) to solve that problem and seems 
to have given up. OS X has some nifty flourishes but it's basically the 
same problem. 

Chandler -- under its broadest vision, where the notion was you'd be able 
to use it to organize all your data -- was another attempt to help us out 
of that mess. It's just a hard problem. Now here's the Web, growing 
insanely from out of nowhere, and -- funny thing: no hierarchy. (I'm 
looking forward to David Weinberger's forthcoming book EVERYTHING IS 
MISCELLANEOUS to shed further light on this.)

Paul, it's almost impossible for me to think of what sort of project I'd 
be looking for today. When I chose Chandler, I was originally thinking of 
it merely as the first in a whole group of projects I'd write about. It 
was pure coincidence that, right at the moment I was conceiving my book 
project, Chandler popped into the news. I didn't actually have a checklist 
of criteria. At various times I idly imagined trying to follow a Microsoft 
project for one kind of contrast (OneNote looked fascinating), a 
lone-coder Web app for another (maybe something like del.icio.us), and so 
on. BitTorrent -- there's a story providing a lot of contrast to Chandler. 
Five or six projects! That would have been a great book to write, had I 
been able to clone myself several times.

Today, obviously, Web apps are the rage, but it almost feels too easy to 
assume that would make the best story to tell. Web apps have one thing in 
common; they all need a browser. So Firefox itself would probably make an 
amazing epic.

Really, I take what I have to describe, unabashedly, as a zen or John 
Cage-y approach to this stuff. Everything is interesting if you look at it 
closely enough. Every software project has a story to tell, and most of 
the stories are probably both fascinating and illuminating, if told well 
enough. 
  

    

Well, it does help to have some great personalities to go along with the
story you're telling (as you did here).
  

    
I'm midway through the book, trying to catch up as fast as possible, 
but early on, I was struck by the amount of attention paid to the
inability to estimate project size.  Having been the "system architect"
on several projects, we were able to quite accurately estimate
manpower needed for coding and testing hunks of those projects.
(The largest was a 15 man-year project that produced 250,000 lines
of 370 assembler code that is still in use today, after 30+ years.)

That was back in the '70's and '80's, though, and I wondered why
it is so much more difficult today.  Do you think it is because
of the massive operating systems of today, and the proliferation
of languages?  When we code now, we have to make so many assumptions
that the OS and interpreter/compiler will do what we think they'll
do, that much of our time debugging is trying to figure out why
the OS or interpreter/compiler isn't doing what we expected?

Are we building systems so complicated that we're bound to fail?
  

    

Fascinating. For some reason that reminds me of this rather famous article,

http://www.wholeearthmag.com/ArticleBin/111-3.pdf
It's the famous Vernor Vinge essay on the technological singularity, or the
end of human-dominated history, when Artificial Intelligence gets really
smart and becomes self-aware.  Recently he's been saying that this might
not happen.  

Acccording to the notes at: http://discuss.longnow.org/viewtopic.php?t=401

"The overall non-Singularity condition he called "The Age of Failed
Dreams." The main driver is that software simply continues failing to keep
pace with hardware improvements. One after another, enormous
billion-dollar software projects simply do not run, as has already
happened at the FBI, air traffic control, IRS, and many others. Some large
automation projects fail catastrophically, with planes running into each.
So hardware development eventually lags, and materials research lags, and
no strong AI develops. "

I'm not sure I have a question, just interesting to see what extraordinary
complexity brings when coordination between human-created parts is required.
  

    
I'm pretty sure that Scott addresses in the book somewhere (I don't have it
here at the moment) that there's a pretty big difference between a software
project tackling a well-mapped space and a project that is setting out
*expressly* to map a completely new space. And if he didn't say it in the
book, he said it in one of the interviews I've listened to. And if not
there, then somebody else said it but I only read them because I was fired
up from reading Scott's book!

I'm don't know whether this is the difference between Earl's experience and
OSAF's, but I'm guessing it's a factor.

Also, it seemed pretty clear to me that hard ship-dates were not actually a
very high priority at OSAF.  Certainly Kapor could have made a hard
requirement at the outset that a usable rev be shipped every, say, three
months. But he didn't. That's a legitimate choice IMO.
  

    
There was something I thought was a very good point - one of the team
saying that his estimates were poor because his assumptions interlaced
with all the other poor estimates.  That's one of the best descriptions
I've heard of why solid teams can be wildly off.

I think also by the time you get an end-to-end schedule estimate from
everyone on a team, that they are all 'raring to go off and slay the
beast.  It's extremely difficult to get to the part of identifying the
risks and mitigating them then.
  

    
(Coincidentally, with the Tracy Kidder mention above, I'm sitting in a
lecture by Tracy Kidder right now. Anything I should ask him?)
  

    
I think I probably missed the window for Tracy Kidder Q&A, but I really 
don't have anything to ask him. Just general admiration for his work!
 
Earl, I think there is definitely a trait that some programmers and 
managers have (and more don't) of being able to come up with good 
estimates for schedules, but it appears, from what I can tell, to be an 
art rather than a science, and it depends on experience as much as 
anything else. I do think that the complexity of today's systems makes 
things harder. The programmer for the most part no longer has to worry too 
much about the many lower layers of abstraction that his Wizzy Cool Web 
App depends on. But when there is a particularly elusive bug, she or he 
will face an even more bewildering maze of possibilities in trying to 
isolate that bug. 

This is what Joel Spolsky so eloquently described as "The Law of Leaky 
Abstractions." And it's why, each time we add a new layer of abstraction 
to make it easier for programmers to leverage new capabilities -- a fine 
thing in itself -- life in some ways gets harder, not easier, for the poor 
coder. In the book I talked about this "ladder of abstractions" and said, 
"No matter how high they climb on that ladder, they will always have to 
run up and down it more than they'd like -- and the taller it becomes, the 
longer the trip."

Gail, the whole issue of the Singularity enters my tale only at the very 
end -- where I describe Mitch Kapor's "Long Bet" with Ray Kurzweil about 
the likelihood of "strong" AI arriving by 2029. Kurzweil thinks it's 
inevitable, as a precursor step to the biggie Singularity he's sure is 
headed our way; Kapor takes the humanist view -- that there are aspects of 
human intelligence that relate to the union of our bodies and minds in a 
single organism, and that duplicating that in logic and silicon is a lot 
further off than the AI true believers think.

Thanks for the pointer to the notes on Vinge's talk -- I have been to 
several of those Long Now events but missed this one. I love his freehand 
graphs!

--------------

Let me turn the tables temporarily in this interview, since we're at 
half-time on the fortnight, as I understand it, and ask this extremely 
clued-in group a question of my own. I've had a certain set of experiences 
that led me to feel that "software is hard." I went out and found 
confirmation of that both in the literature (the original formulation of 
that phrase is Knuth's) and in the field. The result was "Dreaming in 
Code."

But, you know, am I just wrong? Is software really not that hard, and have 
I just been staring at one end of the spectrum?

I'm asking partly in devil's advocate mode, but I hope I'm also 
open-minded enough to be able to consider the argument, and weigh at least 
the possibility that I've just been off-base, or overly pessimistic. That 
is what I'm hearing from some quarters of the development world, and I'm 
interested in what you all think.
  

    
Well, I think it all depends!  Just like "is building a building hard?"
If you're building a corn crib, no, it's not hard.  If it's a skyscraper,
yeah, it'a a bunch harder.

A piece of software is pretty much different from any other piece of
software (which you point out in the book, even).  To code any given
algorithm, might be easy.  Some algorithms are easier than others,
e.g., adaptive Huffman coding is easier than simple Huffman coding,
and bubble sort is easier than heapsort.   

However, when you start putting external requirements on the coding,
the complexity can increase rapidly.  In c, for example, the 
algorithms mentioned above are much easier to code than they
would be if you were required to use Fortran or COBOL.

Even so, this softwre is not all that hard.  But then, these kinds of
algorithms are analagous to the corn crib, not the skyscraper.
They depend on few environmental fators, they rely on the formulas
and the features of the coding language.  This software is not hard.

But now, take a larger project, say the 250k line project I 
mentioned earlier.  Lots of algorithms, a couple compilers, some
graphics output, some report writers, and some very fancy data
manipulations were in that package.    A huge project to us back then,
but still, one that could be partitioned into fairly easy chunks.
Assembler language leaves no guessing what the language will do
at any point, and the operating system (in this case a variant of
the precursor of IBM's VM/CMS) was primitive enough that we
knew what it would do, pretty much.  So, still, not hard.  Tedious,
but not all that hard.

But now, take even a simpler project, such as Mitch's project...
the underlying idea is simple, anyway...but given that it has to
work on multiple machines, multiple operating systems, multiple
display and other I/O devices, multiple data storage devices,
and appear to be the same to the user, now it becomes hard,
very hard.  Each of the environmental variables has it's own
interface, and each of those has a different way of talking
to your software.  Each has it's own share of bugs, as well,
which you don't find out about until too late, and you have
to do a workaround while waiting for the manufacturer of
that component to either fix it or not.

That's what makes software hard, in my opinion.

Don Knuth would probably have a very different opinion of
why it's hard, though!
  

    
>>> But, you know, am I just wrong? Is software really not that hard,
and have  I just been staring at one end of the spectrum?

Actually, software is easy. Too easy. Which is the problem. It's
almost trivially easy to write software that does some simple task. It
requires no controls, no oversight, no review, no nothing except a
small amount of knowledge. Think about the canonical "Hello World"
program.

The problem comes when a large complex project is approached on that
level. 
  

    
Well, sure it's hard, but reading yoyr book I often found myself
frustrated about the infinite regresses the Chandler/OSAF team let
themselves get into.

We can't do X until we do Y.

Over and over and over. But why not? This is all bootstrapping. They
were/are trying to make something that hadn't existed before.
Inevitably they were going to have to kick away the ladder and maybe
even later realize they had used the wrong foundation and build it over
etc., sort of like how Mozilla was a rewrite of Netscape Navigastor

Instead they seem to have gotten caught up in a strict sense of
dependencies that led them often into paralysis, instead of toward
running code.
  

    
Hi folks,

My name is Ted Leung, and I work at OSAF and appear in the book. 
Scott and Salon asked if someone would participate in the conference,
and I volunteered.  I'm not sure what the best/most useful way is for
me to participate, but I just wanted to let you know that I am here. 
We can see where it goes from there...
  

    
Hey, I knew a Ted Leung once, a long time ago.