Jon Lebkowsky (jonl) Thu 16 Nov 00 19:41
Our next interview is with Cory Doctorow. Cory is a self-described "Renaissance geek" with fingers in a lot of pies: author, journalist, critic, technocrat, entrepreneur, obsessive collector...obviously, he doesn't get a lot of sleep. Cory won the John W. Campbell Award for best new science fiction writer at the 2000 Hugo Awards in Chicago, and his short fiction appears regularly in all the notable magazines and anthologies in the field -- most recently, a story in the December Asimov's, "I Love Paree," written with Michael Skeet. His first book, "The Complete Idiot's Guide to Publishing Science Fiction" (written with Karl Schroeder) is on shelves now, and his second, a novel called "Down and Out in the Magic Kingdom," is at a publisher now. Cory has written feature stories for Wired Magazine and elsewhere, and is a regular book and technology reviewer for Wired. His journalism is fuelled by good narratives and wicked-cool technology. The venture that's currently eating Cory's life is openCOLA (http://www.opencola.com), a company he started with two partners in June 1999. Now with 50 employees and offices in Toronto and San Francisco, openCOLA is a funded, commercial open-source initiative that's building a "relevance-switching" mechanism for peer-to-peer networks. The idea is to marry collaborative search technology to self-organizing networks and in so doing to create a viable means to evaluate every document in the world and determine its idiosyncratic relevance to every human being on the Internet. This is just as ambitious as it sounds, and is very, very exciting stuff. Cory's title at openCOLA is "Chief Evangelist." Where traditional software companies have sales forces that sell licenses, openCOLA has evangelists who sell the ideology -- the software is free. Chief Evangelist is not so far from Spokesmodel, with a little bit of bottlewasher and oddsbody thrown in. He's just moved from Toronto to San Francisco to take on the role, and is getting to feel like a for-real Californian (he was just named "Customer of the Week" at the local Peet's Coffee!). Cory will be interviewed by Mark Frauenfelder, whose excellent piece about openCOLA for The Industry Standard, can be read at http://www.thestandard.com/article/display/0,1151,19498,00.html Mark, a writer and illustrator from Los Angeles, is the founder of the pop oddity zine Boing Boing, and the co-editor of The Happy Mutant Handbook, which was published in 1995 by Putnam Berkley. He worked at Wired Ventures from 1993 to 1998, where was an associate editor for Wired, the founding editor-in-chief of Wired Online, and an editor at Wired Books. Today, Mark writes for Wired and The Industry Standard, and writes a monthly column for Playboy magazine called Living Online. He also draws cartoons for Guru.com and Digitallivingtoday.com. Mark has been a member of the Well since 1988. You can see his work at http:/boingboing.net/markf.html. Please join me in welcoming Cory and Mark to inkwell.vue!
Life in the big (doctorow) Fri 17 Nov 00 06:45
Well, I'm here, having just wrapped upa fine week at Comdex, culiminating in the development for a cold. Flying to Philadelphia for my annual appearance at Philcon this afternoon, then to NYC for the Science Fiction Writers of America booze-n-shmooze, then to Montreal, then, (maybe) back to San Francisco long enough to change suitcases before I leave for Hong Kong. Yikes.
Mark Frauenfelder (mark) Fri 17 Nov 00 11:08
Cory has pockets full of gadgets that keep him connected while he's on the road and in the air. Cory, why don't you empty out your pockets right now and tell everybody what you are carrying?
Jon Lebkowsky (jonl) Fri 17 Nov 00 11:55
(And I want to know how you can do the Comdex to SFWA transition... I hope you have a half gallon of Cuervo Gold somewhere in your luggage!!)
Life in the big (doctorow) Sat 18 Nov 00 05:19
In my pocketses: * RIM Blackberry pager * Gerber BYO Multitool * Gerber Legend multitool * White-light ultraminiature 10,000h LED flashlight/zipperfob * Wallet * Change * Eyemodule * Backup module for Visor * Modem for Visor * 8MB flash for Visor * Motorola i700 plus ruggedized cellphone * Xtreme CX-20 rugged cigarette lighter (rated for 500' drops!) * Visor Platinum in Concept Kitchen Bumper case * IDEO dual-mode Visor stylus The hard part abot shlepping all this crap around is going through airport security. I've switched to wearing a jacket with a big marsupial pocket so that I can fill it up with all this stuff and drop it on the x-ray. Otherwise, it can take up to 20 minutes to divest myself of all the metal I habitually carry.
Life in the big (doctorow) Sat 18 Nov 00 05:46
The Comdex transition wasn't easy, but mostly because I'm coming down with a cold -- but my panel last night was "Hackers, Crackers and Wackos" with Eric Raymond and Bud Sparhawk and Ann something-or-other. A room full of sysadmins and a panel full of nerdy writers. It was gold.
Alan Thornton (sd) Sat 18 Nov 00 06:49
Sally Vegetus (viv) Sun 19 Nov 00 06:32
Hi, Cory! I met Cory at Philcon yesterday and 'evangelism' barely describes his energy. He didn't talk like he was suffering from a cold and his enthusiasm for openCOLA helped me make sense of it. But, Philcon did not have his book for sale and this must be fixed.
John Payne (satyr) Sun 19 Nov 00 09:52
So, Cory, what's all this hubub about peer-to-peer, how is it different from host/client, and what's cooking at openCOLA?
Life in the big (doctorow) Sun 19 Nov 00 16:35
Thanks, (viv) -- it's great to hear that I can make openCOLA sound sensical! -- (satyr), that's a giant question! I'll tackle it in pieces: some now and some later. First of all, let's start with what's wrong with the acronym P2P. I wrote a piece of Mindjack where I sum this up: http://www.mindjack.com/feature/p2p.html >The worst day in short history of modern peer-to-peer computing >was the day that some wag coined the banal acronym "P2P." On that >day, an entire field of wonderful, mind-bending, world-changing >networking models were reduced to a single, oversimplified >acronym-with-a-two-in-the-middle (AWATITM). > >The other AWATITMs -- B2B, B2C -- were used to make simple ideas >(selling stuff to people and selling stuff to companies) >excitingly complex. Entire PowerPoint cosmologies were >constructed around B2B and B2C, and MBA programs rejigged their >curriculum to explain the theory and practice of AWATITMs. The >dirty suits grabbed ahold of the idea, pumped and dumped billions >of dollars in bad paper on the NASDAQ and then hastily declared >B2B and B2C dead. > >Turning peer-to-peer computing into P2P implies some common >lineage with B2B and B2C. It places peer-to-peer computing >squarely in the safe and comprehensible realm of the monetized, >commodified Internet of Superbowl ads and Times Square >billboards. But peer-to-peer computing isn't a business model, >it's a *technological* one. That's not to say that you can't make >money with peer-to-peer technology. (You can. People will. Lots >of it.) But peer-to-peer computing isn't a way of making money >per se, any more than packet-switching or error-correction is a >way of making money. > >What peer-to-peer computing is, is a way of making really cool >decentralized networks. In a peer-to-peer universe, servers are >irrelevant. Peer-to-peer networks are composed of personal >computers tied together with consumer Internet connections, each >node a quantum zone of uncertainty, prone to going offline >whenever its owner closes his laptop and chucks it into a >shoulder-bag. In peer-to-peer computing, reliability is ensured >through massive redundancy; bottlenecks are dealt with in the >same fashion. Peer-to-peer networks aren't owned by any central >authority, nor can they be controlled, killed, or broken by a >central authority. Companies and concerns may program and release >software for peer-to-peer networking, but the networks that >emerge are owned by everyone and no one. > >They're faery infrastructure, networks whose maps form weird >n-dimensional topologies of surpassing beauty and chaos; mad >technological hairballs run by ad-hocracies whose members each >act in their own best interests. > >In a nutshell, peer-to-peer technology is goddamn wicked. It's >esoteric. It's unstoppable. It's way, way cool. That's the rant on why P2P is a stupid acronym. As to why it's good, well, here's whaqt makes it awesome for openCOLA: When we started out, we were thinking that oC would run on a centralized server, and that the distributed part would be the "spider" -- the part that fetches new pages from the Web and other resources to be integrated into a database and measure for relative relevance to various user-profiles. But when we started adding correlative features -- the ability to search for a robot based on your "relevance-proximity" to other users, the ability to automatically discover relevant material from correlated users, etc, we ran into an N-P Complete (or at least N-P Hard) math-problem. These are classes of problems that require a nearly infinite amount of calculation to solve. When we projected the computational horsepower required to do real-time correlation of say, two billion documents to, say, 100 million robots, we realized that we'd need an ungodly amount of hardware -- way more than we wanted to buy, and even then, it didn't look like it'd work all that well. But with self-organizing, peer-to-peer networks, we can make horsemeat out of N-P Complete. Think of a simple analogy: You have a thousand people in a subway station, and you want to arrange them by height. The server-side, top-down approach would be to take a tape- measure and measure then entire crowd, sort them out with a spreadsheet, then put them all in order. It's do-able, but just barely, and impossible if everyone keeps growning, shrinking, coming and going. A peer-to-peer approach might go like this: line everyone up, and give them this instruction: if you're taller than the person on your right, switch places. Keep repeating until everyone's in order. This is a really crude algorithm, just a bubble-sort, the kind of thing you're exposed to in a high-school computer-science course. But even with something as inefficient as bubble-sorting, a self-organizing group will quickly find itself standing in something like order. Further, with every "computation" (place-swap), the system approaches order. In the top-down approach, order only happens once the entire calculation has been executed. So that's why distributed computing works so well for openCOLA. As to what's cooking, give me a couple hours, I'll get back to you.
Konsigliari Kafka of the Cosa Nozzo (kafclown) Sun 19 Nov 00 18:18
This reminds me of a clown/theatre exercise I teach. People walk in a room at various speeds. They don't walk in circles, but they make sure to change their focus and their rhythm of walking, and their destination. They also make sure not to bump into anyone else. I ask everyone in the room as they are walking to pick out two people to keep track of-- two people who they will know where they are at all times, they are always aware of where their two people are. They are not to let anyone know by word or movement who those two people are. When every person has picked two people, I ask them to form an equilateral triangle with their two people. An equilateral triangle in space, so that there is equal distance between the two people you have picked and yourself. If one person moves, you must move yourself to keep the unity of the triangle. The point is to hopefully find a balance. If the group can do this consistently (which is very hard) then the point becomes to have the entire group move in space without losing the sense of the triangles. It's a very complex exercise...
Life in the big (doctorow) Sun 19 Nov 00 18:51
And impossible to manage from the top down.
Konsigliari Kafka of the Cosa Nozzo (kafclown) Sun 19 Nov 00 20:16
Exactly, it requires setting the people in motion and relying on their skill guided by their experience in order to make this fragile exercise work.
The music's played by the (madman) Sun 19 Nov 00 20:29
Here's a question that I've been contemplating, ever since I read the OpenCola web pages a few weeks ago: Do you expect to have to overcome paranoia concerning the running of servers on the client computers? I _know_ it's open source, and so I know that there will be people checking to make sure there are no backdoors, trojans, etc hiding in the code... but I would suspect that people would fear it anyway. After all, there are security holes even in good open source projects- the nice bit isn't that they aren't there but that they are found and fixed. So, yeah. Do you expect to have to overcome a certain amount of user paranoia, and do you have a plan to do so? (I almost asked this question of slashdot, though in a more general form and using OpenCOLA as an example...) (not that I expect they would have posted it.)
Life in the big (doctorow) Mon 20 Nov 00 05:17
The short answer is that yeah, oC is open source, yeah, we've retained the services of @Stake (formerly 10pht Heavy Industries) to do code- and architecture-level security reviews -- we're not relying on the good will of the open source community to audit the thing for us -- but in the end, people weigh security against benefits. All kinds of applications open security vulnerabilities on one's computer, starting with connecting it to the Internet at all! People run Outlook, Windows, Instant Messenger, Napster, Gnutella, and a thousand other network apps, despite the well- publicized risks inherent in each. But there are tremendous benefits to running network-aware apps, and these benefits outweigh the risks. Of course, we believe that we've embraced the critical success-factors for a secure app: open code, security audit, eschewing perishable, crypto-based copy-protection in favor of reliable network-audit based Digital Rights Management.
Brian Slesinsky (bslesins) Mon 20 Nov 00 14:41
I've got a thick book at home called *Distributed Algorithms* that goes into things like how to get consensus among peers even when there are network failures. It's tricky stuff and tough to get right - heck, even two-phase commit has a problem with in-doubt transactions if you get a network glich at the wrong time. Seems to me that any distributed algorithm can be greatly simplified by adding a central server, even if it doesn't do all that much work. Given all this, what problems are there to be solved that make a truly distributed network worth all the trouble?
Brian Slesinsky (bslesins) Mon 20 Nov 00 14:44
Oops, I should have said network outage rather than a network glich - temporary network gliches are handled by TCP/IP.
Life in the big (doctorow) Mon 20 Nov 00 22:22
Brian, that's a tough question. I don't want to shoot off my mouth about a subject on which I'm not an expert, so I've passed this on to our research scientists, Ethan Davis and Dan Moniz. I expect they'll get back to me tomorrow some time, and I'll post the response tomorrow.
Michael R. Walsh (mrw) Tue 21 Nov 00 12:29
Trying to look prescient, here, but I'm betting the answer has something to do along the lines of "that depends on how you define 'answer'". To acquire a truth, or universal consensus, probably would be greatly aided by and might even require a central store to hold all of the info. But oC (as I understand it so far) works more on the gossip model. You don't need all the info everyone has when you figure that most of the local clusters have most of the information. Since the 'answer' is "what is the cool thing you know about right now" and not "what is the most maximally cool thing I could ever hope to know", oC gives good answers right away and better answers as the knowledge disseminates. It seems pretty hiccup-free to me. (Cory is of course being responsible by deferring your answer to his experts. I am practicing for the role of media pundit by declaiming on a subject without benefit of actual fact.)
John Payne (satyr) Tue 21 Nov 00 19:45
O'Reilly has a recent (11/14) interview with Ian Clarke of Freenet fame up at... <http://www.oreillynet.com/pub/a/p2p/2000/11/14/ian.html?page=1> ...from the 3rd page of which there's a link to a directory of peer-to-peer projects.
Life in the big (doctorow) Wed 22 Nov 00 06:41
Brian, here's an answer from Dan Moniz, one of our research scientists (who aslo sternly reminds me that our centralized database problems with O^N, not N-P Complete): -- From my own perspective, calling what we intend to do, and indeed, what everyone else in the space seems to be doing "p2p" is a limited, extremist view. Napster isn't *really* p2p because it uses an index. I think what's more powerful here is the idea of what what I like to call blank2blank. Here's another buzzphrase I just made off the top of my head that fits adequately: Dynamic hybrid network topologies. The whole point is more to get away from the de facto centralized nature of the Internet and start sharing more resources in an efficient manner, where approriate, which is what openCOLA is about. Mark Miller, who I talked with face-to-face for the first time while out in California recently, and who I have been communicating with on various projects longer than I've been with oC. Mark is behind interesting things like agoric computing and distributed capabilities and he had an interesting take on p2p. He likened it to an airport/airplane metaphor: technically, any airplane can fly to any airport, but in the real world, because of population density, the amount of business conducted in a given city, remoteness of a location, etc. we find that some airports are naturally busier than others. So it isn't so much c2s (client- to-server) v. p2p necessarily, it's "hub & spoke". Hybrid models, constructed as necessary. And sure, openCOLA is in the p2p space. We're in the c2s space too. We're in the rabbit2ocelot space! We're trans-spatial, which is hard to label and pigeonhole, but is pretty critical. The whole point is to make the network layout change dynamically with the needs of users on the network -- witness the grouping power of networklogically close peers via the relevance engine, for instance. So yes, servers, even that do very little, can un-complicate a lot, agreed. But what's potentially more at issue is the ability for people to be able to determine their own involvement by participation. *That's* new. So people can set up a root clerver for their exclusive collection of Doctorow's Best Fart Jokes if they like, all the while running other clervers that host robots scouring for entirely different media, sharing nothing. It's an agoric computing scheme with a market based around reputation, in turn based on what you offer to the *community* (another key oC concept), so people that provide will find themselves better off than those that don't (just like people who maintain jobs v. homeless -- although there are obvious social/political variables there).. And for what it's worth, we do use servers, as an optimization, for instance, in _____'s [top secret project name omitted] current design. They're not *necessary*, but they help. As for why it's worthwhile, for me, it's a problem space with a lot of interesting possible solutions and a lot of smart people working on those solutions, with enough space for me to make an impact for the positive (I hope). If no one ever did anything simply because it was difficult, well, we wouldn't have gotten this far, that's for sure. Also, I think the fact that so many people have adopted p2p and related technologies thus far says something, even if all it is is "hey, we like this stuff!". We need more software the world likes. So it may or may not be terribly efficient. I think that c2s breaks down in a number of circumstances where p2p succeeds, and vice versa. For me, it's not about being a fundamentalist on either side of the coin, it's about evolving the state of the art. -- Dan's at email@example.com. He's a good guy to talk with if you're looking for more detail.
Linda Castellani (castle) Wed 22 Nov 00 15:52
If we get any more technical, we are all gonna have to sign NDAs!
Cynthia Dyer-Bennet (cdb) Wed 22 Nov 00 18:07
Cory, can we back up a bit to what you said earlier: > But peer-to-peer computing isn't a business model, > it's a *technological* one. That's not to say that you can't make > money with peer-to-peer technology. (You can. People will. Lots > of it.) If you already answered this, I missed it and I apologize. If not, just HOW can people make money with this? "People" being the openCOLA operation, in this case. Though I only have a very hazy comprehension of what you're proposing, I assume there's equipment, labor and a physical site where the equipment and the labor is housed. There's got to be some overhead, right? How does openCOLA generate income off this?
Alan Thornton (sd) Wed 22 Nov 00 22:32
AND...do you see individuals being able to sell their unused computing power when they are on high speed connections 24/7 and only using for an hour or two a day? Is it like taxes where the real burden is carried by the broad middle class? Isn't the greatest computing resource all of this unused potential? ALSO...just reading about beowulf clustering in Wired isn't this a pretty good idea for all those unwanted computers we're reading about having to pay a charge to send to the landfills? www.beowulf-underground.org.
Life in the big (doctorow) Thu 23 Nov 00 00:59
How does oC make money? That's actually pretty straightforward. oC finds stuff, moves stuff, provides for digital rights management on that stuff, and a transaction engine so that the authors of said stuff can charge for it. oC makes the bulk of its revenue from fees taken out of the transaction. This is the classic relationship between a sales-agent and a creator: the sales-agent works on a commission, only making money when the creator does. As to cycle-brokering, that is, farming large, computationally intensive tasks out to consumer machines for crunching, that's an application that *could* be built out of oC's kernel, but it isn't something that we're working on at the moment. There are a number of reasons for this: * A number of very good projects have already sprung up to fill this niche, particularily distributed.net (my favorite), SETI@home, Cosm, and on the commercial front, Popular Power. * Not many jobs are that computationally intensive: when was the last time you waited for a spreadsheet to recalculate? Jobs that *are* suited to such distribution are either noncommerical (scientific research), or confidential (insurance company reports, Postscript RIPs, rendering). As such, this just doesn't seem like a huge opportunity to either make a bundle or change the world. I *love* Beowulf clustering. Some day, I'd like to get myself a few hundred free 486s, put Linux on them, and use 'em to build the world's first gigabyte floppy-disk array.
Alan Thornton (sd) Thu 23 Nov 00 06:19
thanks for the pointers. don't know how i avoided knowing about distributed.net so far.
Members: Enter the conference to participate