Here’s the link to last night’s interview on Brian Lehrer Live. Here’s a screengrab, as well:

Boy, I look serious talking about hyperlocal blogging. I thought I was smiling that whole time - apparently not! This is very very serious stuff, let me tell you…

Two days ago I spoke at the Politics Online conference in DC, on a panel about online mapping. Then last night Steven and I went on the Brian Lehrer Live show, talking about outside.in. Both were really fun - I love the process of live Q&A, having to think on your feet and come up with sharp replies on the spot. I have to get more comfortable with the live TV format - it’s a bit daunting to know that you only get one chance to give a clear and interesting answer a particular question. Even on a panel in front of hundreds of people you can always say “wait, let me rephrase that”, but on live TV you feel like I’ve got to make this make sense in fifteen seconds flat, and no re-takes. The key of course is to be natural and not even think about it, and that comes with familiarity.

I had a moment this morning right as I woke up that reminded me of the morning after a first date (back in the days when I used to date), where I thought to myself, “oh my god - did I *really* say that last night? what the hell was I thinking??” But it’s all just fun and laughs at this point. You just have to accept the gaffes along with the good answers - it’s all part of the game, and that’s what makes it so much fun to play.

Innovation is the creation of new problems. Or rather it is the embodiment of a radical new idea that forces people to reckon with it, and that leads to new problems, or questions, for people to solve. Improvement is then the solving of problems that follows innovation. All of that improvement built on top of innovation translates to progress, or the creation of value, or whatever you want to call it. Most of what we do is improvement, very little is actual innovation.

Why do we care?

Well, when the mixture between innovation and improvement is just right, you have a healthy system of growth and new idea generation. It’s exciting to be a part of such a system.

But when the mixture isn’t right, when there’s not enough innovation to keep up with all of the improving that is endlessly happening, or when the innovation that is occurring is not of the big, paradigm-shifting sort but is of the minor, isn’t-that-nice sort, then you get problems. Things start to stagnate. People keep working on the same problems over and over, trying to eek out improvements, but they start to see less and less return on their efforts. Left alone, without being given something new to work at, some new idea to improve upon, their efforts at improvement will eventually yield no return at all - no improvement. Beyond that, if they continue, their efforts will actually have negative results.

Improvement of innovation over time yields diminishing returns.

As an example, think of the steam engine. The steam engine was an enormous innovation, and after its invention it was improved upon for years and years in innumerable ways. Today nobody bothers to improve on the idea of the steam engine any more, because newer innovations in transportation have allowed people to turn their focus elsewhere. The steam engine is obsolete. But imagine if newer innovations had not displaced the steam engine - imagine if people were still improving on the basic idea today. What would those improvements be like? The answer is that they would be very difficult and expensive to come by, and the value they yielded would be very small, especially compared with the value of much earlier improvements. Improvement, over time, yields diminished returns.

Luckily innovation has moved us beyond the steam engine, and nobody is faced with the problem of trying to improve on an idea that has been essentially perfected.

There is a book that I read a while back, called The Collapse of Complex Societies. In it, the author, an archaeologist, argues that every complex society ever to have existed has eventually collapsed under the weight of its own increasing complexity. As societies grow, they naturally become more complex, and complexity beyond a certain point yields decreasing returns on investment. Constant improvement over the tools by which you keep society operating smoothly becomes costlier and costlier, and yet gives you less and less in return. Past a certain point, it isn’t worth the effort to keep up the work. The work you put in is greater than the reward you get out. When this happens, societies collapse - they revert to a simpler form, with less complexity. That, in the end, proves to be more profitable for the people in question.

The only thing that offsets this ever-decreasing return on investment for societies, which would otherwise lead to collapse, is innovation - the introduction of new ideas that give them new ways to meet their challenges. Those new ideas, being so rough and new, also make improvement very cheap and rewarding. So people can set to work solving problems and improving profitably again for a while before diminished returns set in once more.

Innovation then, at the macro level, is literally forestalling the otherwise inevitable implosion of society, keeping its means for dealing with all of its challenges cheap enough to be worth the effort.

At the micro level, which is the point from which I experience things, it just makes the world a more interesting and fun place to be from day to day. Solving big, rewarding problems is a lot more exciting than solving small problems with little reward.

Whichever way you look at it, we need difficult ideas, ideas that force us to question everything we take for granted. We need big, new problems, a continual stream of them, to keep us engaged, and to keep our work rewarding. We need this more than we need improvement, really. Improvement is essential, but it is also a given - we are creatures of improvement, we do it naturally and consistently on whatever is placed in front of us. Innovation is more of a wildcard, a mutation - it happens sporadically, not necessarily when you need it and are hoping for it.

Which of course makes us love it all the more.

I stumbled on a definition of innovation by chance the other day, and it blew my mind. It said:

Innovation occurs when someone uses an invention - or uses existing tools in a new way - to change how the world works, how people organize themselves, and how they conduct their lives.

then:

Many product and service enhancements may fall more rigorously under the term improvement (than under the term innovation).

Innovation is distinct from improvement in that it causes society to reorganize. It is distinct from problem solving and is perhaps more rigorously seen as new problem creation.
[emphasis mine]

Innovation is the creation of new problems in need of solving. I’ve always agreed with that idea, without ever knowing it consciously. Improvement, then, is what comes after innovation. It is the solving of the problems created by innovation. Innovation is a radical leap in thinking, improvement is the linear filling in of the spaces created by that leap. The two are very different, though they go hand in hand.

It’s a distinction that is for the most part lost on people. Does it matter that we understand the difference between innovation and improvement? I think it does. I’ll try to write something on why it matters soon.

In my last post, I pointed to two interpretations on the state of the web here in 2008. Both were pretty good takes on things in their own way, yet each was wildly different in its assessment. Ever since, I’ve been kicking these around in the back of my mind. And by looking through them both, together, I’ve had a breakthrough in my thinking on where the web is now and where it’s going. Like those stereoscopic 3-D glasses with one lens red and the other blue, considering the two angles of Doug Rushkoff and Fred Wilson at once has given me a three dimensional view of the topic. The result has been a sense of the history of the web, as we are now experiencing it and moving through it.

Rushkoff, talking about his disappointment with the web today, said, “In the 90’s, I thought the experience of going online for the first time would change a person’s consciousness as much as if they had dropped acid in the 60’s.” Speaking of the web now, he says, “”Sadly, cyberspace has become just another place to do business.” Yet Fred Wilson interprets this same modern web optimistically like so: “The web is becoming more open, more mobile, more social, more playful, more intelligent, and more monetizable every day. Happy new year everyone.”

What’s going on here?

Well for one, Doug is a cultural critic and Fred is a VC. But I think there’s more to it than that.

I think Doug is right about the web in the 90’s. It was the equivalent of acid in the 60’s. It changed people. People turned on to it and got swept away by it. Everyone was suddenly talking about 24.4 modems with their friends and having email conversations with people they had never met before, half a world away. It was intense, an overnight paradigm shift. Everyone I know who got into the web got into it because they wanted to be a part of that shift. I got into it for that reason too.

And so if we go with Doug’s interpretation of the nineties as the 60’s of the web, what then, by natural extension, are we living through now?

That’s right - we’re smack dab in the middle of the seventies of the web.

As Fred points out in his post, today there is more innovation on the web than ever. More people are working on it, taking it in more directions than ever, and there are more opportunities for making money in it than ever before. But what’s missing from it is any sense of adventure, of boundaries being pushed, of history being written. The web is all grown up and has taken over, in the same way that rock and drug culture took over as people moved from the sixties into the seventies. No longer controversial, no longer bent on changing the world, it is now the dominant paradigm. Here to stay, mainstream, and content to operate within established conventions.

In rock and roll speak, we have moved from the clubs to the stadiums. The era of Arena Rock is now upon us on the web: huge bands, improving on the ideas of the smaller bands that came before. But not much in the way of radical new thinking.

In this 70’s of the web, Facebook is Peter Frampton: hugely popular, speaking to and understood by everyone in the requisite age group, reinforcing rather than challenging shared values, bland and predictable by comparison with what came before. (The widget, or the Facebook app, is then perhaps the modern equivalent of the gratuitous guitar solo of the seventies: the central pillar that gives order to the whole system: no song - or website - can be without it).

Users of Facebook are by extension the Me Generation of the Internet: not into it because it’s mind-expanding (it isn’t, anymore), they simply want to use it to hang out with their friends and have a good time. They’re the younger brothers and sisters of the people who created the web, and they see it differently than their older siblings, who thought they would change the world. Sure, they may do the same things on the web that their older brothers and sisters did. But whatever they do, they do without greater expectations.

We live in a moment right now where everyone is more or less on the same page in terms of thinking about what the web is and where it’s going. We all hold the same assumptions, and nobody is challenging those assumptions. Like rock in the seventies.

Doug Rushkoff likes radical new thinking. He looks out on the seventies of the web and thinks “Who are all of these kids using Facebook and Twitter? Don’t they realize that the real web, the web that mattered, was the WELL in 1996?”

Fred is a business man and an enthusiast. He is the Bill Graham of the web. He looks out on the seventies of the web and thinks “Woo-oooh! We’ve arrived. This kicks ASS!” He can take or leave all of the paradigm shift stuff. He just wants to see bands, lots of them, going big.

Fred is more suited to life in the seventies than Doug.

As for me, I’m a bit like Doug and a bit like Fred. I can appreciate arena rock and all, but I really like the radical leaps in thought. Which is why I take heart from realizing where we are right now in the history of the web. Because I know that by the time Peter Frampton was up on stage with his talk box, Joe Strummer, Joey Ramone, David Byrne and others were all kicking around in the streets somewhere getting ready to explode on the scene and force another paradigm shift on the world.

So I give us another two years, max, before the whole idea of the web gets turned upside down and the world goes crazy with new ideas.

Here are two comments that I just came across, from two people I like, wildly different in their interpretation of the state of the web going into 2008.

The first, from Fred Wilson:

The web is becoming more open, more mobile, more social, more playful, more intelligent, and more monetizable every day. Happy new year everyone. Let’s get busy because there’s so much opportunity out there I almost can’t believe it.

The second, from Doug Rushkoff:

I thought that (the Internet) would change people. I thought it would allow us to build a new world through which we could model new behaviors, values, and relationships. In the 90’s, I thought the experience of going online for the first time would change a person’s consciousness as much as if they had dropped acid in the 60’s (…)

Sadly, cyberspace has become just another place to do business. The question is no longer how browsing the Internet changes the way we look at the world; it’s which browser we’ll be using to buy and sell stuff in the same old world.

They’re both describing the same picture, from totally different perspectives. I think they’re both accurate, and somewhere in there, by looking from both views simultaneously, you can get a good sense of how I feel about the web these days.

By the way, as I write this I’m listening to Myriad Harbor by The New Pornographers. I always feel somehow like it’s a song about kicking around in New York City at the end of time. Optmistic, but conveying a sense that the clock has somehow run down somewhere. It’s a perfect soundtrack to those two discussions I pointed to above. And a good song for kicking off 2008.

I wrote in a post a while back that corporations are social machines that exist for the sole purpose of making money. Because of that, I said, you can’t expect corporations like ExxonMobil to bother developing new sustainable energies, because they’re already making tons of money selling old, non-sustainable energies. And that’s all they ultimately exist for. To ask them to do anything else is to misunderstand the nature of the machine.

Now former labor secretary Robert B. Reich has written a book on this subject, Supercapitalism, arguing that we should see corporations for what they are, and not anthropomorphize them. The book “urges new and strengthened laws and regulations to restore authority to the citizens in us” and argues that we ought to “separate capitalism from democracy, and guard the border between them” (quotes from a Publishers Weekly review on the Amazon page linked to above).

Haven’t read the book yet, but it looks like provocative reading. Will have to check it out over the holiday.

I’ve been reading about last week’s meeting in Bali on global warming and what’s to be done about it. The meeting was attended by delegates from 187 nations and was a the beginning of a follow-up to the Kyoto Protocol, which happened ten years ago.

After two weeks of raucous debate, the group produced an Action Plan which contained no binding commitments. The boldest thing members of the meeting could agree on was to declare that “deep cuts in global emissions will be required”.

This seemed like the time to point out what is becoming undeniably clear: if the problem of global warming is going to be solved, it will be solved by entrepreneurs, not by governments.

Why? Here are a few reasons:

1. Entrepreneurs look at situations and see solutions. Governments look at situations and see constituents and donors.

Entrepreneurs exist to find problems and solve them in never-before tried ways. Governments on the other hand exist mainly to maintain the status quo - keep people’s jobs, keep companies’ profits up, keep infrastructure functioning, and keep donors’ revenue streams open. So which of these groups is better-oriented to solve the very complex problem of global warming?

2. Entrepreneurs never rely on consensus to get their idea to work.

Any plan that requires global consensus to work is doomed to failure from the start. The Bali Plan got held up for most of the meeting by the U.S. delegates, who refused to play along with the rest of the group. Entrepreneurs plan for adversity, build it into their game plan. They realize that consensus only emerges after the fact, after a particular model is in place and has been proven. Any plan for dealing with global warming will have to take this approach in order to be successful.

3. Entrepreneurs cycle through ideas quickly (governments don’t)

Entrepreneurs are quick to come up with ideas, but also quick to abandon those ideas when they they aren’t panning out. Give an entrepreneur a budget for two years and a small team, and they’ll cycle through a dozen ideas, find the one that works best, and implement that. Governments in the meantime will still be discussing what language to put in the Action Plan…

4. Entrepreneurs know that people have to like whatever they’re selling.

Everything in the world is a pitch, whether it’s global politics, talking to constituents, talking to businesses, or whatever. If people don’t like what you’re telling them, they aren’t going to buy in. You have to find a solution you can sell people on, but that still gets the job done. Entrepreneurs think naturally in these terms - they think it in their sleep.

5. Entrepreneurs trust in their own resources.

Unlike the group of 100 scientists who just wrote to the UN Secretary General pleading with him to accept global warming as unavoidable, entrepreneurs believe in their own abilities and the abilities of those around them. The spirit of NASA’s “let’s get to the moon in ten years even though we have no idea how we’ll do it” attitude is alive and well in the entrepreneurial world. It sure doesn’t seem to be alive in the government these days.

These are just the first few to come to mind. There are probably several more that I’m omitting.

I don’t mean to sound like one of those pro-business/anti-government types. In some ways, governments can’t be blamed for their failure to act on the problem of carbon emissions. At the root of the issue is the need for the world to evolve beyond its current processes, and you can’t legislate evolution. You can only innovate and iterate your way there. People, and governments, need to realize this and do as much as possible to create favorable conditions for innovation to occur and be implemented. That, and not plans and consensus, is how the problem is going to be solved.

I would love to assemble a group of entrepreneurs in the same fashion as the Bali talks that just ended, and see what ideas they could come up with for putting the world on course for lower carbon emissions. I would bet anything that they could do more in one week than the world’s governments have done to date.

Anyone want to take me up on the challenge?

I’ve come in contact with more and more people who work with Yahoo lately - Matt McAllister of the Yahoo Developers’ Network, Tom Coates and the FireEagle team, Bradley Horowitz, VP of Product Strategy, who was at the Open Government Data meeting but left before I could talk with him, and Havi Hoffman, who works on next.yahoo.net. Each time I speak to one of them or hear them talking, I’m totally blown away at how much they get it. I mean, they totally get it.

But that’s so out of step with the image Yahoo projects as a whole. Yahoo, the company, projects a muddled front. What are they trying to be? What do they represent? What is their philosophy? They’re a bit like Pepsi - the only coherent part of their image is defined in terms of what they’re not.

Yet this group of people I keep coming across at Yahoo seem to really know where the web is going, and what Yahoo’s place in it could be. If you took them and made them into a brand-new startup, you’d no doubt have some really amazing product on the market in a few months. When I talk with them I think, wow, Yahoo is really on to something.

So is Yahoo on to something or not? I kept trying to decide. Then it dawned on me that the situation is this: Yahoo is a giant company dominated by an old-timer culture. That culture has been there since 1996, and it is deeply entrenched. It thinks of progress in terms of getting more pageviews to sell more ads, and nothing more. Living in the shadow of that culture though is a subculture of people working to radically reinvent things. They see where the web is going to be in five years, and they’re working to drive Yahoo in that direction.

And what comes out of Yahoo is the result of these two forces blending together. Unfortunately I think that the old-time culture is the one steering the ship for the most part. But it seems like there’s a chance that could change, and the subculture of reinventers could be given leeway to drive the ship. If that happened, it would be like that hypothetical start-up I described above suddenly having a huge budget and gigantic team on their hands to do what they want. That would make Yahoo into a really interesting, strong brand. It would also make the Internet a more interesting place.

So could that happen, or not?

I just got back from attending a two-day meeting on Open Government Data, organized by Carl Malamud of public.resource.org and hosted by Tim O’Reilly. The focus of the meeting was on making government — all governments, at all levels — as open and transparent as possible, with the use of the Internet. It was an exciting discussion to take part in. There was a feeling at the meeting, shared by most I would guess, that we were giving shape to something that could ultimately help to modernize the notion of democratic, representative government.

The reasoning behind Open Government Data is that society has been transformed by the openness of data afforded by the Internet. Open, equal access to information is good for the general public, is good for competition, and is good for the field in question. The political process, however, has been left out of this equation. While access to information of every sort has exploded in recent years, access to government data — non-sensitive, non-priviledged data — is still generally very hard to come by.

That needs to change. By opening up government data, making it freely accessible and allowing it to be downloaded, forwarded, mashed and so forth, the political process could experience the same sorts of benefits that other sectors now experience with increased dataflow.

What would opening up access to government data do? Several things:

For starters, the political process would become immediately more transparent. You could easily see who was taking money from whom and so forth. Everyone should agree that would be a good thing, right?

Next, a big chunk of government data would become part of the public record, available for indexing by search engines and such. Imagine if you could Google “how did my town council vote on that housing development last week?” and see the actual minutes of the meeting.

Finally, and most interestingly, a whole layer of added value would emerge on top of government data as public entities mixed, mashed and mapped that data however they chose. In the same way that Google Maps mashups added value to the web, allowing people to see things they hadn’t before, opening up government data would result in new and valuable ways of seeing that information. Groups and individuals would be able to use government data to find patterns in it, detecting trends, linkages, surprise success stories, or stark failures in their government and reporting on them. And that value would feed directly back into the political process itself.

Open Government Data has never been tried before because it wasn’t economically realistic — the cost of reproducing information was too high. Times have changed, and the cost of reproducing data is now virtually nothing. Government ought to keep step with that change, and reap the benefits. The great thing about the idea is that it ought to have 100% bipartisan appeal. Open data is good for everyone, not just liberals, not just conservatives. It opens the playing field up for all.

Here is a draft of the Open Government Data Principles that was produced at the meeting over the weekend. It still has a few can of worms to be dealt with, as Ethan Zuckerman was keen to point out. Overall though it’s an exciting proposal.