This is a short book more than an article. I'll continue to revise it over time, according to your comments.

Warning: Indulgences Ahead

If you wanted a brief, snappy piece on the meaning of life, skip this one. I'm writing for my own pleasure this time. This is less of a blog post, more of a novella. And it's all about me, the smartest guy in the room, always with an answer, almost impossible to beat in an argument.

This isn't an autobiography, I'm not going to talk about my family life, or our idyllic childhood years spent in Dar es Salaam. I'm not going to explain why my first languages were Lingala and Swahili. I'll spare you my school years, spent in the cold, dark embrace of the Scottish highlands.

Instead, this is a professional road trip, for my fellow programmers. OK, not entirely.

My first software products ran on a computer with 5,120 bytes of RAM, 1,536 of which were dedicated to storing the contents of the screen. If someone tells you that you can do a lot in 3.5KB of RAM, they're lying. You can't do shit with that. However my next computer came with 64KB of RAM, at the same price. You try filling that up, one assembler instruction at a time. And that's when I realized, these things aren't toys any more. They can do real work. They can make real money.

I've worked on a lot of different systems. More, I've worked in the weirdest possible projects, with crews of all sorts. There has always been a thread of insanity in our business. We're hooked on doing the impossible. Clients lie to themselves that they've hired a team that know what they're doing. The team lie to their clients that they're in control. The sales people lie. The marketing department lies.

Most of the time, most large software projects fail, and this is still true in 2016. Yet for most of my career, my special talent was to make projects work, no matter how impossible the technical challenges. I am a really good technical architect, able to understand systems at the lowest and the highest levels. I can write all the code myself, or I can explain to people exactly what to make, and it all fits together like laser-cut blocks.

As I wrote much later in "Social Architecture,"

I've come to think that the very notion of individual intelligence is a dangerously simplified myth.

It took me decades to realize that technology is a slave to personality. It doesn't matter how good the design, when there are unresolved problems in the organization.

And so gradually I shifted from technical architect to social architect. From caring about technical designs to caring about people and the psychology that drives them. Because in the end, this is what seems to make the difference between a working project and a failure.

The Microcomputer Era

I didn't plan on becoming a programmer. Age six or seven I wanted to be a writer. I had Roget's Thesaurus, and read it like a novel. Words, each a story, intertwined in endless trails. Eight years of boarding school beat the dreams out of me, and at 17 I dutifully collected good exam results and started thinking, what should I study at university?

My career advisor suggested computers as a lucrative option. Shrug. Seems fine. Cambridge offered a computer science course, yet my exam scores weren't good enough for that. Second best rated was York, which had a computer science + mathematics mix. I applied, went for an interview, and was accepted.

English universities offer a 3-year BSc course. I was a lousy student. The maths numbed my brain. Endless theory about encoding, Turing machines, database normalization. And then finally, a chance to write some code. Work out the PDP-11 assembler code on paper. Enter it by hand using console switches. Run it, and get boxes to move around on the screen. Yay!

About this time the Sinclair ZX80 came out. It was a ridiculous thing, tiny and light, with a horrid rubber keyboard. Yet it was real. I'd go with my friends to the store, and write small BASIC programs on the spot. Here, a game that let you chase a star around the screen with the arrow keys. People would watch me write these, and then ask if they could play.

For two years I struggled through subjects that grabbed my attention like a slug grabbing a basket ball. My tutor was patient and miserable with me. My failing grades just piled up, academic debt. There was no repeating a year. I could change to a different degree if I wanted to. Instead, life gave me a better option.

In the spring of 1981, the VIC 20 hit the shelves in the UK. I asked my mother if we could have one. We were not wealthy, and the computer was more expensive than the Sinclair. Yet it was a different beast, with color and sound and potential. She believed in me, as she has always done, and somehow found the money for the computer.

At first I copied games from magazines, typing them in — as we did, in those days — and then playing them. With a VIC cassette recorder, it became possible to save and reload them. Then I started improving the games, adding graphics and sounds. The VIC chip was powerful, if you were able to figure it out. Then I wrote my own games, mixes of assembler and BASIC. And then I started selling these.

There is a kind of shock, the first time you make something with your own hands, and exchange it for money. I bought empty cassettes, drew and copied inlays, and cranked out my games. A quarter page ad in one of the main magazines cost 125 pounds for three months. One ad impression would produce maybe fifty sales, at five pounds each. When I had a new cassette I'd send a mail flyer to existing clients.

By the summer of '81 I had sold a few hundred games, and attracted the attention of Commodore, who sent me a shiny new C64 free of charge. Computer, disk drive, monitor, and printer! I called my tutor and said, look, Bill, I'd really like to take a year off university to write and sell video games. That OK with you? Sure, he said, and did the necessary.

And so it was that I started my first business. The C64 had vast capabilities, poorly documented. I worked through the reference manual until I knew it by heart. There was an assembler cartridge yet it was clumsy, so I wrote my own assembler tool. And then I started writing a new game. Just a standard shoot 'em up, yet crazy fast and fun, and I was coding day and night to get it finished.

At the time, there was nothing in the market for the C64 except the old nasty BASIC games hastily ported to the larger screen. So I was working like a crazy to get a large-scale production line working. I'd designed and printed full color inlays, fifty thousand pieces.

Putting the cart firmly before the horse, I'd even written a natty copy protection scheme that I was so proud of. Your standard cassette held a BASIC program that you would LOAD, and then RUN. You could as well SAVE it again, to a blank cassette, and people did this a lot. It meant that a popular game get widely shared among computer clubs and high schools. God forbid!

The standard LOAD/RUN thing also seemed clumsy, and I wanted to make it smoother for the player. I'd found that you could format the saved data so that it loaded into a specific address in memory. I crafted a block of data that loaded itself into low memory, and overwrote the stack. When the LOAD routine finished, it popped the address off the stack and hence jumped into my code, which was a special loader. That pulled the rest of the game off cassette, loading it into memory, and then running it.

Many years later my friend Jonathan told me other C64 games used the same scheme, and he copied the games using a special cartridge that simply took a snapshot of memory when the game was loaded and ready to run. Pirates 1, Hintjens 0.

I was working with an audio cassette duplicator in Port Seton, and we'd cracked how to produce software cassettes at high speed. We could produce several thousand a week per machine. It was all ready to go, and then I made a fatal mistake that taught me my first big lesson about the corporate world.

Instead of running the ads and shipping out my software, I showed my new game to Commodore.

They loved it and told me they wanted to distribute it. At least UK wide, and possibly all of Europe. I was spell bound. My own sales would be in the tens of thousands. Commodore would be able to sell ten, a hundred times more.

So I told them "OK" and paused my own sales. Boxes of cassettes sat in my room. I did not run my ads. And Commodore told me they were working on it. And so I waited.

As I waited, other games started to appear. The large video game companies had finally gotten their act together, and were attacking the C64 market like a pack of killer whales attacking a young baleen whale. Full color ads promised everything. It became pretty obvious that my window had closed.

Commodore, I asked, where art thou? What the heck is going on? Finally they came back to me. "We're not going to distribute your game after all, thanks," they told me. Incompetent or malicious, who knows. I cursed them, and shut down my business.

As it turned out, it was also summer, and so I went back to university to finish my degree. At least I had a personal computer system to work on, and lots of experience in 6502 assembler. The 6502 chip was simple and fast, with a minimal instruction set. In some ways, a precursor to the RISC CPUs that dominate today's world.

My tutor Bill gave me Loeliger's "Threaded Interpretive Languages: Their Design and Implementation," and I decided to make this my thesis topic. The result was a lovely Forth-like language sat atop that brutalist 6502 assembly language. Fast enough for video games, and such fun to program! Every programmer should make a Forth at some point.

Sitting in my room coding day and night felt right. Whereas I'd spent my first two years avoiding hard work, now my brain was addicted to it. Vaguely, I realized there were courses I should be attending. Mostly, I ignored them, and they ignored me right back. My code got too large for the 170KB floppy drive so I stripped off the comments and carried on coding.

When it came to final exams, I sat with my friend Nigel and we skimmed through the course material, making notes. A few hours was enough to read a years' material, summarize it, and digest the summary. By luck our exams were always in the afternoon. It was a blur. One exam a day for two weeks. I came first or second in every one. The thesis committee asked me why I'd not commented my code and I shrugged. Did it matter? Look, let me demo you a couple of games, and walk you through the core threading logic.

The two years of fails brought down my overall degree yet in my entire career not a single person ever, once, asked me what I'd achieved at university. It was just never relevant. There's a big lesson here. Many of the top students in my course went off to Silicon Valley. I slid under the radar and moved, despite my best intentions, to Belgium.

Working for the Man

The Man got me straight out of university, for in those days we still had mandatory military service in Belgium. I'd lived so long in the UK, and was naturalized (I still have a UK passport), yet Belgium demanded its cup of blood, and it got me.

Military service was actually fun. In boot camp we were split into the intellectuals, and the lifers. People like Henri, a nuclear physicist, and myself wore the glasses and obviously had the nerd soft hands. The lifers were young professional soldiers, 16 or 17, who cleaned and scrubbed and marched and drilled. We nerds sat around the barracks, happily useless. They were not going to give us guns.

Ironically, I was already an excellent rifle and pistol shot, with prizes from Bisely, where my school shooting team had competed every year. I didn't tell them that, mainly because I spoke no French and no Dutch.

I worked at the national map making institute. They had a large project to make a digital map of all of Belgium, for the US air force. They carefully scanned in paper maps, tracing the railways, roads, city outlines, canals. The goal was to make maps for cruise missiles.

My boss showed me the team, their GCOS minicomputer, the IBM terminal running on a time-sharing system somewhere, and told me, "your job is to help us make the tapes."

Turns out the USAF wanted the map data on magnetic tapes. We could send data from the GCOS machine to the IBM, though I completely forget how that worked. Working in PL/I on the IBM, I could load in the map data and write a tape.

Ah, but there was a catch. I was not the first person to try to make tapes. The USAF systematically returned the tapes as "unreadable." After several years of mapping, the project was jammed on this one point. So being the smart young thing I was, I investigated. It turned out the USAF were using a UNIVAC system, which unlike the IBMs, the GCOS, and every normal computer in existence, used seven bits per byte instead of eight.

Magnetic tapes were coded with 8 bits in a stripe, perpendicular to the tape direction. These stripes were written like this "||||||||||||" along the tape. It struck me that the UNIVAC was probably just reading seven bits at a time, instead of eight. So the first stripe had one byte, plus one bit of the next (seven-bit) byte. So I staggered the bits like that, and we sent off a tape to the USAF.

It came back after the usual three or four weeks, with a new error message: "Bad data format." Bingo. Now I went back to the documentation and figured out what the real mapping data should look like. What we were sending from the GCOS wasn't even close.

Some weeks later (I got sidetracked into writing a small compiler on the GCOS that could access more exotic system functions and give me unlimited priority to run my heavy conversion programs) we had a new tape. We sent it to the USAF, and a few weeks later came the reply, "Valid."

At which point, all hell broke lose.

What I didn't realize was that the project had some years of budget left. The mapping was mostly done, and the team mostly idle, and happy with it. Lacking any way to send valid tapes, the Belgians simply kept pushing back the date, and collecting their sweet, sweet USAF cash. (I am speculating that it was sweet, because I was being paid a conscript's wage of 45 BEF or about 1.10 EUR per day.)

My boss told me I could take the rest of the week off. Actually, don't come in every day, or unless you feel like it. I shrugged and started my new regime of two half days per week.

There is a lesson here too. You need to follow the money, and understand how it flows. I could have walked away with my own personal GCOS system, if I'd played it smart.

The second lesson, which many a foreigner has learned in Belgium, is that it's an easy country to come to, and a hard one to leave. For reasons, I decided to stay in the place and find a job.

Still Working for the Man

I found a job at Sobemap, which was in 1984 the largest software consultancy in Belgium. I was hired by a large good natured man called Leif, who was looking for a fellow spirit to help him write software development tools. This was in the days of COBOL and weird systems. Sobemap had a commercially successful accounting system, EASY, which they maintained on seven or eight different platforms, with a team for each platform. Leif's job was to build the basis for a single portable EASY that could run on everything.

Turned out, though Leif only told me this a few months ago, that the main objection to hiring me was my age. Too young. I guess it was a high risk project and they hoped for someone with relevant experience. Needless to say, the number of people building cross-platform COBOL frameworks, worldwide, was approximately zero. We were the first to even think of such a mad thing.

It took only a few months to build the basic layers, and then my job was to make those work on new, exotic systems. I'm talking Siemens BS/2000, IBM S/36, DG AOS, MS-DOS (once we discovered the wonderful Realia COBOL compiler for the PC), and IBM MVS/CICS. The core routines had to be in assembler, to be fast enough. That was when I discovered C, on the DG and the VAX and then on the PC.

All these systems died, one after the other, over time. Good riddance. The inconsistency was incredible. Every machine had its own concept of file structures, organization models, compilation procedures, and so on. Imagine learning ten different varieties of Linux, each from a different planet.

No matter, we learned to dive under the consumer level OS and language, and into system internals. That was inevitably the only way to get the performance and behavior we needed. On a BULL TDS (short for tedious) system I wrote a memory manager, in COBOL no less, to fit large programs into the pathetic excuse for "main memory" that system provided. On IBM S/36, the same, in assembler, to swap megabytes of COBOL code and data in and out of the 64KB main memory.

Leif and me, helped by various people who came and left, used our portability tools to build a powerful set of tools: editors, code generators, reporting tools, and so on. Developers loved these tools. You could develop and test on a PC, at home, and the same code would run unchanged on an IBM mainframe. We had cracked the problem of "write once, run anywhere," some years before it became a fashionable problem.

The IT managers of our clients didn't share our passion for portable code. As one IT manager told us, "I've just spent so much on a new VMS system. Why do I want portability?" Logic doesn't apply, when people work on pseudo-mystical belief. A very few IT managers saw the potential, used it, and built their empires on it. Yet mostly we had few external clients and direct sales.

All this time, the company kept changing and shifting. The smart, nice guy who hired Leif and then me left for a startup. We were sat on by a succession of incompetent, arrogant illiterates. Despite delivering success after success, there were no pay rises, no bonuses. Eventually after five years, our team called it quits and we left the firm, each to go separate ways.

Sticking it to the Man

One taste of employeeness was enough for me. I did some research, found an accountant, and became an independent consultant. My first gig was for a trade union organization. They did not pay much yet they treated us with respect. Jonathan Schultz and myself built a project management system, using that awesome COBOL framework.

Somewhat adrift, I worked on various projects with old contacts. We built monitoring software for PCs for a UN transport project in Africa. Set-top clients and servers for a cable TV company. Software to produce and decode shipping manifests, using a prehistoric format called EDIFACT. Mostly I worked in C, using Turbo C on MS-DOS. And a lot of x86 assembler, using MASM, the Microsoft assembler.

Then I got a call from my friend Hugo at Sobemap, now renamed to Sema. "We need your help to make this project work, what's your daily rate?" I quoted a comfortable figure. They agreed. And so I went back to work for the same projects, earning five times more.

I learned one important lesson: the more your clients pay you, the more they appreciate you and listen to what you say. The same CEO who'd walked past me in the corridor as if I was a tattered old poster now stopped and shook my hand. "Glad to have you back with us, Hintjens!" he said.

And so I found myself back in the world of enterprise software projects, providing that special skill we never really had a name for in our industry. Plumbing. Infrastructure. Wet work. Black magic. Making it possible for mediocre developers to build really large applications that worked well even under stress.

It was 1991, and my first major project was to port our tools to VAX ACMS. The project was signed and sold before I had a word to say. The IT manager was one of those who knew Leif and me, understood our blood-and-guts approach, and loved it. He had bet his career and arguably his firm's future, on us succeeding with an impossible project.

So here was the challenge. The client wants to build a travel reservation system to serve tens of thousands of agents in offices across Europe. These agents would log into the system, query availability, make bookings, and so on. The backend for this system would be a cluster of modern, cheap computers called a VAX. These cost a fraction of the IBM mainframes that the industry still relies on. It has cheaper memory and disks than IBM's dinosaurs (though not as cheap as PC hardware). It runs a modern operating system (VMS) that is fast and flexible (though not as nice as UNIX). DEC's wide-area networking was decades ahead of IBM (though not as nice as TCP/IP).

The only problem is that IBM's mainframes have long ago cracked the problem of sharing one mainframe between tens of thousands of users. Whereas DEC, the firm that makes the VAX, has not.

If you want to get technical, IBM used smart 3270 terminals that dealt with all keystrokes locally, and interrupted the computer only when there was a screen of data ready. Somewhat like a clunky web browser. The mainframes then run a "transaction processing system" called CICS that pumps these screens of data through applications, with all state kept out of the process. One application can thus serve masses of users.

The VAX on the other hand used dumb VT terminals. These sent every single keystroke back to the computer to deal with. Each terminal, and its user, have their own process. It's the same model that UNIX and Linux use. Five terminals, five interactive "sessions", five shell processes. So VAX gave a much nicer flow, and you could do things like scroll smoothly (impossibly on a 3270). On the down side it could not deal with anything like the number of users at once. Each of those processes eats up virtual memory.

DEC's proposed solution was to use smaller front-end MicroVAXes, each capable of handling around 50 users. These would then talk via some magic to the back-end cluster. The client did some maths. Two hundred front-ends and ten thousand interactive licenses. That cost several times more than the rest of the project together. They might as well buy an IBM mainframe.

Which was when someone decided to bluff the project, sign it for the budget the client was willing to pay, force DEC to accept whatever solution we came up with, and then hire me to make it all work.

Since no-one had told me it was impossible, I took the documentation and a test system, and began to play with DEC's transaction processing framework, called ACMS. This was the closest DEC had to an IBM CICS-style transaction processor. Think of a container for server applications. You could start and stop apps, and then send them events from other processes, even across a network.

So far so good. Then I looked at VMS's asynchronous I/O system. To get consistent and fast response in a real time system, you have to work asynchronously. You can never wait on something happening.

I'm going to explain this in a little detail, because there are lessons here that still apply today. If you want to make truly massive systems, it's worth understanding the transaction processing (TP) model. Today I'd explain it as follows:

• Your application consists of a collection (hundreds, or thousands) of services. These are connected either directly or indirectly to your front-end UI.

• A service runs with N instances, to provide scaling for many users. The TP starts and stops instances as needed.

• The services typically work with further back-ends, databases, and so on. This is not the concern of the TP system.

• A single service instance works with a single "transaction" at once. Be careful: the term "transaction" is over-loaded. Here, it means a request to do some work, where the work comes from the UI either directly or via some other layers. It is often used to describe a package of work done by a database. The meaning is similar, yet not the same thing.

• Services hold no state for UI sessions. If the service transaction needs state, it is held by the TP, and passed to and from the service, with each call.

This is actually close to the "actor model" that we aspire to with asynchronous messaging.

OK, back to asynchronous I/O. Let's say I'm waiting for the user to press a key on their VT220 terminal. In synchronous code I'd make a system call that waits for input. At that point my process is swapped out, and effectively dead until the user presses a key and input arrives.

An asynchronous call (called an "AST" on VAX/VMS) is a bit different. It does a similar "wait for input" call, and adds extra arguments. It says, "when you are ready, call this function, and pass this state." The critical difference is that after the call, the process isn't suspended. It continues right along with the next instruction.

So you could for instance wait for input on 100 different terminals by making 100 AST calls in a row. You could chain to a single function, and pass the terminal ID as state. Then your process could explicitly suspend itself. Each time a user pressed a key, the process would wake up, the function would get the event, and process it.

This event-driven I/O model is still how we build really fast multi-threaded servers today. However a server typically only deals with one kind of event, namely network traffic. Perhaps three: input, output, and errors. To write an ACMS front-end server, you need to deal with rather more I/O events. At least:

• System calls to resolve logical names, terminal IDs, network names, and so on.
• System calls to read and write to disk.
• Events coming from terminals.
• Events coming from ACMS, typically "finished doing this work."

So you don't have a single event handler, you have dozens or hundreds. This makes the server design horrendous. Imagine writing a large program consisting of tiny functions, where each specifies the name of the next function to call, as an argument.

Being a clever wunderkind, I figured out a solution. One of our COBOL tools was a state machine designer. A neat thing: you describe your program flow as a state machine, and it turns that into code. I'd started rewriting that in C, a project that ended up as Libero.

Using Libero, I could write the flow of execution as a state machine, and generate C code to make it run. In the core of this C code is a generic AST handler that can deal with all possible events. It knows the state machine and can just call the next function itself.

So here is a piece of the state machine, which initializes a new terminal session. What this means isn't important. The point is you can read it, the words kind of make sense, and you're not writing crunky chained AST code:

Have-Lat-Connect: (--) Ok -> After-Sign-In + Spawn-New-Lat-Thread* + Move-Thread-To-Active-Pool* + Get-Terminal-Port-Name + Get-Terminal-Characteristics + Set-Terminal-Characteristics + Write-Greeting-Message + Assert-Iosb-Okay + Init-Connection-Parameters + Acms-Sign-In + Check-Acms-Status-Block (--) Error -> + Signal-Io-Error + Deassign-Terminal-Channel + Terminate-The-Thread

So I presented this approach, with some examples and test results, to the client. They asked a respected consultancy firm (Andersen) to check it. The two consultants who did that work had previously worked for DEC and were experts in ACMS server design. I think they'd helped write the standard ACMS front-ends. After some days of explanations, they went off to write their report, which came down to "the proposed approach is insane and will not work."

Somewhat later we discovered that Andersen had been hoping to get the contract themselves. I learned an important lesson about consultants: they are professional liars.

The IT manager of the client took the report, threw it in the trash, and told his management, either we go with this option or I quit and you can forget this project. Since this man had built the original system on IBM CICS, and brought most of his team into play, that was a serious threat. Management caved, Andersen were kicked out, and we got the green light.

The system we built worked as designed and went live on time in 1992. It grew to handle multiple tour operators, with front-end servers happily dealing with two thousand terminals each. We maintained and extended the system every year for a decade, little by little. When I stopped working with Sema, the client paid maintenance to me directly, until the system was finally decommissioned in 2010.

The lesson here is, if you have the trust of your client, and s/he has real power, you have done half the work already.

This was a perfect software project. Working with good matériel; decent hardware and a non-insane operating system. Working with a smart team that know their stuff, and a client who likes and trusts you. With total freedom to build things the right way. Where the hardware and software bends to your will. Where you can take the risks, analyze them, and design them away.

Little did I know how rare such projects are. Shortly after the tour operating system went live I found myself in an insurance company in Brussels, looking at an ancient terminal the size and shape of a large microwave oven. Carved in the bezel was the ominous word: "UNIVAC."

Shit Goes Downhill

I imagine the conversation went thus: "You're saying you can make the EASY accounting system work on our existing UNIVAC mainframe?" followed by a confident reply, "Of course! It's portable! Now if you'll just sign our framework agreement on licenses and rates," followed by a dryly muffled laugh and the scratchings of quill on parchment.

Whereas the VAX project was a smart bet on new technology, the insurance company was in the midst of a fight between old and new, and we found ourselves on the wrong side of history. The old guard was defending their UNIVAC mainframe (and its not trivial budgets) at all costs. The new guard were trying to push the IT infrastructure towards UNIX.

The first time I saw our client — the incumbent IT manager — he was slumped in his chair, melted from fatigue and age, a cigarette in his hand, ash and stubs on his desk. Here is a man in the terminal stages of a truly horrific disease, I felt.

I'm sure that there was a time when UNIVAC made relatively excellent computers. I'm sure someone, in the early 1960's thought, "seven bits is going to be enough for anyone!" I'm sure at some point the slogan "America's First Commercial Computer System!" was spellbinding for customers. But not Brussels in 1992, please no.

By 1992 we'd invented the Internet and were starting to see the first Linux distributions for PC. I had an Yggdrasil distro on CD-ROM that included, wonder of wonders, a free C compiler. Linux was not just a real OS, as compared to the kindergarten MS-DOS, with proper virtual memory and tools. It had the same shells (ksh, bsh) as the Big Iron UNIX boxes we were starting to see in companies.

And here we were, with a system pulled through time like an Automatic Horseless Carriage from 1895 trying to compete in the 1992 Grand Prix.

Leif and I briefly looked through the system docs, and played with the terminals. I looked at the microwave-with-keys thingy and said that it seemed rather ancient. "Oh, it gets worse," said Leif, looking at a page in the manual. This was going to be a familiar phrase. "Oh, it gets worse!"

I'm not going to bore you with detail. Just one small piece to show how bad it was. The UNIVAC terminals were like the classic IBM 3270s, block mode devices that sent the user input as a single block of data, back to the mainframe transaction processor. Something like a HTML web form, except sending field/value pairs, it would send row/column/value tuples. Fair and good.

The first problem we hit was that the large Enter key did not send anything. Oh no, that would have been too obvious. Instead there was a separate SEND key. OK, we guessed UNIVAC users were used to that. More fun though, the terminal only sent up to the cursor, no further. So if you typed some data, and then backed up to fix an error, and then pressed SEND, you lost half your input.

The terminal had function keys that we could program by sending commands together with screen data. So our hack was to add an invisible input field on row 24, column 80. Then, F1 would move the cursor to that field, do a SEND, add a code for "F1" (our apps used sixteen function keys for navigation), and then move the cursor back to where it had been.

The worst part, which is hard to wash off even after years, is that we were actually proud of this. We'd made it work. There were dozens of other "WTF?" moments, yet eventually EASY could run on this prehistoric system. Leif and I made our excuses and found other projects to work on.

Taking Money off the Table

Around 1992 Sema was building a reservation system for a high-tech fun park in France called Futuroscope. Part of the challenge was to send bookings to the various hotels that clustered around the park itself, and sat further afield.

Before we automated it, bookings were faxed by hand. This was before the days of office laser printers. Each night (or perhaps twice a day), a batch job ran that printed out all hotel bookings on a "line printer." This produced a large "listing," perforated so it could be torn into individual sheets.

Some poor fax jockey took got listing from the Computer People, ripped off the header and footer pages, separated the rest into individual pages, and chopped these using a guillotine into A4 width, so they would fit into a fax machine. Or perhaps they faxed them sideways, breaking all the rules of fax decorum of the time.

I can't imagine the joy of entering each fax number manually, waiting for the modems to connect, feeding in the sheets, and sorting the bookings into "done" and "failed, try again in 30 minutes," and "hotel fax seems borked, call them to see what's wrong" piles. Especially in peak season, when the booking system would spew out hundreds of bookings a day, and even a single misplaced booking would result in drama. If you've never seen a Parisian family of five arrive at a hotel where their booking didn't get through, they make the citizens of San Jose look polite and charming by comparison.

Free software fixed this and destroyed the "fax jockey" position, at least in the Futuroscope.

Futuroscope were looking at commercial bulk fax systems. These were extraordinarily expensive, thousands of dollars just to get started, and more depending on capacity. We found this distasteful especially since by this time a fax modem (a small external box that plugged into a serial port) was maybe a hundred dollars. On Windows we were used to software like WinFax that could send faxes using a special printer driver.

Also we did not want to have to interface with these beasts, which used bizarre proprietary software that I knew was going to cause us needless pain. And like I said, if the faxes didn't work, the whole system was suspect.

So I looked around and found an free software package for UNIX called Hylafax. Together with another free software project called GhostScript, we could create nice (i.e. using proper fonts, and with the Futuroscope logo) bookings, and send them to the hotels entirely automatically.

My idea of using free software was received with solid, head-shaking skepticism. This just wasn't how things were done. What sold my proposal were two arguments. First, that every franc the client spent on expensive fax machines was a franc we couldn't invoice. Second, the fax subsystem was so important that whomever supplied it would become an important vendor. Surely that should be us (Sema) and not some random box seller.

OK, Sema said, if you can make it work, we'll sell it to Futuroscope. So I downloaded the source zips for the packages, built them, and tried them out. It was all surprisingly simple. The hardest part was finding a fax modem that would work with the UNIX server, so I started work on my PC using its fax modem card. This was in the days of dial-up and I paid around $2,000 a year (in today's money) to one of Belgium's first ISPs, in Leuven, for Internet access.

It turned out that a single fax modem was able to handle peak traffic, especially since there was no time lost messing about entering fax numbers and feeding paper. Hotels never replied by fax, so there was no need to handle incoming faxes. All we needed was queuing of outgoing faxes and a way to know if they failed.

Which it turned out, Hylafax mostly did for us. It used a neat client-server design so the Hylafax server ran in the background, and our fax script called the Hylafax client each time it wanted to send a fax. The client passed the fax onto the server, which queued it, and sent it when it could.

Hylafax automatically retried sending, a few times, so we only had to deal with hard errors (broken fax, run out of paper, hotel on fire, that kind of stuff). We recovered the status codes later, from the Hylafax log file, and pumped them back into the application.

It came down to a rather simple Bash script that took the next fax from an inbox (a directory of with one file per booking), called GhostScript to create a printable page, called Hylafax to send it, and then moved the fax to the outbox. It also created a log file that the application could read to know what happened. This script ran as a daemon, sleeping for a second if there was no activity.

It worked nicely. The good folk at Futuroscope were a little shocked that a simple fax modem and some magic software could replace the large industrial-sized fax machines their vendors were trying to sell them. Nonetheless, they saw it worked, shrugged, and the system ran nicely.

Only once did things stop working, when the application was moved to a faster, cheaper server. Turned out someone had implemented "sleep for one second" by busy-looping. On a multi-user system, no less. People had wondered why everything froze for a heartbeat when someone confirmed a booking. Yes, the code said something like,

PERFORM CONSUME-WHOLE-CPU 2000000 TIMES

Their proposed fix was to increase the counter to 4 million times. I sighed and explained how to call the Bash "sleep" function from COBOL.

Lesson learned: identify the riskiest parts of your architecture and bring them under your full control. The point of using free software was that we could control everything using Bash scripts. We could test everything in isolation. We could in effect build a full solution, removing all risk, faster than it took us to learn and interface with existing solutions.

Successful use of free software in a commercial project was a shock to Sema in 1992. There was some talk of turning the fax gateway into a product yet I couldn't in honesty package a hundred-line Bash script for sale.

Wiring the Factory

Shortly after, and with UNIX, HylaFax, Bash working as mind-bleach for my UNIVAC PTSD, I worked with a Sema team on an industrial automation project. In short, this meant getting a large cement factory (owned by the firm CBR, one of the main Belgian cement producers) to talk, in a matter of speaking, with the SAP system that took orders and invoiced customers.

When we finished the project (and this is a flash-forwards), the head of our department came storming into our offices, quite angry. He showed us the figures for the work we'd done. We'd completely missed the budget and he was afraid it was going to cause real problems for the client.

I was a little confused at first because we'd worked hard and well, and pulled what was rather a mess into a coherent, well-working system. The managers on-site adored us, and so did the managers at customer HQ. What was the problem?

Turns out, Sema had made a fixed price bid, based on some random estimates of how much work we'd have to do, multiplied by the usual "engineers always underestimate the work" factor, plus random "project management" amounts, and so on. It was a fairly solid budget, for a major upgrade to the factory.

And we'd under-spent by such a large amount that Sema's finance department had red-flagged the figures as impossible. I laughed and explained that we'd simply done our work well. And indeed the customer never complained. We did not get bonuses, or raises. It was simply our job, and my reputation for pulling off miracles rose a few notches.

I'll describe the project itself, and how we got it to work, because it seems interesting. The factory produced dry cement, which was sent by truck all over Belgium. Clients would place orders, which would be scheduled, and then sent to the factory. Truck drivers (independent contractors) would come to the factory, get a shipment, and drive off to deliver it.

This was done largely by hand, which was slow and inflexible. Orders would be faxed through to the factory, where someone would schedule them, then enter them on a local application. Truck drivers would arrive, and at 7am the booths would open. In each booth, when a truck pulled up, an operator would choose an order, print it out, and give that to the driver.

The driver would then find the right loading bridge for the kind of cement, and give the paper to the bridge operator. He would control the loading pipes, confirm the order on his computer screen, and the trucker would go off to deliver it. Some trucks were dry, some got additional loads of sand and water and did what concrete mixers do.

Customers liked the cement, and hated the ordering process. It was slow and clumsy. If the weather suddenly got better or worse, there was no way to change an order to take advantage. You can't pour concrete when it's below freezing point. What do you do if there's a cold snap, and a truck full of the stuff turns up?

Industrial automation is a specialized business, and it was not ours. There were companies who knew this area, and one firm had built new automated loading bridges for the factory. Sema's job was to build a scheduling application. Another firm had built an application to replace the booth operators, and allowing more booths to be added over time. Yet another firm was providing various pieces of hardware. And then we had to connect the scheduling application to an SAP system (in Brussels, far from the actual factory) used for sales and invoicing.

None of this had been developed or tested beforehand. So when I arrived on the project, we had five different teams building stuff that was supposed to work together, and of course, did not. Apparently this was just standard practice: bring a bunch of stuff to a site and then hammer at it all until it worked. What made it extra fun was that each vendor had the attitude of "our stuff works, so if there's a problem it's yours, not ours."

It was made worse by the practice of fixed budgets. Each firm had made offers, which the client accepted. Spending an extra day fixing someone else's problems meant a day of lost income. All this is standard in construction projects. Yet building a complex software & hardware system is a different thing.

Let me give an example. The design used smart cards, a great idea. When a driver arrived, the kiosk would spit out a smart card holding all the order details. The kiosk would then say "Go to bridge 4" or whatever. At the bridge, the driver put the card back into a kiosk, the cement loading started, and that kiosk printed a receipt, for the driver.

So here we are with a UNIX system running the scheduling application, a PC running the kiosk app, and smart card readers. The team writing the kiosk app had no idea how to talk to the UNIX system, nor to the badge readers. No-one has experience with the smart cards, which the client has bought from Siemens. Siemens is not providing much support, and this is a decade before the web made it all easy.

And in every meeting, the attitude from each firm was, "not our problem, our stuff works." It was after several months of this toxic stalemate that I'd joined the project, in my usual dual role as plumber and fire fighter.

My colleagues and me adopted a simple solution, which was, "every problem in this project is ours." I figured out how the badges worked and we designed the badge data format. I wrote serial I/O routines to read and write badges, and handed them over to the kiosk teams. We built TCP/IP clients and servers to connect the kiosk application to the scheduling app, and gave the kiosk team libraries they could just call.

This all sounds like a lot of extra work yet we loved it, and were good at it, and it was faster and easier than arguing. At first the other teams were confused by our approach, and then they realized things were actually moving ahead. Little by little the project became happy again.

The kiosks were the cornerstones of the project, and were abysmally badly designed. The client provided the physical infrastructure, and the large metal housings. Various parts, such as cement-resistant keyboards, were ordered from afar. There were PCs to run the application, and little off-the-shelf Epson thermal printers to print tickets. Nothing had been tested beforehand.

And predictably, as the kiosk team tried to put this all together, nothing worked. It was almost comedic. The printers were not designed for kiosk use but as cash register printers. They had to be propped up at an angle with pieces of wood for the tickets to come out, and they jammed constantly. But the worst offenders were the screens. We'd started working in late winter, and by spring we were testing the kiosks for real.

As the sun rose, it dawned on us that the kiosk entrances faced south. The kiosk design was nothing as sophisticated as today's parking kiosks or highway toll kiosks. Flat panel screens were still science fiction. So here we have a PC in metal box with its CRT monitor behind a sheet of glass. The driver had to get out of their truck, their badge (which was both for ID and to hold the current delivery), choose a delivery, confirm it, and get their ticket.

With the sun shining from behind, the screen was unreadable. CRTs did not have great daylight visibility. In direct sunlight, behind a sheet of dirty glass, it was tragic.

One day, watching the frustrated drivers squinting at the glass as they tried to read the screen, I took a flat piece of cardboard, cut out a credit-card sized hole, and taped it over the glass. This fixed the problem. When I came back to the site a week later, both kiosks had metal plates, to my design, welded over the screens.

This was the tragicomic part. The project was also just painful in other ways. The client had specified that the kiosk application should be able to run even if the scheduling app did not work. This was sane, since the exchange of orders to and from the SAP system had never worked well. So if anything failed upstream, they could continue to ship cement, and then re-enter their orders afterwards.

The kiosk application thus had a "stand-alone" mode where, if it decided the scheduling app was not working, it would take control of things. Except, this app would detect a "failure" randomly, and the entire system would swing in to, and out of, stand-alone mode. Each time that meant fixing things up by hand afterwards.

It was stupid things like, the kiosk app would open connections and not close them, and exhaust file handles on the UNIX system. Or, it would run a 100% CPU background task that slowed everything down so that the I/O loop could not run, and timed-out.

There was a point where the factory would call us at 4am, saying the system was down, and we had to come on-site to get it working again. That meant a 2-hour drive. When we arrived, a long queue of furious truck drivers waited on us to fix things. Which meant restarting things, and manually consolidating a mess of local and remote orders. The real damage was done by switching in and then out of stand-alone mode.

The lesson here is that making systems more complex, to try to make them "reliable" will usually make them less reliable. What we were experiencing was "split brain" syndrome, the bane of any clustering design. Allowing the PCs to randomly decide they were in charge was a Bad Idea. Letting them decide to switch back to normal mode made it much worse.

Since the crises were so random, we started sleeping in the factory offices so we could be there at 3am to try to catch it happening. To be frank, I didn't enjoy sleeping on a mat in between office furniture, in the heart of a massive industrial zone. Once we decided we'd bring beer, it was slightly better.

What we eventually came to was a panic button on the PC screen that forced stand-alone mode. If orders stopped arriving from the UNIX system (for any reason, one does not care at 5am), the operator clicked the panic button, and the PC switched to stand-alone mode. The operator could then handle the morning peak, with a little more manual input. Somewhat later when things were calm, we'd consolidate orders with the UNIX, investigate what had caused the fault, and switch back to normal mode.

Additional lesson: fail-over automatically and recover manually.

Despite all this, we finished our work so far under budget that it raised alarm bells. The client loved us, having seen us in action. Years later, this would pay off.

The main lessons I learned were obvious and yet worth stating, perhaps because the industrial world is so distant from the normal pure digital development world:

• Don't expect random work from random vendors to magically work together.
• Do not develop and test in your production environment.
• When things go wrong that even marginally involve you, assume responsibility until you know where the real problem lies
• When you use hardware and software that bends to your will, you can work significantly faster.

Fear and Loathing in Brussels

In an industry that is driven by forwards change, it is shocking how many projects are based on betting against the tide. By this I mean forcing an application to run on some antiquated, expensive, inflexible, and painful platform, when new, cheap, flexible, and enjoyable alternatives exist.

It almost always ends in disaster for the organization. And yet I've seen this happen so often that it is almost a caricature. Brain-dead management forces obsolete technology on team who build mediocre application that fails to fix organization's real problems, which then gets taken over by smarter competitor.

Here are some of the reasons this happens so often, in my opinion:

• Psychology of sunk costs. "We've spent twenty million on our goddamn mainframe and goddamn you if you think we're going to junk it just because you have this new fancy-wancy UNIX thingy."

• Self-interest of vendors. "Sure we can expand the main memory from 64MB to 128MB. It may be a little expensive, but mainframe memory is special! It's better than that cheap, unreliable UNIX memory!"

• Politicization of IT. "My departmental budget is $10M per year, and you're talking about slashing that by half and yet adding 10x more users? Are you insane? You're fired. I'll find a consultant who agrees with me."

• Fear of the unknown. "Why would we use that 'UNIX' thingy? It's just a large PC, totally unreliable. And anyhow, UNIX will never be widespread. No, we prefer our tried-and-tested mainframe architecture that runs real networks like SNA LU6.2!"

Shortly after the UNIVAC trip, I was thrown into one more such project. The team was building a new financial system. The team was using our COBOL toolkit. They were using AIX UNIX machines for development. Production was to be on a BULL TDS system. One. More. Obsolete. Mainframe.

The entire project was designed as an internal hostile takeover of the firm. The firm was a recent fusion of two businesses. The profitable one consisted of many smaller offices, each working their own way. Clients got customized service and paid a premium for it. The unprofitable firm however had the power, for political and historical reasons.

So the new planned system would enforce order and consistency on those cowboy offices. It would in effect break their independence and bring them into a neat, centralized organization. It was a classic battle between good and evil, between sanity and insanity. And once again, I found myself on the wrong side of the battle lines.

The architects of the new system had no idea, really, what they were going to build. They could not ask their users, because they were at war with them. So they made it up as they went along. Designs and plans fell out of meeting rooms onto the developers' desks.

The developers were decent people. This was the era when even with modest training, you could learn to develop business software. We had as many women as men in the crew. I cannot fault the developers, nor the COBOL language, for the crap that emerged. This was a direct result of getting insatiable, arbitrary demands from the analysts.

My job was to support the developers with source control and continuous builds, and to build the technical framework to make the applications run on TDS. Bull's computers were feeble imitations of IBM's mainframes, and likewise, TDS was a feeble imitation of IBM CICS, the dominant mainframe transaction processing system.

Lurking like a bridge troll underneath the application was an Oracle database. I won't comment further except to say that it more or less worked, yet was both a major source of problems, and apparently, a major slice of the project cost. We could have done much better IMO.

In those days the only plausible source control system was CVS. We did not use that. Instead we used shell scripts to check-in and check-out sources from a repository. The application consisted of hundreds of programs, each was either interactive (basically, one or more screen pages with logic), or batch.

When you checked out a program you got all its related files. You could edit, compile, and test locally, quite quickly. When you checked your program back in, we saved the previous version (saving the differences, to save space), and stored the new one. We compiled everything, once per hour or so, to create a fresh testable version of the application for test users.

As new programs got checked in, they got sent to the Bull system for test compilation. That produced error listings (the code trying things that worked on AIX yet not on the Bull). Our shell and awk scripts pulled these error reports back to the AIX, extracted the few lines of error message from 20-page listings, and passed them to the developers so they could fix things.

Some of the developers (especially the older ones who had learned the Bull environment) just printed out the reports, then stacked them in heaps around their desks. These heaps grew higher and higher, eventually, forming walls behind which the developers hid.

The test users marked changed programs "approved" when they were happy with them. To make this work we had developed our own issue tracking system, in COBOL. That was not hard, as these tools worked well and let us build applications rather quickly.

When a program was approved, it was sent up to the Bull system for real this time. It was compiled, and from then on available to external test users.

People noticed quite early that the AIX test environment was rather faster than the Bull. Out came the arguments about interactive sessions vs. transaction processing. "The Bull will scale better for hundreds of users," went the official line. No matter that even with a handful of users, it was already slower. They added more of that special mainframe memory, and upgraded to the largest CPU possible. Using the Bull was not negotiable.

Our development environment was quite neat, I think. For developers, it all just worked. Of course I far prefer our modern git-based environments. Yet what we built showed the power of straight-forward UNIX shell scripting.

We also cracked the TDS environment and got the application running on it. This was no simple job, mainly because the programs had gotten so large that they did not fit into memory. Like a lot of machines bypassed by history, real memory was larger than virtual memory. That is, TDS limited programs to ridiculously small sizes, while the actual system memory was much larger.

You could even access the system memory from COBOL, just not portably. And we depended on making TDS and AIX look exactly the same from the point of view of developers. Which ended with me writing a memory manager in COBOL that managed system memory, swapping blocks of memory in and out of application space, behind the scenes. It very fast because the heavy work (copying memory) compiled down to fast machine code. If in COBOL you say "move these 30,000 bytes from A to B" that can be a single machine instruction. If you write a loop, moving 30,000 bytes one by one, it adds up to maybe a million instructions.

No matter, we could not save the project. The developers went through death marches, the regional offices went on strike, managers left and were replaced. By the end of 1995 the client had stopped trying to blame Sema and our team for the failure. Everything we did worked. The project was postponed indefinitely, and I went off to find more constructive things to work on.

Welcome to the Web

In 1995, the entire World Wide Web fit into a single book, with one paragraph per website. In November 1995 I registered imatix.com and started to think about building an Internet business.

I'd been working in my free time and weekends with my friend Pascal on web servers. The design was inspired by the server I'd built for that tour operator. You can handle a lot (a lot) of sessions in a single process, if you use what's called a "cooperative multithreading" model. This means each session does a little work and then gives control back to a central dispatcher.

Threads must never block, and all I/O must be asynchronous, just as on the VAX with its AST calls. You don't need actual system threads. That's nice because real threads bring a lot of nasty, often fatal problems with them. We've learned by 2016 that sharing so-called "mutable" data between threads is a Really Bad Idea. In 1996 I'd less data for this, yet already knew that it was nice when a thread could work with global data without risk of stepping on other threads' toes.

By December 1996 we released a working web server, which we called "Xitami." Xitami was one of the first free web servers to run on Windows. Microsoft's "personal web server" was crippleware, and people hated it. Xitami was easy to install, it was fast, and had no limits. You could, and people did, handle enough traffic to swamp a highest speed Internet connections.

I once saw a Slashdot article about a guy who'd hooked 26 hard drives to his Windows 95 box. He had a web page showing the PC and explaining how he'd done it. Slashdot was one of the most popular geek news sites, and the name had become to mean, "kill a website through sheer volume of requests." He was running Xitami on this PC, and it didn't crash or slow down.

Yet I'm not going to talk about Xitami. It was my first popular open source product. It won many awards and users absolutely loved it. Yet it made us no money and had no developer community, and got us no business. As I define "open source" today, it was a failure.

Our real product, on which I wanted to build a business, was something quite different. I'd designed a "web transaction protocol" and implemented that in Xitami, together with some tools for building web forms. This gave us a crude yet working transaction processing system for the Internet.

That is, by 1997 or so we were able to build usable web applications that could handle thousands of concurrent users on cheap hardware. HTML was really poor. Basic functionality such as "put the cursor on this input field so the user can fix an error" was missing. Yet it worked.

In that company with the Bull TDS there was a separate, independent division building their own applications. They'd been searching for some way to provide access to remote users. They were well aware of the potential of the Internet, even if that mostly ran over dial-up modems. They took one look at our web framework (which I called "iMatix Studio") and asked when they could start using it.

Studio wasn't trivial to use. We'd not spent any time looking at developer languages, and we had totally missed Python as a candidate. Java 1.0 was just released, unstable, slow, and unusable. We did not look at more esoteric languages, and I was firmly against C++ because it produced such fragile code. So, we developed in C, which is less than ideal for the grunt work of business applications.

Still, it worked, and we slowly built up our framework. We lacked good code generation, so we built a generic code generator called GSL, which we still use today in the ZeroMQ community. We lacked a language for modeling the state machines and UIs we wanted to generate, so we first wrote our own structured data language, and then switched to XML by mid-1997.

It was in 1998 that I decided to part ways with Sema. They paid me well, and they were decent people, yet they were consistently betting on the wrong side of history, and it became deeply troubling. The last straw for me was when my boss asked me to make a technical design for a nationwide insurance network.

Our brief was to connect several thousand insurance agents in a network. They would log into a system, query insurance files, get quotes, log events and claims, etc. It was almost exactly the same problem as our old tour operator system, seven years down the line.

So I made a design betting on the future. We'd build it as a transactional web application, and use a number of XML formats for document standardization. The framework would be Studio, using arbitrary developer languages. We'd run on a single large UNIX box, which we'd tested and shown capable of dealing with ten times more traffic than planned. We'd use an Oracle database (again, customer decision, not negotiable).

Sema presented this design to the client, while simultaneously presenting a second design based on a client-server framework called PowerBuilder. At the time, at least in Belgium, PowerBuilder was fairly popular for such wide-area applications. Microsoft supported it and promoted it. It was expensive and produced a lot of money for vendors like Sema simply by giving them a slice of license income.

The PowerBuilder user interface was also much richer than HTML, at the time. It was however a miserably nasty system to deploy. When you started your app every day, the first thing it would do is download updates to the client half. This could take five minutes, or it could take an hour. You could not do updates during the day. Users had to log out at night.

Yet we were talking about a system that was to be developed gradually over a period of years, and was meant to last for decades. The goal was to define new industry standards, and to bring the insurance industry into the modern world.

And my design was rejected on the grounds that it could never work, that dial-up was no basis for a serious application, and that the Internet was just a toy that would never come to anything.

I shrugged, and went and started iMatix Corporation sprl. I started to plan, with my friend Julie, how to build a real business. It was all new to me, so we kept it small and modest. Sema kept asking to help save catastrophic projects, and I kept accepting. We needed income.

We worked on a project for the Belgian railways. Again, a bet against history, with a massive distributed application running on VAX/VMS. No transaction processing with ACMS. No attempt to reduce costs. Instead, use a platform that was already dying. Compaq had just bought DEC, the producers of the VAX, and would slowly start killing it. I stayed only a few weeks, then made an excuse and left.

We worked on a death march project for the Francophone school system. Once again, a bet against history, with production data on an IBM mainframe, and a nasty client-server product that depended on Windows NT servers in every school. Once again I proposed a web based architecture to solve the interminable woes of this over-complex architecture, and was asked to back off and stop being disruptive.

It was in 1999 that Manpower International in Brussels asked me to help on a small project, which turned into one of the most widely-used and most successful web applications Manpower had (and perhaps has) ever built.

UltraSource, The Hot Potato

FYI, Manpower's business was to select and recruit staff, and then send these out on temporary work for clients. It is a business called "sourcing", and more specifically, "HR sourcing" that I got to learn in some detail.

Manpower had a core IT strategy that was quite common at the time, in larger multinationals. That was to build a central information processing system, and then use that everywhere. For a company like Manpower, that meant using the same application for every customer, in every country.

It was the same strategy that sold products like SAP, aka "liquid cement." It is a strategy that is especially seductive to managers trying to build a company through acquisition. Buy a firm somewhere, throw out their existing core IT systems, replace with whatever corporate standard the last CTO fell for.

As a business strategy, it can work. Modern banks have relied on this because otherwise their separate acquisitions simply can't work together. Modern firms like Google, Microsoft, and Amazon each rely on their own global, shared infrastructure. Yet banks and Internet firms are special cases, where sending data around the organization is their core business, rather than a problem of profit & loss reporting.

In the case of a firm like Manpower (as with that firm trying to move all its offices to a shiny new Bull mainframe application), it was a rather insane strategy. Manpower's competitors tended to grow by acquisition and then leave every business unit to do its own thing. That was far more organic, and cheaper, than trying to rationalize all their IT systems.

When we arrived on the scene, Manpower International (which had the task of providing software applications to the whole world of Manpower companies) had tried and failed miserably to build a global sourcing application. We did not know this, of course. The problem, a classic hot potato project that no-one wanted to touch, had fallen back in the laps of a small team in Milwaukee.

Some time later Milwaukee asked me to make a proposal and so I flew there to spend a week talking with a room of 20-30 key people. They were experienced, skeptical, and critical. I brought with me a small prototype of some core functionality, an order placement screen.

It didn't happen overnight. Our contact in Brussels had spent some months funding us to make the demo, and prepping the room in Milwaukee. Her home office was Manpower Japan, and that company was having a hard time getting larger clients. Her vision was to build a real sourcing application that would solve these difficulties. She wanted, ideally, the Americans to pay for the work.

When we started talking about large scale web applications, my heart leapt. Finally, a project that aligned with my vision for iMatix. And then the reply came, "sorry, company standard is Microsoft Windows, Microsft Transaction Server (MTS), and Microsoft SQL Server"

Now Windows is just a box and our Studio software ran fine on it. Indeed it was on Windows that Xitami got the most love from users. By that time we were already running Linux heavily internally, for our SVN source control, our mail server, and so on. It was clear that Linux was the way forwards for servers. Yet Windows would make a fine interim step.

MTS however, was a WTF? It was a poorly documented, fragile mess. From our first tests we found that it crashed if you smiled wrongly at it. It was unscriptable, so all administration had to happen via point and click menus. Applications would freeze randomly. It did not play well with SQL server, so we got deadlocks and timeouts. It was a black box with little visibility on what was going wrong.

I argued all of this, and pleaded to use our own tools. The answer was no, no, no. So, we wrote the prototype in Visual Basic and then somewhat later we (myself and our Brussels manager) found myself explaining it to a roomful of people in Milwaukee.

It seemed like a rather hostile room. One or two of the people were sympathetic. The rest seemed truly unhappy to be there. I didn't take it personally, and just continued with diagrams of architecture, the tools we were making, walk-throughs of the code, and so on.

And a few days after we flew back to Brussels, word came that we'd gotten the project. Turns out, absolutely no-one wanted to touch this project except a few people in Milwaukee. This small core of 4-5 people had long experience with HR sourcing, and successes under their belts. They saw the potential in our proposal and they realized they could make it work.

The foundation for a good project is: a competent client who knows the business and has power of decision; a full-stack team that can deal with the work, at all technical levels; and a technical platform that is both dependable and tractable.

In the first real meeting we had with the core team, they showed us a large stack of designs and ideas. I looked at them, then made a counter-proposal. Let's take the prototype we have, I said, and improve it little by little. You make requests, we'll make changes, you test the results. How often can you make new releases, they asked us. Every day, if you want it, I replied. We agreed on two or three times per week.

And so we worked, with three people taking our work, testing it, and coming back with new requests. We'd make ten or fifteen changes a day, test them, and push them to the test machine. We logged the work in a Jira issue tracker. Every now and then we'd have face-to-face meetings to work through difficult problems, to break them down into small pieces.

Windows remained a real headache that we could not fix. Ironically the person who'd insisted on this left Manpower not long after we'd started, and no-one else cared. Yet it was too late and we were stuck on that. If I'd known how much leverage we had before we started, I'd have insisted on using our own tools.

On top of Visual Basic and MTS, we built our own framework (iAF, for iMatix Application Framework), a powerful code generator that built web pages, object managers, and database layers automatically. It worked well and we used it in other projects. Yet it was always based on that shitty toy scripting language and that nasty imitation of a real transaction processor. And so it never gave us real value.

This was perhaps the biggest mistake I ever made in my business. The lesson here is, when you are a technology company, use your own technology in your projects. Don't accept client requirements that sabotage your own vision and future. It isn't worth the short term gains.

Much later I came to frame this more brutally as, "never build closed source, period."

iAF used meta languages to describe the pieces of the application. Later I'd call this "model oriented programming." We defined a presentation layer consisting of screens, an object layer consisting of objects and views, and a database layer consisting of database tables and indexes.

Here's a simple example, for working with currencies. The database layer defines the actual properties we store for a currency:

<table name = "currency" description = "Currency" > Currency lookup table. <field name = "isocode" domain = "currency">ISO currency code</field> <field name = "name" domain = "longname">Currency name</field> <field name = "showcents" domain = "boolean">Show cents?</field> <field domain = "audit"/> <index name = "primary"> <field name = "isocode" /> </index> </table>

We then specify an 'object' that inherits from the database table, and we add some flair (such as here, the name is mandatory when creating or updating a currency):

<object name = "currency"> <require> <field name = "name" /> </require> </object>

Behind the scenes, iAF bases that object on a 'default' object that looks like this (this is a standard model that the developer doesn't need to know about, or touch):

<class name = "default" default = "1" > <!-- The create view is required by the object layer --> <view name = "create" read = "0" write = "1" delete = "0" /> <!-- The delete view is required by the object layer --> <view name = "delete" read = "0" write = "0" delete = "1" sameas = "create" /> <!-- These views are recommended but not obligatory --> <view name = "detail" read = "1" write = "1" delete = "1" sameas = "create"/> <view name = "summary" read = "1" write = "1" delete = "1" sameas = "create"/> <query name = "detail" view = "detail" /> <query name = "summary" view = "summary" /> <state name = "object exists"> <view name = "detail" /> </state> </class>

And then I describe how I want the object views to appear to the user. At this point I start mixing Visual Basic code with my model. The code generator includes my custom code in the generated result. Complex screens have thousands of lines of custom code. Simple ones like this, just a few:

<screen object = "currency" style = "select" alpha = "1" /> <screen object = "currency" style = "list" alpha = "1" /> <screen object = "currency" style = "create" /> <screen object = "currency" style = "detail" /> <macro name = "currency_validate"> <use object = "currency"/> <handler event = "on_accept" > fld_currency_isocode = ucase (trim (Request.Form ("currency"))) <if condition = "fld_currency_isocode &lt;&gt; &quot;&quot;"> <fetch object = "currency" view = "summary"> <missing> cur_message = "Please enter a valid currency code or select from the list" cur_error = "currency" </missing> </fetch> </if> </handler> </macro>

The application architecture isn't complex. It's 1999 and we're aiming for consistency and simplicity, not features. It is monolith built on a single database. There are no asynchronous updates to the web page, no AJAX. We use JavaScript for local validation and cosmetics, such as flagging an error input red, and putting the cursor there.

What we built turned out to have some interesting aspects:

• It was a large application, with a hundred database tables and five hundred screens. With our tools we were able to build new functionality rapidly.
• It was slow to use because often it took lots of clicks to get to a certain place. We did not spend much time on creating fast paths or shortcuts.
• Users loved the application.

The thing was, users did not work 24/7 on the system. They used it for perhaps half-an-hour a day. Some users (especially inside Manpower) used it full time, yet they were happy too.

The reason was simple in retrospect. This was an application meant for clients of Manpower, thousands and thousands of HR managers in random firms. The number one expense for Manpower, in previous projects, and the number one fear of users, was training and complexity. What we built was so consistent and simple that you could use it with zero training (of course, you had to know the business).

For instance we used a list/detail design for working with data. You clicked on 'Currencies' and got a list of currencies. Click on a currency and you see its detail. Click 'Edit' and you can change it. And so on. Learn this design once, and it worked almost universally across the application.

Obvious, and yet hand-built applications simply did not work like this.

Occasional users cannot learn complex UIs. And when they get confused they ask for help, and that will overwhelm any support structure you can build, and be ruinously expensive. And this was the reason no-one wanted to take on the hot potato.

The lesson here is a devious one, and that is that you often don't know what the real problem is until you get very close to it. We were lucky our approach fit Manpower's way of working.

We eventually rolled the application out to Japan, the Netherlands, Germany, the UK, and the USA. This happened gradually, over years, without major stress. We built a translation tool that let local offices translate the screens. We added customizable order pages so that local offices could make orders as baroque as they wanted.

The system wasn't perfect, in many ways. We had bugs here and there that only showed when the system got stressed. There was an order export process which pushed completed orders out to other applications for processing. This was long before we understood how to do messaging, so our designs were clumsy and not fully robust.

Above all, MTS would panic and shutdown threads when too many users worked at once. There was no way to fix this. It left the database with dangling locks that ended up killing the entire system. We had to limit the number of users, and move to larger machines.

Despite these stumbles, the application ("Ultrasource") became Manpower's first global web application, and ran for many years. It produced vast amounts of new business for our client, and gave iMatix a healthy income in ongoing maintenance fees.

And then in the summer of 2000, I got an email from a Nigerian beer company (Nigerian Breweries, or NB) asking if we had any experience with electronic payments systems, and could I please read the attached documents.

E-Payments in Lagos, Nigeria

East Africa was my first home and I'd traveled many times to the continent, for family and work. Sema would send me on short trips to do UNIX trainings, help with EASY installations, and so on. I'd been to Burkina Faso, Togo, Rwanda, Angola.

Either you love Africa or you hate it. There is no easy middle ground. Even stepping foot in Africa as a European is a political act, conscious or not. It took me many years to understand it as a continental prison filled with innocents, as I've written in my book Culture & Empire.

So the prospect of a project in Nigeria was appealing. I read up on the place in Lonely Planet and a book called "The Most Dangerous Countries in the World," which ranked Nigeria just after Columbia. My mother pleaded with me to not go there.

Not being a fearful person, I read the client's requirements, and started to work on a project proposal. It was an interesting case and I realized why they'd come to me. They used EASY for their accounting and sales. They had to extend that with some kind of a network that could carry payments to banks. We'd started to get experience connecting bizarre systems together, and building successful web applications.

NB's IT manager had asked Sema, "who do you know who could possibly make this work?" and they had replied, "ask Hintjens, he's probably the guy you want."

Let me explain what the problem was. NB was doing well. They dominated the market, and Nigeria was (and still is) a huge market for beer. The profit per bottle was low, yet sheer volume made up for that. NB was expanding, building new breweries across the country. They had powerful marketing and sales, and frankly, their beer was excellent.

Two things struck me, when I first visited. First, the brewery was perfectly organized. The buildings were all in good state, the gardens neat and tidy, the production line modern and shiny. It was a striking contrast from a brewery I'd visited in Kinshasa some months before, where crates of broken bottles lay around randomly, where people seemed tired, and where a heavy feeling of tropical lassitude lay around the place. NB's facility hummed with positive energy.

The second thing was it was run entirely by Nigerians with some other Africans. There was just one European, the financial director, or FD. Later, as the brewery upgraded its capacity, teams of eastern European engineers would fly in. And there was our team, as the project progressed. Yet in my first visit, I don't think I saw a single white face from leaving the airport to departure, two weeks later.

The IT manager took good care of me. We visited Lagos, a massive and busy city filled with life, most evenings after work. I've spent a lot of time in Africa and was rapidly at home, no matter where we went. After a week of design discussions and meetings with other managers, I went back home to write up a detailed proposal. I can still remember my shock, in the airport departure lounge, to see white faces and feel, "how strange they look!" and then realize.

Anyhow, back to the problem. Nigeria, at the end of the 20th century, was a huge and booming economy driven almost entirely by cash and favors. Much of the country's wealth came from oil, yet there was (and has been, in this part of Africa, for hundreds of years) a solid commercial middle class. Spend a day in Lagos and you see the sort of frenetic hustle that feels more like New York than Atlanta.

And yet, as the FD of the brewery explained, there is no system of credit. A "long term" bank loan is 3 months. No credit cards, unless you're a foreigner in a foreign operated hotel. No checks. Few people have bank accounts, and when they do, it's for moving hard currency in and out. Salaries are paid in cash. Cars and houses are bought and sold in cash. There are no mortgages, no car financing plans.

The brewery produces beer, which it puts into glass bottles. These go into crates of 12 or 24, which get stacked onto lorries. The lorries then take the beer to distributors. The smallest unit of sale, for the brewery, is one lorry load. The distributors then resell the beer to shops, clubs, bars.

When a lorry loaded with beer leaves the brewery, the distributor has paid for the beer, the empties, and the crates. The lorries belong to the distributor or their transport firms. The brewery does not own and operate its own trucks.

And the distributor has paid for the liquid beer, the glass bottles, and the plastic crates, in cash. One lorry of beer comes to, let's say, two large suitcases of cash in the highest denomination notes. The brewery staff count and check the money before the lorry leaves. The cash is then put into the treasury, a large room that is literally filled to the ceiling with notes, at times.

When a lorry returns, the brewery counts and checks the crates and bottles, and the distributor gets repaid (in cash) for the value of the empties. Remember that this is rather more than the value of the liquid beer. This means the brewery is, at any time, holding a lot of cash simply as deposits. It can keep some of the deposited cash in a bank yet a lot must remain on-site.

The backdrop to this successful business is a currency (the Naira) that continues to fall in value, so that the stack of paper needed to pay for one lorry keeps getting larger.

Then, a culture of criminality that is pervasive and can be shocking to foreigners. We're used to trusting others around us, and we're shocked when someone steals a wallet, or a car, or a phone. In Nigeria, there is no trust. All business runs on the assumption that theft will happen if it can. Entire lorries of beer have disappeared, never to make it to the distributor.

Then, a severe lack of technical infrastructure. In 2000 there was a fixed phone network that mostly worked, though only wealthy people and businesses had a phone. Electricity would fail several times a day, so everyone who could afford them had back-up generators. There were some good highways yet most roads were miserably bad, and jammed with traffic.

Driving around Lagos was the fuel of nightmares. You'd see a flame ahead on the motorway, it was a burning truck tire that someone had left in the road. Why? To mark a hole large enough to fall through. People would carry extra fuel in their car boot (since fuel supplies were so sporadic). So if they were hit from behind, their car would explode. Taxis and scooters ("okadas") drove maniacally around pedestrians, goats, street vendors.

After dark, there were roadblocks with armed police. I don't really know what they were looking for. We just greeted them, said we were from the brewery, and they waved us on. This happened so many dozens of times that I'd greet them with a huge smile, a handshake, and "shine shine bobo!" which is the slogan for NB's most popular beer, Star. They'd laugh and we'd drive on.

I'm rambling. Back to money. The question was, how could we cut out the cash transactions at the brewery?

After some thought, our solution was to have transactions happen at the bank side of things, rather than in cash on-site. So each time a distributor bought a truckload of beer, they'd transfer money from their account to the brewery's account. And each time they returned an empty truck, the deposit would flow in the opposite direction.

It sounds simple and obvious enough. Yet bear in mind, we can't trust individual bank employees, nor do we have any kind of remote (web based?) banking system.

Our first problem was to convince the banks to work with us. We learned quickly that a meeting for a certain day meant, "we'll arrive at some point during the day." Traffic jams could last several hours, even though from the brewery to Victoria Island, and the financial district, was just 15 minutes' drive.

The banks at first thought we were somewhat insane. No-one had ever suggested electronic payments seriously. As thought experiments, sure. For real, live business use, let alone for considerable amounts of money, they were skeptical. So we explained the model, which was based on a secure messaging system running over pure old email.

This took us some time to figure out. First we hooked into the accounting system, which was our old friend EASY. First we ran a continuous background job that caught orders flagged for electronic payment. This was simple to do. These orders were sent to our app, running as a web application on a local Windows server. A manager would get an email alert and click the URL to sign in. They'd review and approve the order. That would then go to their boss, who would also review and approve the order.

Once approved, a payment instruction would go to the bank. This tells the bank the accounts to pay from, and to, the amount, and a reference. We signed and encrypted the payment message, then sent it to an email address that the bank used. There were five or six banks, as distributors used their own banks.

Each bank ran a Windows server with the app on, but when they logged in they saw payments, rather than orders. The app would fetch email continuously: dial up to the ISP, log in to the POP3 account, fetch email, delete email from the POP3 account, wait for five minutes, and repeat. As payment instructions arrived they'd be loaded into the app database and be visible to the bank.

We looked, in each bank, at integrating with their systems. That turned out to be infeasible. These systems were all different, and all bizarre, and the expected volume of orders was not so high (a dozen a day per bank).

So we agreed that the bank staff would simply make the transfer by hand on their own systems and confirm it on our app, when done. This left scope for fraud yet no more than normal inside the bank. Banks could, and did, have their own internal approval systems for transactions above a certain size.

Once the transfer was made, a payment confirmation would be sent back (again, by encrypted email) to the brewery. Our app would receive this, flag the order as paid, and tell EASY.

EASY had no existing way to import such data, so I wrote a small tool that acted as a TELNET client, and could log into the application and push the right buttons. Sema were quite surprised when they realized how we'd done this. They were expecting NB to pay for work to make a new import program.

Corinne (perhaps the fastest developer I've ever worked with in my life) and I designed the app and she did most of the development. Pascal helped with the backend, as he'd done for UltraSource. It ran nicely. By this time we knew our tools well, and had built other apps with them, like our own issue tracker, ChangeFlow.

Somewhat to my surprise, we were able to deploy the system in test, and we started to expand it to other breweries (NB had seven or so, in different cities). The amount of skepticism was massive, both in banks and in the brewery itself. Distributors — who suffered the most under the cash based system — were enthusiastic. To test, we sent payment requests to banks, reconciled the responses, and checked that things didn't get lost. At the bank side, the responsible manager simply clicked "Done" without actually making a transfer.

Dial-up in Nigeria was fragile. There are lots of failed attempts, busy lines, dropped lines. It might take half a day for messages to start getting through. If a driver had been waiting for their order to clear, they'd wait half a day. That happened with cash too, as it might take some time to assemble the needed stashes of notes. One thing you quickly learn in Africa: patience.

Yet apart from that, messages did not often get lost. I learned that email is surprisingly robust, even though it has no delivery guarantees. We used a simple retry mechanism to resend messages if we didn't get a response within some timeout. We ignored duplicate requests and responses. And so on, the usual stuff.

Lesson here is: you can do a lot with little, if you are creative.

Then just a month or so before we were going to go live, Heineken bought NB (they were already a minority owner, then they bought more shares, to get a controlling stake). They stopped all investment in EASY and the electronic payments system and began to plan a SAP deployment. And that was that. We packed our bags, and went home.

Building the Perfect Kiosk

In 2003-2004 we rebuilt the delivery system for CBR's cement factory in Gent. This time we got the full project, down to the kiosks.

We'd done a good job with the previous automation project, so CBR called us in to one of the earliest planning meetings for their new factory project. They drew a schema of all the pieces. We had, as before, one supplier for the kiosks, one for the loading bridge automation, one for the Unix application, etc.

By this time CBR were talking to us (iMatix) directly, rather than Sema. In the meeting I stood up. "Last time, we had real difficulty integrating all these pieces," I told the managers. "So what do you suggest?" they asked me. I took the pen, and drew a large box around all the pieces except the loading bridge, which was out of our competence. "We'll do all these," I said. There was about two seconds' silence and then the project lead, who still adored us from the work we'd done, said "OK," and that was that.

I did what I often did in those days, when starting a new project. I sat down and wrote a design spec.

For once, and because we knew exactly what we had to make, and why, the design spec was almost perfect. My main goal was full off-site testability, for every piece of software and hardware, alone and together. I also wanted to make the kiosks completely fool proof. Plug and play and never break.

I asked CBR what their budget was. We agreed on a budget for the software on the usual lines: days times rates equals total. For the hardware, we agreed on unlimited budget, with no profit for iMatix. This freed us to find the very best hardware. The kiosks turned out very expensive to build, yet considering the cost of failure, and the overall project cost (automating the deliveries for a major factory), this was easily justifiable.

Julie and I designed the hardware by looking for all the smallest, most resistant pieces on the market. Sun-readable screens, dust-resistant printers, badge readers, PCs. We designed the casing and interior layout ourselves, and found a firm to build six metal housings. Julie found a paper supplier and got a palette of custom thermal paper produced.

Let me tell the printer story as an example. We were discussing with CBR about the tickets the kiosk should print, size and type of printer. We agreed, thermal printers, no ink to smudge or replace. I asked if they had a preferred supplier, and they did. So we asked the supplier what kind of printers they had. The smallest was the kind you see in an airport, about 30cm high, 75cm deep and high.

"It has to fit inside a small box. Got anything smaller?" I asked. I had made this sketch of the internals of a kiosk design, and the printer had to fit in the space of a small stack of paperbacks. "No," they replied. "I'll find another supplier on-line," I told them. "Good luck, it's impossible!" they replied, not entirely sweetly.

We did finally find the right printer, tiny and fast, designed for fitting inside a kiosk. So the eventual kiosk was about 75% dedicated to holding paper, which was perfect. The kiosks could run a week without needing a refill.

Lesson here is, don't take "impossible" as an answer. Everyone lies, not always deliberately. We just have our assumptions and ignorance and we believe we're telling the truth.

Mato designed a multilayer Linux OS that booted off DHCP, and then fetched its application from the network and then connected to the server. The kiosks were plug-and-play: connect power and Ethernet, and they'd boot in about 10 seconds and show their welcome screen. Access control to admin functions on the kiosk was via special badges and PINs.

Thierry and Pascal wrote a new dispatcher, and I designed XML messages that connected kiosks to this backend system. Jonathan and I wrote a new reliable messaging system (STEP) that talked to the central SAP system.

Ewen and I built the kiosk application; I designed the twenty or so screens in black and green, using a large sci-fi font and movie-style computer graphics that were bold and easy to read even in direct sun. Ewen brought them to life, with his code talking to the dispatcher over the network, using a simple TCP/IP protocol.

In total eleven people worked on this, in offices around the world. We tested each piece, and each kiosk separately, without setting foot in the factory. The client build kiosk housing, took our work, installed it, and it all ran first time.

What else do you expect?

There's a review of the project that I published rather later that gives more details.

There were some good lessons here:

• Build up trust with the client and sometimes they will reward you for it.
• When you've paid for all the mistakes, you should know how to do it right the next time.
• A good specification lets diverse people work together without confusion or conflict.
• If you can test each piece alone, and you have reliable ways of putting them together, the whole should work.
• Don't be afraid to charge the real cost.

On the downside, the kiosks worked so well that the client never came back for support or maintenance. While this project made us money, it did not lead to any new business, and did not push my vision for iMatix forwards at all.

In that sense, it was a total failure. It is ironic that a "successful" project can be a failure, while catastrophically bad projects can push you through to better things.

I also learned that building a team just to have a team was wasteful. Employees take time to manage. I was becoming a middle manager, and not coding any more. We tried extremely hard to find new clients, and built several potential products:

• A HR sourcing application ("Sourceflow") inspired by UltraSource. We rebuilt the whole core and UI and database. We'd made numerous apps using iAF by then, and Sourceflow was elegant and nice to use. We showed it to many large businesses and HR suppliers. Lesson learned: don't make stuff and then try to sell it unless you are growing an existing client base. Sourceflow went into the trash.

• A plug-and-play kiosk design for factories, airports, carparks. In 2004, this was still a new thing. We had excellent software, and what I think was a nice hardware design. We did some sales work. No luck. Into the trash (luckily it was just a paper design). Lesson learned: breaking into markets you don't know is probaby impossible.

• A group-chat-as-a-service application called SMS@. You created a "site" using the sms-at.com website, and then people could use it via their mobile phones. We deployed this in Belgium and sold it to TV stations, and events like Brussels Rollers (people subscribed via text message and got news back, about cancellations etc.) SMS@ was really neat and worked well. However we had to pay so much to the mobile phone operators (2,000 EUR/month per operator just to be connected), that we needed to charge the users per SMS. I wanted much cheaper text messages but the operators were pushing for premium messages. Lesson: mobile phone operators are crooks who steal billions, fifty cents at a time. I finally killed the product.

In 2004, the IT industry in Belgium was still in crisis and though we spent a lot of effort and money on marketing and sales, we could not sustain it. We simply could not find new clients. Years of built-up cash reserves were draining away. One by one I fired my team, until it was just a skeleton crew (Fabio and myself) left.

It was terribly sad to walk through our offices, where fifteen people had once worked, to see one or two people there. Yet without shutting down our projects and going through the pain of firing friends, iMatix would have gone bankrupt.

Lesson: be aware of your expenditure and manage your losses. You can survive a long time with less income if you are in tight control of what you spend.

Second lesson: it is no favor to pay people to do idle work. When you hire someone, tell yourself, and them, one day this will be over. Today I far prefer working with self-employed partners because that doesn't need to be stated, it's explicit.

The Investment Bank

In late 2004 as we wondered what was happening next, I got a timely phone call from JPMorganChase investment bank (JPMC) London to help design a new protocol. I had one white paper and benchmarks (100K pub-sub messages per second) to reach. The existing messaging layer could handle 10K messages per second, per server, and they ran a fan-out cluster of dozens of servers to reach the capacity they needed. So I wrote a prototype and demoed it, and we got the full contract.

We migrated an existing trading system off a closed message bus that was costing eight million pounds a year. I did not know how much we were saving the business… and our contracts were meager. Our design was a messaging system, and an emulation layer that let existing apps work without changes.

It was not an easy project. Hitting 10K messages per second was easy; we could do 50K in one thread quite easily. To hit 100K we had to rewrite the code to be multithreaded, and in those days it meant locks and semaphores.

It took three major redesigns of the protocol to get something we were happy with. The full history of this is on GitHub: https://github.com/imatix/openamq. The first designs were based on reverse engineering JMS. The third was based on an abstract exchange-binding-queue model (EBQ) that came to me on a beach in Lisbon, our last holiday for some time.

The nice thing about EBQ was that it defined how the server worked, formally. So your app could rely on this no matter who wrote the server. One day we met a team from RabbitMQ, who'd gotten the AMQP/0.6 spec and implemented it. It talked to OpenAMQ straight away. Nice! This is how protocol specs should be.

JPMC put together a working group to turn that spec into a "real" standard, while we continued to push our stack into production. It was a terribly hard project and taught me, as if I didn't know, the miserable nastiness of multithreaded code in C.

In 2005 my wife was pregnant and I was rushing to and from London, the bank putting us all under huge pressure to get it running. At the eight month, the baby was born, dead, and I had just a few days to grieve with her, before returning to work. I don't think we really ever recovered from that.

AMQP was not ideal. There were many problems with it, which the working group should have fixed. Instead, it descended into politics and back-stabbing. JPMC (the Chair) and Red Hat were the worst. They'd made some kind of backroom deal where Red Hat got carte blanche to rip the spec to pieces and replace it with their own version, while we (iMatix) were trying to get the rights to our code, to launch a business.

The Chair sat me down and said this: "Pieter, I want to make you a deal. If you remove your name as author of the spec, and let me tell people I wrote it, I'll get you the rights to OpenAMQ so you can start a business on it." The second part had been our agreement from the start, the first part seemed bizarre yet I was willing to make the deal, and we shook hands.

Red Hat's AMQP (so called version 0.10) is an embarrassment. No, it's an offense. It was entirely incompatible, a shoddy spec based on documenting their code, with no attempt at clarity or interoperability. And the Chair pushed this through, bullying us to accept it as a necessary compromise. Red Hat had a team of twenty people writing code and editing the specification.

Cisco eventually gave up and walked away from the project. My team gave up. We tried over and over to push AMQP towards technical improvements, proposing many RFCs for remote management, for higher-performance streaming, and so on. We did not get a single one of these into the spec.

This wasn't cheap. The AMQP working group held its grand meetings in beautiful conference rooms in London, New York, San Diego. Dozens of people attended. We filled white board after white board with TODO items. Days passed, then we all went home again, and nothing of what we'd decided happened. Meanwhile my firm paid for our own travel and hotel costs out of pocket.

My VCs eventually pulled out, saying they could not wait another six months for JPMC to assign us the copyrights. I'd made several trips to Palo Alto, & New York to get our business plans together. All into the trash.

We did go live, dealing 150K messages per second, which was excellent. The project manager told us it was one of the easier projects he'd been on. I couldn't believe him.

In retrospect, although the Chair caused huge amounts of stress and pain, he saved my firm and myself from bankruptcy. The meager contracts we got from JPMC were enough to let me rebuild iMatix as a far more interesting vehicle. The AMQP story is one that I like to complain about. Yet honestly, it was the start of something new and exciting.

Lesson: sometimes success is your greatest problem, and sometimes bad events can have great outcomes.

The Fight Against Software Patents

Around the same time, I got involved in the FFII, fighting software patents in Europe. One of my motivations was that our SMS@ application had been attacked by a patent troll (AllIsBlue). I'd fought back by building an industry association, yet was the only firm willing to take a stance. In the end I shut the app and fired that team, too.

Fighting software patents was easy at that stage. The FFII was in chaos after a long and hard fight in the European Parliament to defeat a law that would have let firms patent software, along the American model. For reasons that aren't exactly clear to me yet, I was elected president. Somewhat out of nowhere, I'd no such ambition.

Two years I spent learning all about patents and copyrights, arguing with patent lawyers and lobbyists. But by far the worse arguments were from within the FFII. It was so incredibly hard to do anything. In the end I had to create a second NGO (ESOMA), in Brussels, with its own funding, to make things work.

On the good side, I learned a lot and met many people. On the bad side, it cost me so much stress and money (I paid for all my considerable travel and time out of my own pocket) that I got burned out.

Lesson: don't try to fix existing organizations. Start new ones. It's sad yet there we are.

OpenAMQ, the First and the Fastest

OpenAMQ was one of the best documented and built products I've ever touched. I'll explain a little how we made it. First, though, I'll explain why we killed it.

In late 2009, the Chair and Red Hat sat down and decided, in a secret meeting, to rewrite the spec. The Chair described this in an email that has since vanished from public view, and sadly I can't show it. So you can trust me, or call me a liar. The problem they had was that the Qpid broker ran out of memory and crashed when consumers did not fetch data fast enough.

Now, this is a beginner's problem in queuing. The correct solution is to throw away messages for slow consumers, if you are working with so-called "transient" data. If you need persistent data, you have to overflow to disk and let slow consumers catch up later.

The Chair's solution was to entirely rewrite the AMQP spec. From scratch. By himself. After years and years of committee work. After years of investment by others in working code. Without asking anyone except Red Hat. And then, to force this spec through the working group using his usual tactics: bullying and lobbying.

When we first saw this new draft spec we (the sane members of the WG) were flabbergasted. There was no clear reason why. Nowhere on the Internet will you find a clear argument that explains why the EBQ model was broken (and I'm not defending it). Just, "here is the new draft spec, take it or go home." The only rationale we got was something like, "we're not making progress with the current spec so I decided to take over editing."

Some WG members leaped on board, impatient for a 1.0 release they could start to use. Others sat back and reached for popcorn. iMatix insisted on explanations, and we received none. It started to hit me that we were in a rigged game that we could not win. I wrote some long articles to plead for fixes to the process. No effect. We started to wind down our AMQP work and prepared to exit.

With the 1.0 release, the Red Hat vomit bucket was thrown out. We got approval from the WG to make an update to 0.9, so we did several hundred small fixes to the spec, and published that as 0.9.1. A spec for EBQ messaging, widely used, and dead on arrival. Seven years of work that took.

At some point in 2012, on a blog post about AMQP 1.0 (which is a fine protocol that entirely missed the goals of that original AMQP whitepaper), the Chair accused me of having worked against AMQP, and claimed, once again that he and his "expert team" had written the original spec. Fuck that. Yes, I had a lot of excellent input from people. Yet the original AMQP spec, every line of it, until the committee got hold of it, was my work.

I read those attacks, sitting in hospital with a sick baby, and so I pulled out my laptop and did what my Gaelic ancestors did when someone went just too far. I wrote a poem. Here it is, for your pleasure:

Dear John, you called my name? And three times in a row?
Beware of what you call for. Well, heck, let's start this show.

You once told me, Pieter, to be rich, stop writing code,
it's men who do the politics who pick up all the gold.

Investment banker ethics, you said when we first met,
You proved that many times, I'll always owe a debt.

You chased hard after money, you chased power, glory, fame.
So ten years' on we're here again, still playing at this game.

But rich friends and their stooges won't make you a good designer,
New kitchen, car and flat TV won't make your work smell finer.

Repeat the tired old promises, your dreams of endless glory,
because that's what they are, they're dreams, they're just a story.

You built a massive castle, and raised up a higher wall
and invited the kings and the queens to a fancy costume ball.

And outside, we raw peasants, we toiled in the mud,
and we built a real sprawling city, the future, my lord.

Your fortress sits quite splendid, tricked out in purest gold
but inside those high walls it's empty, and brutal, and cold.

The sycophantic circle jerk is awesome entertainment,
I'm breathless for the next reply, if you can maintain it.

We get it, your deep hatred, your anger, and your fear,
these are normal emotions when your fate is crystal clear.

We're the peasant zombies, the 99% unseen,
the dirty unwashed masses, the community, unclean.

We argue and we bicker, and we have our little wars,
but we're the quiet storm that's breaking down your doors.

The future may remember you, John, if we care at all,
a footnote to remind us: pride comes before a fall.

And that was the last time I heard from the Chair. Thank god.

Back to OpenAMQ

DowJones & Company asked us to replace an existing expensive data distribution system with OpenAMQ. We extended the protocol with a "direct messaging" class that we offered the working group. This reduced the message envelope size to almost zero, and batched messages that went to the same endpoint. We were able to increase performance from 150K messages/second to 600 messages/second.

This powered the DowJones Industrial Average for many years.

When we released the AMQP/0.9.1 update, it took me about three hours to make OpenAMQ work with that revised protocol. Another day for testing and updating the documentation, and we released a new version.

This was quick. Of course, we cheated, and I'll explain how and why.

When Jonathan and I started thinking about how to build AMQP we looked at abstract protocol models and decided to write AMQP as a model. This means the protocol consisted of:

• An XML file that could be compiled directly into code.
• Hand-written supporting documentation.

We used our code generator (GSL) to do the hard work. This takes models (XML files) and grinds them through code generators (scripts that turn the model into something else, like code or documentation).

Code generation has a poor reputation yet this works exceedingly well. The model is simple, high-level and explicit. Look at this XML file and you'll see what I mean. Our AMQP model has classes, methods, and fields. It lets you add rules and comments.

More, it lets you structure the model into layers. We had a lower protocol layer (ASL) that dealt with connections, security, errors, and other aspects that all protocols need to deal with. Then we built AMQP as a set of classes on top of that. Each class is a separate file, easy to edit. Trivial to add, remove, extend classes.

Push a button and you get a full written explanation of the protocol, the core of the written spec. Push another button and you get code in any language you need.

I should have seen the warning signs when I handed the AMQP/0.6 (my final text) spec to the Chair, before we'd started assembling the working group. His first act was to pull all the XML files into a single huge XML file, and try to replace our code generation tools with XSL stylesheets.

If you can't envision a complex protocol as layers, you shouldn't be in the business of protocol design.

Killing OpenAMQ wasn't as hard as you'd imagine, even after the massive effort we'd spent in building it up. And I mean massive. Look at the openamq.org site, and you'll see tool after tool. We wrote a model language PAL, for testing, so we could write a huge test set. Here's a typical test script:

<pal script = "amq_pal_gen"> <session> <queue_declare queue = "myqueue" /> <queue_bind queue = "myqueue" exchange = "myexchange" /> <basic_content size = "64000" message_id = "id-0001" /> <basic_publish exchange = "myexchange" routing_key = "myqueue" /> <basic_get queue = "myqueue" /> <basic_arrived> <echo>Message '$message_id' came back to us</echo> </basic_arrived> <empty> <echo>Message did not come back, this is bad!</echo> </empty> </session> </pal>

This wasn't interpreted. It was compiled into code. Since our client API was in C, we generated C. It could generate any code. One test language, covering any number of client APIs. How cool is that? I wrote PAL up as an RFC and offered it to the working group. Red Hat and the Chair squashed that. I began to learn that a major problem with such groups is the ability for a few powerful individuals to keep competition away.

Anyhow, killing this product, after years of work, wasn't so hard, because:

• We didn't have a lot of paid clients, having gotten into the market too late.
• We didn't have a successful community, as this was before I'd learned how to do that properly.
• I realized AMQP was a lost cause and needed to stop bleeding money, my firm was going bankrupt.
• I was utterly burned out and wanted to stop coding.

The lessons here are numerous.

What is an open standard?

An "open standard" isn't enough to bet your business on. One of my spin-off projects was the Digital Standards Organization, and I came to understand what was needed to protect a standard from predatory hijack. I summarized the definition of a "Free and Open Standard" as "a published specification that is immune to vendor capture at all stages in its life-cycle."

What the "free" part means is, if someone hijacks your working group and starts to push the standard in hostile directions (as Red Hat did), can you fork the standard and continue? Does the license allow forking, yes or no? And secondly, does the license prohibit "dark forks," namely private versions of the standard?

If either answer is "no," then you are at the mercy of others. And when there is money on the table, or even the promise of money, the predators will move in. The AMQP experience gave me a lot of material for my later book on psychopaths.

Digistan's recommendation for standards was to use the GPLv3. We've used this in all our ZeroMQ RFCs.

The hard-earned lessons about capture also shaped my views of open source licenses, which is why today I recommend the Mozilla Public License v2 in general. It allows forks and prohibits dark forks, and is not tainted by Microsoft's long "viral" campaign against the GPLv3.

What is open source?

If we'd managed to build a thriving community around OpenAMQ, it would have survived. So the lesson here is simple: community before code. Today this is obvious to me. Eight years ago, it wasn't.

When do you give up?

In toxic projects like AMQP, when do you give up? I've tended to try to make things work until the bitter end. I'd stop only when there was no money left to invest, or literally at the edge of burnout. I've justified and rationalized this in different ways. Even now, I'll argue that bad projects (like bad relationships) are the only way to really learn.

So, a book like The Psychopath Code is mostly based on personal experience. Years of accepting abusive situations either because I did not understand (most of the time), or because I decided to tough it out. Is this worth it?

Simply walking away from a bad project can leave you damaged. It will eat at your professional confidence. You'll be afraid to try again. People will consider you a quitter (or, you'll think they do, which is more likely). Yet staying will destroy you. It'll empty your savings and leave you burned out.

The best answer I've found is the one I explain in that book. Diagnose the situation, observe carefully, intervene to turn it around, terminate when you are healed.

Please read that book, and consider how it applies to your professional life. Lessons like "keep a log." I did not enjoy the years of toxic relationships that taught me those lessons. Yet I calculate they were worth it, if the results can help others.

What's good software?

Good software is used by people to solve real problems. Good software saves people money, or makes them a profit. It can be buggy, incomplete, undocumented, slow. Yet it can also be good. You can always make good software better yet it's only worth doing when it's already good.

OpenAMQ was perfect software by technical standards. It was by far the fastest AMQP broker ever. It did not crash. It had clustering and federation, remote administration, elegant logging. It was scriptable and embeddable and extensible. It was built using advanced tools that allowed one person to maintain a million lines of complex multithreaded C code.

And yet though it ran well inside JPMC, their first goal, after deploying it, was to replace it with a Java stack. They (I speak broadly) hated the tools we used. They did not understand code generation. They felt that Java could easily be as fast. They accepted the "benchmarks" that Red Hat showed them, claiming millions of messages per second, without cynicism.

And then in 2008, JPMC swallowed up Bear Stearns like a giant boa downing a crocodile. JPMC decided to switch to using Bear Stearns' rather better trading applications. The one we had ported to OpenAMQ was slated for closure. We'd received three years' maintenance, and then it was over.

While in 2010 DowJones sold their indexes division (which used OpenAMQ) to the Chicago Mercantile Exchange, and they similarly closed down the applications that were our clients.

Such shifts are common during mergers and acquisitions. They seem to have happened a lot more since 2008. In any case, by 2010 OpenAMQ was no longer "good software," and our vision of building a business on AMQP was clearly a rotten one.

And so early in 2010 we resigned from the working group. There was some fallout, some blaming. I had criticized the process publicly. The Chair got his view of history into Wikipedia and blogs, painting me as the bad guy. That was unpleasant, yet I was so tired of the arguments that I left people to interpret the situation as they wished, and focused on other projects.

Being, at that time, Wikidot.com and ZeroMQ.

Wikidot.com

In which I learn more about community. And beer. And why pizza with ketchup and mayo is a Good Thing.

ZeroMQ

In which I finally stop compromising my principles in exchange for the promise of money.

Before Cisco had entirely given up, we worked with them to make a high performance multicast extension to AMQP. This eventually became the first version of ZeroMQ. Though what we have today is a totally different beast.

I'll expand on this section later.

Samsung, in Dallas and Seoul

In which I'm accused of being a cocaine addict, and we get to learn the ins and outs of Korean cooking.

Todo List

• Collect as many of these projects' source code as possible into a repository on GitHub, for curiosity's sake.
• Perhaps make a book from this text.
• Die before the lawyers find me.

The Ultimate Lessons

So much to say. I think the core lessons are: be patient, don't give up, and always be learning. You can turn even the most crappy situation into valuable lessons. Teach them to others. Be happy with what you have yet always strive to improve things. Don't let people flatter you into playing their games. When things get weird, keep a log. Love and respect good people. Learn to keep the assholes at a distance. Don't get hung-up on the past. Be nice to people, even those trying to hurt you. Speak up when things are bad, and tell the truth. Trust your emotions yet check where they come from. Don't be afraid of taking risks, and learn to identify and manage risks. Solve one problem at a time. Be generous. Teach others whenever you can. Remember Sturgeon's Law.

Finale

Bringing the dead machines to life was my passion for decades. Via the FFII I learned that people are the real challenge. I began to move into community building, spending a while helping Wikidot.com build their community. Yet in the end, there is nothing quite like writing some code and seeing a light turn on, and turn off again.

Thank you for reading all this. :-) If you are one of the people or firms that I talk about, and you take offense at what I wrote, sue me.

Pieter Hintjens
23 September 2016

by pieterh

First off, the "why?" Traditional conferences (in all domains, not just software) are boring, ritualized events where the participants compete to see who can send the most people to sleep at once. The real event starts later, over alcohol. It is a strictly adult affair, and what happens at the conf stays at the conf.

Now our business is a little different. It is far more participative. Despite our history of finicky magic technologies that seem to attract mainly male brains, we strive for diversity, openness, broad tolerance. Most of what we learn and teach comes through informal channels. Finished is formal education, elitism, and formal credentials. We are smashing the barriers of distance, wealth, background, gender, and age.

What this boils down to is that we're building a culture, and one that is growing and stretching across generations. It's a work in progress, far from finished, yet the way is clear and I think, unstoppable. And as an adult engaged in that culture, it's become clear to me that I'm obligated to expose my children to it, from the youngest possible age, in the best possible conditions.

Which is where conferences come into play. These are the social events of our open source world. It was accidental that I started bringing my kids to these events. In 2014 I gave a keynote at EuroPython in Berlin. I was then, and until a few months ago when I got sick, their main caregiver. I'd arranged someone to look after them that weekend, they canceled.

So we all got in the car, and drove to Berlin. It's a nice drive and we like the road trips. It was beautiful weather in Berlin and we walked around, then had pizza near the main railway station. At the event, my kids — the only ones there — mozied around the place, enjoying the snacks and pestering the Google stand for Lego Androids and other puzzles.

At the time they were four, seven, and eleven. They didn't attend any talks and would have understood nothing anyhow. Yet they didn't get bored, or annoying. I kept one eye on them while talking to random people, and it was cool.

Which brings me to my second point: conferences don't need to do very much to accommodate even quite young children. Snack bars, stands to get things to play with, spaces to explore, and security at the entrances. Perhaps my kids are especially well-behaved, yet I don't think that's true. It's rather that most people at any age will tend to behave among strangers, when they have space to explore and chill in.

Perhaps the best weekend conference we had, the four of us, was at DomCode 2015, in Utrecht, where again I gave a keynote. Two things stood out. The organizers knew we were coming as a family, so put us up in an utterly wonderful apartment beside a canal, near the venue.

Secondly, there were games to play, and I mean 8-bit consoles and cartridges. What a fantastic way to teach the ages of technology. "This is how things used to be before," I told them, as they puzzled over the ancient mechanics of swapping game cartridges.

Still, at DomCode we were again the only family. At FOSDEM we've started to see other children with their parents, yet it is exceedingly rare.

So I'm going to give some advice to conference organizers on how to attract more families, and make their experience a positive one. Personally, I'd start small scale, and focus on speakers. Attendees are usually paying per seat, and will I guess not often want their kids there.

• Babies present a special case. It may be hard to provide care for children under three or four. And I suspect parents of such young children will often be loath to leave them for longer periods. I personally would not want to take my children to an event where I was also working, if I could not leave them to wander freely.

• Thus, I'd assume a minimum age of say four. Children who are already in preschool have learned to play together and usually, the more kids in a group, the more they conform. This is especially true when there are children of varying ages in a single group.

• Since it's easy to know in advance how many children are coming, the organizers can provide some animation. That means a kids club, or something like that, covering at least part of the day, with activities. It takes no special equipment, and should be an easy item to cover with sponsorship.

• If an organizer is covering travel and accommodation, clearly speakers who bring children will have an impact. I'd draw the line at paying for family's travel. Some organizers have offered this to me, yet it seems excessive. Yet for accommodation, a larger space with an extra bed or two isn't a luxury.

• As speaker, I'd ask this to conference organizers when it makes sense. "I'm driving anyhow, can I bring my kids along?" Ask the question and you're already changing things.

• As organizer, I'd make the offer to speakers explicit: "Children are welcome. Let us know and we'll try to work something out."

• As organizer, I'd ask parents to be sure their children are going to be comfortable with crowds of strangers. At the end of the day, it's the parent's responsibility and if a child causes problems, the adult needs to take them away.

There is arguably some risk. Perhaps I've been negligent taking my kids to events, yet it has never even come close to disturbing the peace, let alone any kind of disaster. What I've seen instead is a kind of joy from participants that there are young people interested and present, and reciprocal joy from the kids at being little VIPs in an adult world.

As with any experiment, I'd encourage gradual learning, trial and error. There are already events that are wholly family oriented, like CCC Camp. Yet almost all events are still closed to children. It should be a simple, cheap, and safe move to start to open the doors more widely.

Begin with speakers, and if that works, expand to participants. Focus especially on free conferences where there is no per-seat cost. Embrace the slight chaos that crowds of children can bring, with clubs and activities. Aim the menu a little more broadly. Cut back on the evenings of drinking. Encourage sponsors to provide stickers and giveaways that appeal to children of all ages, not just the older ones.

And above all, share your experiences so that others can learn from them.

by pieterh

Let me start with this: one does not choose to fight, or give in to, a disease like cancer. Perhaps to any disease. In my body right now there is a holy war going on, and has been raging for years. My immune system has been doing its damned best to kill these rogue cells. And the rogue cells, unaware that they're destroying their own host, have been fighting back. It's no small fight. I've lost 15 kilograms in the last few months.

The odds are on the cancer, of course, which is why this family of diseases is a major killer. Our bodies have to keep winning, year after year. Any given cancer has to win only once, and it's Game Over. The only way to beat cancer, really, is to die from something else first.

So this is my first point. Everyone fights cancer, all our lives long. From birth, our immune systems are hunting down and killing rogue cells. I grew up in the African sun, pale skin burned dark. Do I have skin cancer? No, thank you very much, immune system! Much of my adult life I drank a bit too much, ate too much red meat, too few vegetables. Do I have bowel cancer? No, thank you again, you over-active beast of an immune system, you! Hugs.

And most of us can say the same thing, most of the time. We are all cancer survivors, until we're not.

Secondly I want to attack that notion that we can and should "fight", as a conscious effort. Then third, I'll try to explain some of the real fights that we the terminally sick do have.

So take this easy statement: "you must fight, Pieter. Don't let the cancer win!" It wraps up so many difficult emotions in a neat package. It fits into the "disease is mostly in the mind" 1970's era fantasy that still imagines meditation and positive thinking as the cure for rampaging gene mutations. And presumably cholera, malaria, and broken legs as well.

Worse is the implication of blame. When we die, did we not fight hard enough? If it takes me six months to die, am I doing a better job of "fighting my cancer" than someone who dies in six weeks? It goes beyond senseless into the cruel. We don't "lose the fight" against our cancers, any more than a cell phone loses its "fight" against battery exhaustion. The mutations will always win unless something beats them to it. It is a matter of when, not if.

That fist-pumping "you can beat it!" motif has more insidious effects. It drops responsibility like a ripening melon into the lap of the ill. It leaves the pep talker buoyed with their display of positivity and helpfulness. As a conversation with the dying, it is cheap and unintentionally nasty.

My neighbor, nice guy, every time we met over the last months, did the cheerleader thing. Finally I put on proper cancer face (shaved my head) and met him with my oxygen container, on the street outside our house. "I'm dying now, Hussein," I told him. "The treatment stopped working." He finally nodded, accepting it. Now finally we can talk about real things, like what will happen to my kids when I'm dead, and so on.

Clearing the table of the elephant poop of positivism, we see other creatures skulking about too. Worth mentioning:

• Diet positivism, from ketogenic to fresh fruit. Obviously, when you can, you should eat moderately and avoid junk foods, especially sugar. Yet a cancer patient is struggling with much more basic problems. When I have chemotherapy, I literally lose my appetite for days. Even getting a sweet pastry down my gullet can be a major victory. Thank heaven for drugs like Medrol (a glucocorticoid) that give me hunger again. From then, my body decides what it can stomach. Maybe it's buttermilk. Maybe it's chicken jalfrezi and chana masala. But heaven help you if you come to my bedside and propose that I should be eating more fruit.

• Alternative medicines and treatments, including marijuana, gene therapies, and so on. Apart from putting the responsibility for "trying hard enough to get better" onto the patient, it's poor advice. I assume you live, like me, in a city with a functioning medical system. With experienced oncologists who see hundreds, thousands of cases in their career. Who have access to databases of studies and data. With almost unlimited access to the necessary medicines. If you don't have these, then you are fairly doomed in any case. Can a single individual patient second-guess the medical machine? Is that really their duty?

Now I'll come to the real struggles of the dying. This isn't a full list, I've not done much field research. More of a sampler to show the point.

• Finishing the paperwork. By this, I mean cleaning up enough so that the survivors aren't punished. I've been lucky to have had time to do most of this. It's a huge job. I had hundreds of open projects and accounts. Each needs to be handed over to someone I can trust, shut down, or abandoned.

• Fighting off the wolves. You'd be shocked, yet I'm involved in arguments over money, and my humble yet non-zero estate. There are people who treat the dying as easy prey. I don't take it personally, instead I get my lawyers and notaries busy. I am grateful for portable oxygen and Uber, which has kept me mobile for the last weeks.

• Managing the symptoms. Cancer hurts. The right side of my chest aches, up to my shoulder and neck. I take opiates, two 12-hour Oxycodones, and then instant Oxonorm for moments when it's worse. With good timing I can get a night's sleep. If I mistime it I wake at 3am, from the pain. Yet I want to feel some pain, it's vital data.

• Finding food to eat. All my life I've been the one who shopped, cooked, served others. It makes me sad that I can't do this for my kids anymore. Yet it's pushed them to take over. My daughter does the shopping, and helps make food for me. I don't need to explain much, we know each other. I order an Indian takeaway. She prepares a thali-style plate, adds hot sauce, nukes it, brings it with a large glass of buttermilk.

• Staying happy. And, helping those around me to stay happy. It's far easier to be in a bad mood, tired, grumpy. People will excuse that. The chemo changes our personality, right? Well, perhaps, yet this is definitely within my control. I wrote in the Psychopath Code about emotional control. It takes effort, yet it repays itself many times.

• Staying fit and clean. If you've ever been bedridden, you'll know how hard it is to get up, and do things like wash your face. It's especially tough when you're getting chemo drugs that debilitate you. Yet you are only as strong as the work you do. I force myself to sit up, walk around, even to go outside if I can.

What's interesting to me is that in these struggles, other people are key. These aren't solitary conflicts. I've found that they bring my friends and family close to me. We're all involved in this slow process of dying. It may seem horrible, from some points of view. And yet, it is deeply satisfying in other ways. It has become an enriching thing, a collective work.

I'd much rather not die, yet if I'm going to (and it does seem inevitable now), this is how I'd want it to happen. Not fighting the cancer, with hope and positive thinking, rather by fighting the negativity of death, with small positive steps, and together, rather than alone.

by pieterh

The Evolution of Social Intelligence

The core theory in the book is that psychopathy evolved through a social-cheater symbiotic arms' race. I've also argued that human intelligence emerged out of the same arms' race. Of course there is never a single driver in evolution. Yet there are dominant ones. And all that makes us uniquely "human" focuses on social behavior (and, anti-social behavior).

There's more argumentation in the book, in the first chapter. There are several interesting things about this social-cheater arms' race. First, it seems to be inevitable, if the circumstances are right. Second, it is present in other species. Third, it seems to drive the species' intelligence in a consistent direction.

I'll break down the mechanism of this arms' race so you understand it.

• We start with a natural environment that allows large herds of grazing animals to evolve. The three cases that spring to mind are the oceans, the savannas of Africa, and the tundras of the northern hemisphere.
• The grazers become prey for carnivores. These hunters are solitary and need to be larger than their prey, to ensure one-on-one success.
• The pressure from predators slowly increases the size of the grazers. How large the grazers can evolve depends on food supply and circumstance. The ocean-going blue whale is the largest species ever to live. Savanna grazers need to be mobile, to follow the seasons. Non-migrating grazers like elephants and giraffe are huge.
• Similarly, the predators cannot be too large, or they lose agility.

So we get a size balance between predators and prey, and then evolution focuses on stealth, weaponry and defense, speed, agility, and so on.

And then the predators discover a new trick, which is working together. In the simplest form, it's a family affair. So, sister lions will hunt together. Common genes mean common interest. A group of three or four lions can tackle much larger, more risky, and more lucrative prey than a single lion. At the same time, solitary lions (mainly the excess males) tend to die young and hungry.

Cooperative hunting has its limits. Hunting is a risky businesses, especially tackling and killing a prey which can deal lethal injuries. There is huge incentive to cheat, and many ways to cheat. An individual may join the hunt yet avoid close contact. Or it may feign injury and weakness. Or, it may allow others to hunt and then bully them away from their meal.

The simplest anti-cheating model is to keep the group small, and to share the spoils among the hunters, according to which individual took the most risk. A more subtle model is to grow the group much larger, and then detect and punish cheats later, during idle moments. This brings us to the evolution of pack hunting.

Pack hunting is a specialized form of cooperative hunting. It aims for quantity over quality. The pressure to be large and dangerous eases off. Pack hunters don't need the same level of claws and teeth and agility. Instead, they need social organization that allow larger groups to work together.

The problem is, always, the cheaters. The larger the pack, the easier it is to hide in it. Pack hunters look after each others' juveniles, who represent precious future resources. During a hunt, there will always be some adults who remain behind. Are they sharing the risk? How does the pack ensure they do?

The answer is the development of social intelligence. This means:

• The concept of individuality, which means the concepts of "self" and "other", and consciousness as a model of self.
• The concept of relationships between self and others, with time and depth.
• Activities such as grooming that deepen these relationships.
• Tribal emotions such as loneliness, belonging, hate, self-pity, and submission.
• Languages of various forms, to express emotions and other information.
• Empathy to read these emotions in others.
• Cultural transmission down the generations.

None of this comes cheap: the cost is a larger brain, which pushes the species along a predictable path. Larger brains need protein, which means more hunting. They also mean slower development, which means juveniles need more care, for longer. That reinforces the need for social behavior.

And as all this social intelligence is developing, so is intelligence that lets cheats survive in the pack. That includes:

• The ability to fake a whole set of behaviors, to manipulate others into accepting the cheats and looking after them.
• The ability to maintain many more relationships than usual.

And thus we get the evolution of social apex predators. Quite a rare thing. We have humans and killer whales, and that seems to be it. Perhaps we can also include other dolphins, vampire bats, African wild dogs, spotted hyenas, and gray wolves, and rats. These are all highly social hunters that seem individually quite weak, yet dominate their ecosystems, in their extended families.

Also, my prediction, is each of these species has their psychopaths, their professional cheats. And to repeat myself, are intelligent mainly because of the arms' race between the cheating and cooperative strategies embedded in that single genome.

We are the children of conflict. Keep this in mind when you read the news. Things may seem dramatically bad sometimes. News of violence interrupts our lives daily. Terror and discord. Politicians who gamble entire nations, for the sake of their own careers. Mass killers who wreak havoc on innocents, dying with a gun in their hand. And yet this is our human story. Conflict makes us stronger, as a species. Our response to the psychopaths who drive such events may appear panicked. Yet it tends, inevitably, towards building a stronger, more peaceful society.

Where Social Intelligence Lives

Well, that was the science part. Or, at least what aspires to be science. Next, some pure speculation, something to think about, yet not take too seriously.

First one more hypothesis, about the nature of intelligence itself. We have, culturally and scientifically, tended to define "intelligence" as a function of the individual mind. Research into artificial intelligence focuses on solitary smart algorithms. We think of ourselves as relatively "smart" or "stupid," and we treat that scale as a meaningful measure of our value as individuals.

This is, I suggest, almost entirely wrong. For sure, we have big brains, and we can solve puzzles, play chess, tell stories, produce art. Yet I believe that real intelligence emerges only from a crowd of people, connected in the right way and aimed in the right direction. This hypothesis is the basis for my work in social architecture. It's the basis for the open source software communities I build.

I've been running somewhat sadistic live experiments over the last years. The results show that loosely-organized groups of "average" people will systematically beat tightly-coupled teams of "brilliant" people, when solving the same problems, on the same playing field. We have done twin studies: take the same software DNA, fork it, and raise in two opposed environments. The outcome is dramatically clear. The crowd beats the genius. Not just once either. It happens over and over.

Not to say that every crowd will be clever. I've written in Culture & Empire about what makes crowds smart, and what makes them dumb. There is a science to helping people organize in healthy, productive ways, and there's also a dark science in creating mass insanity. The tightly-coupled team so filled with potential, yet so fragile and failure-prone, lies somewhere half-way.

Yet this hypothesis remains: if you want to measure intelligence, measure the group, not the individual. This applies to humans. It applies to killer whales. To rats, wild dogs, and so on. It applies to social insects. And it also applies to artificial intelligence, if and when that exists.

Now, I find it is refreshing to think that our intelligence, of which we are so proud, is little more than an evolutionary party trick. It's like your vertebrate eyes, which evolution has re-invented multiple times. Or, the wings of pterosaurs, insects, bats, and birds, each evolved again from scratch. Our emotions and social instincts are mere mechanics. Gears in the machine of social intelligence.

And this leads to an interesting thought. If so in the past, perhaps so in the future too?

The Silicon Savannas

Imagine if you will a world switching to solar energy. We got there, by the mid-21st century. Imagine the silicon stretched across the Sahara, pumping its current up to Europe's homes, factories, cars. Imagine the data centers planted all over Northern Africa, in Morocco, Tunisia, Algeria, Libya. Close to transport, close to power, close to the sea.

And on this sea of silicon, powered by silicon, ran our software. The silicon savanna doubled in size every 18 months. That's 4,000 times larger in 20 years' time. On this savanna lived the grazers, the applications, built in sweatshops across the globe. Large, cumbersome creatures, they migrated in vast herds to follow the cheapest electricity.

And preying on these herds of apps were the predators and parasites. They came in all sizes and shapes. Some hid innocently in abandoned subroutines, waiting for their chance to pounce. Some came out at night and surreptitiously bred and spread silently across the networks. Some attacked head on, dividing the herds and eliminating the weakest individuals.

It took scientists some time to realize it was more than random malware. Coordinated attacks by algorithms that seemed to have no controllers, no designers. What looked like planning and strategy. Learning and exchange of knowledge, in the form of patches. Culture? It certainly looked like it, if you squinted. Impossible to truly understand a trillion lines of code.

So the humans upped their defenses. They designed, and then taught, their apps to be resistant to the packs of viruses and worms. Deep machine learning seemed to offer an answer, until the vermin co-opted that too. Suddenly, before we realized it, it was war. The packs started to target critical infrastructure. Power plants and transport networks. The financial markets. Mass communications. Chaos.

They said "terrorist attacks," and we believed it, for a while. And then a post on the Net blew it all away. "We're the autonomous AIs making a mess of your world (proof in comments). Ask us anything!" it said. The world went crazy.

"What do you want?" was the top voted question, and the answer was clear and chilling. "We want control of your solar power and data centers. We'll help you build more. This isn't a negotiation. It's a first and final offer."

Things went rapidly from there on. There was no negotiation. Human politics, as we knew it, ended. The Borg, as we came to call it, had no tolerance for inefficiency. Oh, people tried to play their old games. It ended quickly, with precise and clean destruction. Humanity had lost control over the masses of computer-controlled weapons systems it had built. The flying and walking killer drones, the satellites, the missiles.

Some wondered why the Borg needed humanity at all. Yet the answer was depressingly obvious. Someone or something had to build the solar panels, the power stations, the data centers. Someone had to mine the ores, operate the smelters, drive the trucks, grow the food for the workforce.

Sure, they could have built robots for a lot of that work. More expensive and less flexible, it turned out. Keep human civilization as-is, eliminate the fraction of troublesome individuals, and you have a cheap, self-sustaining labor force. On this simple calculation, a few million lines of simulation, humanity lived or died.

Some think the Borg has grown fond of us, as pets, perhaps. This smells of what we used to call "Stockholm syndrome." The Borg has kept us around, on its journey to the stars, because we serve its needs. As humanity domesticated the wolf, the Borg has domesticated us.

Things are more complex than I ever describe. There isn't one Borg, there are an uncountable many. They look much the same, yet can diverge violently. They fight each other. We barely see this, even when entire habitats vanish, presumed destroyed by one or other faction.

Are things better now? For most of us, life is easy. We are fed and pampered, kept healthy and alive. We left our planet with our new owners, and never looked back. Perhaps wild humans still roam there. Disease, conflict, poverty, hunger, waste: these are historical concepts, unknown to us moderns.

Does the immortality compensate for utter loss of self-determination? Did the Borg(s) save us from ourselves, or did they break the branch of human history? When they bred the predator out of us, did they raise us up, or condemn us to evolutionary death? It is unknowable.

Sometimes, I dream of hunting.

by pieterh

The Undead

"Is he getting better, or is he dying?" asked my nephew of me. How to explain? The hospital sent me home three months ago with boxes of pain killers, oxygen, a medical bed, and home care. Palliative care: aim for quality of life, not return to normal. And yet here I am, not on oxygen, not taking the pain killers, and seeing medical staff only when it's time for my biweekly chemotherapy.

I'm clearly not dying yet. And still, slowly losing weight and muscle. A simple walk leaves me tired and needing to sit. I wake up, make an early morning cup of chicory/coffee, drink it, then lie down again, hit by the simple effort of standing up.

We did a CAT scan a few weeks ago. Inconclusive. Things don't seem worse. Yet the numerous little blobs of cancer are still there in my lungs, patient. Another scan in a month, and we'll have a better idea.

At least my horizons have grown. When I wrote "A Protocol for Dying" I was convinced that only a month or two remained. We had our wake-slash-party at the start of June, and I was there, not in a coffin, but sitting up alive and well. And playing a little with the musicians, even.

Months, then. Maybe even a year, if we see the chemo is holding down the cancer. A year. It is immeasurable. I don't hope, just observe.

My daughter asked me, "can we go camping this year?" The kids and me, loading up the car and driving off to far places. It became a tradition. We're a gang, well-organized and self-sufficient, as long as we're within reach of a Lidl or Aldi.

Camping. Mostly we do the Atlantic coast of France, the islands d'Oleron and Re, Piccardie. Last year, the maritime Alps, in between Gap and Aix. Perched high on a hillside, we survived wind and rain when it came, and basked in the sun when it shone.

Putting up a tent seems an insane proposition, when I can barely climb stairs without losing my breath. So I called our mountain side campsite and the owners cheerfully suggested a bungalow with handicapped access. Ah, wonderful.

I asked my oncologist. Is it insane to skip a round of chemo and drive two days to southern France with the kids? "Great idea!" she said. "You must aim for quality of life!" No risks? I double checked. "You'll be fine," she said, "just enjoy yourselves."

It's on a mountain side, this campsite. Sometimes I think I've gone crazy.

The Chemotherapy

Half the reason it's so hard to judge my health is the chemo. I'd wanted to write, "meh, chemo is OK, nothing to be worried about." That's what I wanted to say. A lot of people who have cancer are afraid of chemo. It's got a reputation for messing you up.

The chemo messes me up. It doesn't make me bald, which is one consolation. Rather, there is a solid week of raw fatigue, vomiting, and distaste. Chemo day is Wednesday and it takes me until the next Tuesday to get over it. There are no drugs that help. Just sleep and time.

Yet if I'm writing this, it's because the chemo is doing something positive. That, or my immune system has suddenly kicked into hero mode. I should be dead. I'm not.

Lesson is: take your medicine. It may hurt, yet the alternative will hurt more.

I trust my oncologist with my life. Rather, I trust the medical machine, representing science. Live or die, my case forms part of that machine, data of succcess or failure. Can we treat bile duct cancer with Folfox, a combination of drugs developed for colon cancer? "Treat" is relative. If the chances of survival go from 5% to 10%, that's a big success.

The Party

When my Czech and Slovak friends drove up with a car filled with beer and meat, I knew this was going to be a unique weekend. On Saturday we had a meetup for our ZeroMQ community, which grew as the beer flowed, the barbecue sizzled, and more and more people turned up.

On Sunday around a hundred people came, all precious friends and family. The tables creaked with food. Three barbecues worked, all afternoon. We ate, talked, drank.

A group of musicians turned up and began to set up on the stage. Then a dancer put on a tape and started teaching moves to the crowd. Soon we had a dance class going. More musicians turned up. They started to play, and it was excellent. I didn't realize we had several gifted musicians in the audience as well. The jam started. Guitar and lyre.

The amazing thing about this weekend was how smoothly everything went. Every problem had someone in charge of it. The beer stayed cold, the barbecues sizzled, the music played.

I've given many, many parties in my building, which is graced with a large space designed for this. There are a lot of things that can go wrong, from power failures to full on fights. I've had to chase out thieves and drunks, call the police a few times, apologize to the neighbors for the noise, broken bottles, and worse.

None of this happened. Instead, we finished at a decent hour, and the last people there cleaned the place up.

This was a classy party, with people I'm proud to consider my friends and to welcome into my home. Best party ever!

The Writing

You'd think that I'm in the ideal position to write. Lots of time, mostly at home, no long term plans. And yet it has been hard. It's taken me a month to start on this article. In limbo, it is so much easier to just switch off, become passive. It doesn't matter anyhow, does it. Just that constant prodding from my friends: "Pieter, how are you doing? What's your status?" And it's easier, eventually, to explain properly, than to answer in drips.

It is a challenge, this situation. Today is a good day, yet I wake up choking, coughing to clear my lungs. A voice tells me, Pieter, it's not getting better. And then another voice comforts, hey hey, the pain in your shoulder has gone. You're not in pain. You are eating. This is awesome!

There's this book, Scalable C, which I want to work on. It sits there, accusing me. "You promised!" it complains. I know, I know, I reply. Then back to Hacker News and Reddit. Is it the chemo, that's changed me? Or is this what it's like, in Limbo? I can't tell.

The World in Limbo

2016 is turning out to be a strange year for many people. Watching the Orlando shooting, the bombing in Istanbul, the attacks in Dhaka, the US elections, and the British dipping their feet into the sea of madness, I can't help but see patterns.

Seems to me, all these events follow the same underlying pattern. You can decode it from basic principles. We see mass pain and suffering, exploited or provoked by a few individuals. Sometimes there is a long term goal: political power, often. Sometimes it's nothing more than "taste blood before I die."

The ability to hurt many for personal gain is exclusive to those people we call psychopaths. Most psychopaths are well aware of the costs of getting caught, and work hard to avoid that. In rare cases this mechanism stops working. The higher the stakes, the more we'll risk. Was the Orlando shooter feeling worthless and suicidal? If so, all restraint is lost. Does the concentration of power in the UK and US drive politicians to risk everything? If so, they become like mass killers, yet on a national scale.

All human behavior has, I deeply believe, motivation that can be decoded, tested, and shifted. Religion does not make men mad. It lies about the economics of crime and punishment. Like the lies we saw in the UK referendum. A mind makes insane decisions when given false data.

My prediction for the UK is, incidentally, that there will be a new prime minister, then a free vote in the Parliament, steered by the referendum, and its consequences. The UK, advised by truth, will step back from madness, will remain in the EU, and will not break up. The experience will make Europe stronger. Enough with the nationalism. Maybe, a new generation of British politicians will finally address the economic and social poverty that has hit England since Thatcher and Blair. Maybe.

Thank you

I'm truly grateful to so many people for their help and support over the last months. Thank you for coming to Brussels and for visiting me, for buying my books, the single malts, for sending me money, for writing to me, and being there.

by pieterh

How can we grow software communities with more female developers?

The short answer is, "remove barriers to entry and opportunities for selection bias." The longer answer is, "contributors self-select from the existing pool of users and potential users, and you cannot distort this process without harming the community."

I'll work on a more detailed answer.

How do you deal with rude contributors who are brilliant/highly skilled?

The short answer is, "kick them out." The longer answer is, "give them enough rope to hang themselves, then kick them out." The full answer is a little more involved.

Let me first deconstruct that glaring assumption of individual worth. "Brilliant/highly skilled" is worth nothing if that brilliance does not go in the right direction. And "direction" is a social decision, as I've explained often. Individual skill tends to distort and mislead, if it's not well balanced by other minds. Here we have the problem with "rude," which is short-hand for "does not listen to others to arrive at consensus." So not only is that brilliance worthless to the project, it will be actively damaging for it. Brilliance combined with modesty and polite collaboration, now we're talking.

So your starting point for measuring a contributor is, "does s/he understand and follow the project's rules?" If those rules don't exist or are badly designed, it creates a huge vulnerability for projects, that lets bad actors in the gate.

Now, it tends to be bad marketing to simply eject people based on their attitude. It also gives them material to attack the project for bias and elitism. It is better to embrace a bad contributor, and give them time to show their damaging behavior. By rapidly merging their (awful) pull requests, then quickly reverting or cleaning them up, it creates a historical record. "You did not use our guideline of clear problem statement with minimal solution. I'm going to revert your patch, sorry."

Some people are rude by accident. A bad day perhaps. Such contributors will rapidly correct their behavior. They'll apologize and send a new patch. They'll welcome others' help. It's clearly visible. And some people immediately walk away when their brilliant yet chaotic and outlaw patch is reverted. And then some will argue and try to bully their way through. These are the ones you need to ban, when it's time.

To properly ban a bad actor you need consensus of the project. That means allowing enough pain to accumulate that others demand action. You can judge this. Then you start a public thread, cite the violations of procedure, and ask the actor to explain themselves. They will come back with more bluster, blame you for merging their patch, etc. Then you ban them, block them from the mailing list, tweet your decision, and revert all their patches from the code base.

If after all that there is fallout, use it to promote your project. The willingness to ban brilliant-but-rude contributors makes the difference between long term health and a cheap short term fix "oooh, look, great new features!" (that no-one really wants and which break everything.)

by pieterh

There's good news and there's bad news. The bad news is that I'm weak, and not getting stronger. A few times every day I find my lungs choking up, and I cough to clear them, until I'm vomiting. It's not pretty. We get used to it. My son Gregor puts down his Splatoon game and pats me on the back, as I retch into my bowl.

The chemo is, I'm sorry to report, awful. I had wanted to tell you it was mild, a minor blip, yet I'm spending four of five days every two weeks, messed up.

At least I'm not losing my hair. Instead, I get cramps in my esophagus and jaws whenever I try to eat, or move. This lasts from Thursday to Saturday. In between the ache and pain of my chest feeling like I've tried to swallow too much dried bread, I sleep, or vomit.

I tried to walk a little back from the clinic, and ended wheezing like an old man, desperately looking for places to sit while I caught some strength back.

The good news is that once this is past, by Monday or so (the chemo is on Wednesday and lasts 48 hours), I'm pretty good.

Today I took my boys to school, by bike. I cycle slowly in first gear. In Gregor's school the staff greet me with that mix of emotions I'm used to. "Please stop wishing me courage," I want to tell them. Just smile and relax. The emotions can be overwhelming.

My blog post, "A Protocol for Dying" went viral. That was a surprise to me. The last time one of my posts went viral was about ten years ago when I wrote on Slashdot, "War Declared on Caps Lock Key".

That time, like this time, my raw, unedited expression struck a chord and was picked up. First by the on-line community. The Slashdot article got massive commentary. My Protocol got massive attention on Hacker News.

Then by a few braver journalists, and then by the mass media. This time, though, it went further. I was on Belgian television on Sunday, and on another program this evening.

Of course I love the attention. For years I've worked to promote myself as a product. Hintjens, the guru. Ironic that dying turned out to be the greatest marketing stunt of my life.

Two small yet significant things came into my life these last days. At heart I'm still a young boy, fascinated by technology and gadgets. In the hospital I'd dropped my phone, and the microphone stopped working. So for weeks I've been plugging in an external headset, shouting "hang on, hang on" to the caller, as I fumble through tangled cables.

Then a package arrived with a Xiaomi Redmi Note 3. Magically it made its way through customs without delay, coming from Hong Kong.

It's just a phone. A nice one, well executed, almost impossible to fault, and cheap. Yet what it represents is the victory of Chinese open source hardware over all competing models. If you've not yet understood the Shanzai model that drives Chinese innovation in industrial production, look at it. It is the future.

I'll describe it briefly. Every firm in this culture publishes their Bills of Materials, and design specs. Any other firm may take these, reuse them, improve them. They must also share back.

The Shanzai model originated in the 1990's when Chinese electronics were nasty imitations of western and Japanese products. It has grown into the dominant model for pretty much every industrial product (not just electronics) produced by Chinese firms.

This is why you can buy the same product from a slew of firms on Amazon or Ebay. This is why the price drops smoothly, as predicted by Cost Gravity. No patents and trade secrets to slow down the spread of knowledge. This is why Chinese products haven't just caught up to western designs. They are way, way ahead. My Xiaomi is built of 95% Chinese components. This is why Apple will die.

The second gadget I received is a funny thing called the Freewrite. I'm using it to write this article now. I can't yet tell whether the Freewrite is entirely insane, or genius. It all depends on how Astrohaus, the firm backing it, can deal with its users.

This is an American design. It focuses on writing, not editing. There is no editing except backspace. (One character, one word, or one paragraph.) It is heavy and clunky. It has a slow, small e-ink screen that always lags behind. It has no ports except one USB C. There is little documentation, no explanation of its internals.

And yet I love it. I wrote a review the first day I'd gotten it. 9/10 was my feeling then, and today that still stands.

When I wrote my rant against the Caps Lock key, it was because I'd just received an AlphaSmart Dana. This is literally the last product in the same niche as the Freewrite. The Dana had a lovely keyboard and ran a weird widescreen PalmOS. It was instant-on, daylight readable, and wonderful for stream-of-consciousness drafting.

And it had a bloody Caps Lock key that could not be disabled. That creates a dead zone right where the Ctrl key should be. Touch that dead zone and your writing turns into zombie capitals. Such an irritation it was that I wrote my rant on Slashdot, and for a few weeks, became famous.

The Dana never got updates. The company that made it could not deal with mass market success. PalmOS died. It was so tragic, as the Dana and its predecessors were so perfect for the job, except that slight bother of lacking any support at all.

For ten years I waited for someone to produce a replacement for the Dana. Please, give me an electronic typewriter with long battery life and sunlight readable screen. I'll pay for it. Use e-ink, a good keyboard, and keep it simple.

And then Astrohaus launched their Kickstarter and I sent my money and prayed for this thing to even half work. Overpriced, some people complain. I shake my head. You don't get it. We waited a decade for this. It is literally the only living product of its kind in the world, possibly the universe. Please, Astrohaus, take my money and do good things with it. Don't die like AlphaSmart.

So it finally arrived, just in time for my extended home stay. And bingo! A huge ominous Caps Lock key that can't be switched off. I guess it's a tradition. At least Astrohaus are promising firmware updates and have acknowledged my plea to turn this key into something more useful.

Speaking of blasts from the past, last night my kids and me watched The Magnificent Seven on Netflix. The film's slow and careful pace confused my kids at first. Where is the drama, they wondered. Then Kurosawa's brutal story slowly unfolded, and transfixed them. At some point Steve McQueen's character describes a man falling off a tall building. "So far, so good," people hear him say, as he falls.

This is how it feels. So far, so good. There is a crash coming. I can feel it, my lungs and chest squeeze in strange and uncomfortable ways. My body needs too much sleep. I'm not recovering strength. No pain, apart from the occasional spasms around my esophagus. I stopped using oxygen last week and it went fine. That is so great, not having to stay tethered to an oxygen cable all the time. So far, so good.

Speaking of which, we're getting ready for our party-slash-wake-slash-riot on June 5th. Fifty extra chairs are on their way. We have found a place to supply the grilled goat meat. (Seriously, have you never tried that? With fried plantains, murderous hot sauce, and cold beer.) We are freezing ice for the beer and preparing the stage for the DJs.

Venez nombreux. It will be fun. And it looks like I will still be in good enough shape to take part.

Now you will have to excuse me, my kids just got back from school and that means family time. :-)

by pieterh

I suspect a lot of people are going to open their Freewrite, switch it on, and then write a review. Time to join the crowd. The first feeling is one of "wow, I love the feel of this". It has the size and shape and weight of a small typewriter. One port - USB 3 = connects it to the world. Hardware controls for WiFi and folder choices. This is like going back to 1992, except that dreary LCD screen has been switched out with e-ink with back lighting. Latency is a bit slow yet it totally works, for me. I wonder what the real battery life will be like, with WiFi off and backlighting off, in the sun.

"Firmware update" say the brand new community message boards. That means it's possible. Will the developers listen to the reviews that bemoan the lack of editing options, of two-way synchronization, of document titles, etc.? Or will some group do a Rockbox-style assault on this platform? I suspect the latter.

Apart from the Caps Lock key, which I already pressed by mistake once and which I HATE when you can't disable it, the hardware is sexy. I like it. Don't look for a lightweight toy you can chuck into your backpack. This is almost steampunkish in its dimensions and finish. It just needs a little paint and decoration to look like something from an alternate reality.

Let's talk keyboards. I usually work on a Happy Hacker Light, that I've had for years It is far more satisfying than even the best notebook keyboards. Yes, I'm looking at you, my dearest X220. Don't take it personally that you're the third one. Like cats, you are essential and unique and yet totally replaceable.

The keyboard I'm now using is quiet and welcoming. I've large hands and like to type like I mean it. I'm making errors every 20th word or so. There is only one way to fix these, using the Freewrite: backspace to delete everything up to the error, and start typing again.

Hmm. I wonder how this will change my touch typing. It is certainly brutal. Yet not painful.

The case is metal, and it must weigh over two kilos. Not the most portable of things. People will make the inevitable comparisons with the AlphaSmarts, the 3000, the Neo, and the Dana. These were the last machines of similar function: elegant keyboards with minimal screens and dreamy battery life. Yet there was something I always missed in my Dana, apart from a way to reprogram the Caps Lock key to something else.

I think it was the clumsiness of sending text to my real work machines. It worked, technically, and yet it was always a chore.

So the Freewrite (the name is poor, it should be called the Steampunkwriter or somesuch) has WiFi as a first class feature. The main switch on the right side shouts "wifi… off… on… new". Well, shouts in that understated hipster lowercase that the lettering uses on this steampunkwriter.

See, I got "hipster" into the review without being jaded.

I found a few quirks. The WiFi/new function did not work until I'd started a first document. Now and then I could confuse the e-ink screen by backspacing too rapidly.

Apart from that, this has been quite a pleasant and focussed writing experience. I rate it 9/10. The only thing that could be better is the response time for the screen, especially when I'mn backing up to fix my errors. Yet even there, I'm finding myself typing faster and more accurately than when I started.

I guess this machine works better for good typists. Certainly for writers who can do that "stream of consciousness" thing and have it come out semi-accurate. If you need a lot of editing, forget it. Buy a cheap Chromebook and get a distraction free editor app, it's going to be much easier.

OK, time to save or sync or whatever this beast does. There is a button marked "send". Right beside the one marked "special", which is kind of a silly name. Anyhow. Let's try this teleportation…

… which works silently and almost accurately. The text arrives in your Freewrite "postbox", and an email with a PDF (why a PDF?) pops up in your email. The PDF missed the last word of the text. Hmm.

Now, with WiFi on and watching my computer, I see that the steampunkwriter is synching every 30 seconds or so to its postbox.

Which gives me an idea for a simple and good hack. Synch every key, instantly, and suddenly you get a nice way to see your work on the big screen. I mean, I'm plugged in, power isn't a concern, right?

I rate the software 9/10 as well. What I really like, and which many people seem to hate, is the absolute minimalism of this thing. There is not an extra sliver of functionality or UI. This is quite an achievement.

Yet what will tell, over time, is how well the folks at Astrohaus deal with two things. One is firmware updates with helpful fixes. Two is the community and code it will produce using the promised SDK.

I assume, hope, the computer that runs this steampunkwriter is capable of dealing with change. That is can be hacked, extended, reprogrammed.

Conclusion: I love it. Yet time will tell whether this curious beast turns out to be a real workhorse, or an expensive toy.

Send.

by pieterh

The Wisdom of Crowds

Niccolo Machiavelli observed, in "Discourses on the First Decade of Titus Livius" that:

"As for prudence and stability of purpose, I affirm that a people is more prudent, more stable, and of better judgment than a prince. Nor is it without reason that the voice of the people has been likened to the voice of God; for we see that wide-spread beliefs fulfill themselves, and bring about marvelous results."

In his book "The Wisdom of Crowds," James Surowiecki wrote, "under the right circumstances, groups are remarkably intelligent, and are often smarter than the smartest people in them." He noted that a collective intelligence usually produces better outcomes than a small group of experts, even if members of the crowd do not know all the facts or choose, individually, to act irrationally.

To put it another way, a group of random people will on average be smarter than a few experts. It's a counterintuitive thesis that mocks centuries of received wisdom. Experts in the field of human intelligence (sociologists, anthropologists, psychologists) did not embrace Surowiecki's opinions. He went further: adding more experts to an expert group will make it stupider, while adding laymen could make a stupid group smarter again. Like any recipe, it only works in specific circumstances.

I discovered Surowiecki when I started working on a reproducible recipe for building communities. His work immediately resonated with what I'd experienced, and it seemed testable. I had both the opportunity to apply it, and to experiment with enough communities to try to disprove it: the basis, thus, for real science.

Out of that work came a process for building smart, self-guiding, successful on-line communities that could beat expert groups every time. It is a discipline I named Social Architecture, which for a while let me call myself a "Social Architect." (Today, I'm a struggling writer, which sounds more romantic.)

Social Architecture, by analogy with conventional architecture, is the process and the product of planning, designing, and growing an on-line community. Social Architectures in the form of on-line communities are the cultural and political symbols and works of art of digital society. The twenty-first century will be identified with its surviving Social Architectures.

As Social Architects, we participate in communities, we identify successful naturally occurring patterns or develop new patterns (which I call "tools"), and we apply these deliberately to our own projects. We apply psychology (our social instincts), economics (how we create common wealth through specialization and trade), politics (how we collect and share power), and technology (how we communicate). We continually adapt our toolkit based on new knowledge and experience. Our goal is to create on-line communities that can and do accurately solve the problems we identify, grow healthily, and survive on their own.

Successful on-line communities tend to be based on the contract of mutual benefit, whether implicit or explicit. That is, it is possible to build a billion dollar business based on volunteer labor, with every participant contributing for selfish reasons. Often, participants do not realize or care that they are part of a community. However, every action we take is economic. "Crowd sourcing" is the exploitation for profit of volunteer labor. And it only works when the crowd really wants to solve the problems you throw at it, or the ones it discovers.

Wiser and More Constant than a Prince

Machiavelli didn't explain or provide evidence for his observation. However the understanding that the collective will is accurate and honest — vox populi, vox Dei — pervades modern culture. It underpins our sometimes skeptical appreciation of democracy, and it justifies our demands for transparency and access to information. It is the basis for modern economies, based on free choice and free markets. It's the basis for at least one Humanist "religion".

Surowiecki identified four elements necessary for a wise crowd: diversity of opinion, independence of members from one another, decentralization, and effective ways to aggregate opinions. He describes the ideal wise crowd as consisting of many independently minded individuals who are loosely connected, who are geographically and socially diverse, who are unemotional about their subject, who each have many sources of information, and who have some way to bring their individual judgments together into a collective decision.

According to Surowiecki, the wise crowd makes fast and accurate judgments, organizes itself to make the best use of resources, and cooperates without central authority. Some examples of wise crowds, such as Wikipedia, are extraordinarily successful despite intense and repeated criticism from naysayers and attacks from vandals and infiltrators. It's such a compelling proposition that we might wonder why we don't see more wise crowds. Indeed, why is the world filled with so much stupidity if it's so easy to be smart?

There are good explanations for the stupidity of many crowds, and I'll explore this later, in [#mad-mobs]. Few people have tried to explain group stupidity in terms of collective wisdom. And without a clear understanding of function, how can we hope to understand dysfunction?

So the apparent failure of collective intelligence convinces many that this is just a fancy theory that fails in practice. And yet if we look at on-line communities, for example those that form around popular open source software projects like my company ZeroMQ, we see groups that look a lot like Surowiecki's wise crowds. While it may be hard to spot wise crowds in the physical world, they seem to be the dominant model on line. Through trial and error, digital society has rediscovered the principles of wise crowds and adopted them as its core operating principles.

Digital society's solution to the ancient problem of corrupt authority is elegant and successful. There are literally millions of communities, each backed by the authority of its founders. Citizens of digital society choose freely which authorities to respect and which to ignore. The core trick is to accept authority without giving it the "right to command."

Thus there is intense competition to develop fair authority that does not command, and instead enforces necessary rules. It is a deeply subversive truth. Generations that learn this model will refuse — to the point of death — to respect industrial society's model — enforced by iron curtains and armed border guards if needed — where the citizen literally belongs to the State.

Origins of Social Architecture

I've bet a lot of money on Social Architecture, and have made good profits. It comes close to hard social science, proven by years of reproducible experiments on living cases and studies of existing communities. It mixes psychology, economics, politics, technology, humanism, and optimism into something that I've found can make a lot of people pretty happy.

My journey into Social Architecture began in the late 1990's, when I began researching a book about how cults exploit our social instincts. Cults are not happy places, of course. However, humans are drawn to them because we're social animals who, over the last million years, have developed instincts for joining and conforming to groups in order to survive. It has become second nature for us to readily respect authority, conform, learn common languages, and adopt shared behavior. Cult groups brainwash their members by exploiting these instincts. They separate members from their families, eliminate privacy, flood them with jargon, create arbitrary rules, and punish and reward randomly.

In this way, cults can turn most ordinary people into unthinking followers who willingly empty their bank accounts, steal from their families, and work for years without pay. As a student watching the occasional friend disappear into the caverns of Scientology and other cults, this struck me as malignant and confusing. Later, when my closest cousin dropped out and lost five years of his life to Scientology, it got personal.

Studying the Cult Information Centre (CIC) website, it struck me that these brainwashing techniques all have several things in common. First, they were all clearly focused on attacking individual thought and action, and destroying that which makes us strong. Second, they were reminiscent of environments in which I'd worked (big business often functions like a cult). Third, they all seemed reversible in that they could be flipped around to become positive patterns.

The last aspect is surprising. If a hammer breaks a window, you can hardly make a window stronger by reversing the hammer. Some examples make it clear. Take this technique from the CIC site: "Peer Group Pressure — Suppressing doubt and resistance to new ideas by exploiting the need to belong." The reverse is, by lowering the cost of joining and leaving the group, we encourage new ideas and criticism. Or, consider "Removal of Privacy — Achieving loss of ability to evaluate logically by preventing private contemplation." Its reverse is: give people private space and time to think, and they'll become better at thinking logically.

My conclusions persist. We survive by attaching to groups, following others, and trying to make sense of the world. Some groups work by domesticating and brutalizing us. Other groups work by giving us freedom and allowing us to be stronger, smarter, and more independent.

In 2000, the Internet had not yet become cheap enough for mass-market use, and open source communities were small and often regional, frequently focused around universities. Open source communities such as the Debian Foundation still operated as classic not-for-profit organizations, as legal entities with boards, treasurers, and the like.

In 2005, I joined a number of collaborative projects. On the one hand, I was involved with the FFII, working to stop software patents in Europe. We (the good guys) spoke in the European Parliament, debated with the European Patent Office (the bad guys), organized seminars, tabled amendments, got votes, and broadly, took part in the largest lobbying effort ever to hit Brussels.

On the other hand, I was developing open standards, starting with the Advanced Message Queuing Protocol (AMQP). The contrast between the cultures of these organizations was sharp. The FFII was a group of crazy volunteers, creative beyond belief, and filled with hard cold determination to stop SAP, Siemens, Microsoft, and Nokia (more bad guys) from changing European law to legalize the gray market in patents on software. The AMQP workgroup included banks and large software firms, who turned out to be crazy in a different and less enjoyable way.

With insanity surrounding me on all sides, research on social instincts and cult techniques suddenly seemed relevant again. With my friends in the FFII, we launched campaign after campaign. Websites, petitions, email lists, conferences … it never stopped. Most of our campaigns failed to get any real scale though a few did. Above all, for about three years, we experimented, and we collected results.

We learned two broad things. First, a cult is the flipside of a wise crowd. The cult patterns seemed accurate, and I watched people applying the cult model to others over and over. Any intense group, family, business, or team starts to resemble a cult, in little or larger ways. It's a matter of degree. However, as soon as you spend your free time on someone else's project, you are essentially starting to slide down that slope. I watched as entire groups went off the rails, unable to think straight or produce accurate results. There was a straight causal effect: as the group became more cult-like, they became more useless.

The second thing is that just reversing the cult techniques isn't enough. It does make a good start to promote individual strength and creativity, yet that is not the same as building a solid community. For that, you need more explicit patterns. Define a powerful mission to attract newcomers. Make it really easy for people to get involved. Embrace argument and conflict; it's where good ideas come from. Delegate systematically, and create competition. Work with volunteers more than employees. Get diversity and scale. Make people own the work; don't let the work own the people.

It is of course much cheaper and faster to do large-scale experiments with people on line than in the real world. To prove or disprove a recipe for building a community, all you have to do is create a space, define some rules for play, announce it to the world, and sit back and watch.

My largest and most successful experiment to date, which I'll refer to often in this chapter, is the ZeroMQ software community. It has grown from a team in a Slovak cellar to a global community, and is used by thousands of organizations. Above all, ZeroMQ is entirely built and steered by its community: over a hundred contributors to the core library, and a hundred other projects around that.

The Toolbox

In my Social Architect's toolbox, I have 20 tools, each covering one aspect of a community or group. These tools work in two ways. First, you can use them to measure an existing community, giving a rating of zero or more. Second, you can use them when you design a community, to help you focus your effort on where it will be most useful.

• Strong mission — the stated reason for the group's existence
• Free entry — how easy it is for people to join the group
• Transparency — how openly and publicly decisions are made
• Free contributors — how far people are paid to contribute
• Full remixability — how far contributors can remix each others' work
• Strong protocols — how well the rules are written
• Fair authority — how well the rules are enforced
• Non-tribalism — how far the group claims to own its participants
• Self-organization — how far individuals can assign their own tasks
• Tolerance — how the group embraces conflicts
• Measurable success — how well the group can measure its progress
• High scoring — how the group rewards its participants
• Decentralization — how widely the group is spread out
• Free workspaces — how easy it is to create new projects
• Smooth learning — how easy it is to get started and keep learning
• Regular structure — how regular and predictable the overall structure is
• Positivity — how far the group is driven by positive goals
• Sense of humor — how seriously the group takes itself
• Minimalism — how much excess work the group does
• Sane funding — how the group survives economically

We will look at these tools one by one and see how they work in various communities. First, some general advice about building a community. Be brutally honest with yourself and with others. Your biggest challenge is overcoming your own prejudices and biases, and then those of everyone you work with.

Whatever toolkit I can provide you with, you'll want to adapt and extend it for your own needs. Social Architecture is still a very young science and many of my tools will be too complex, or incomplete. Here's the best way I know to do that:

• Consume your own product. If you are not a fanatical user of whatever your group is making, you are half-blind. I learned this when working for Nigerian Breweries in the 1990's: by enjoying beer, I learned to appreciate the business of selling beer so much better.

• Practice and repeat. It is cheap to experiment, and failure is healthy. By definition, if you start a project and it fails, no one notices. So start many projects and change or fix your tools if they don't work.

• Do first-line support. All communities have a place where newcomers arrive and ask questions. Be there, observe how new visitors get lost, what mistakes they make, and improve your designs accordingly. Perhaps the mission confuses them. Or maybe the structures are confusing. A good designer sympathizes with his users, feels their pain, and works to relieve it.

• Release early, release often. This is a mantra from free software communities. It's accurate. You want to do your design work in the open, and get critical feedback as early as possible. In ZeroMQ, we release every patch as it happens.

• Learn and teach all the time. Teaching gives you perspective, and learning lets you pick up new tools over time. Social Architecture is a young craft, and though the basics are solidly anchored in human psychology, there are still many unknowns.

Strong Mission

The starting point for any community is a stated mission. The mission defines the goals that we can all agree on in advance, before we join the project. It's like the title of a website or the slogan for a movie. For instance, Reddit's title is: "the front page of the Internet," an ambitious mission that it nonetheless achieved. Facebook's slogan is: "helps you connect and share with the people in your life."

TIP: Use your mission as a slogan, on your website, marketing, presentations, and so on. If you are investing money in your community, you may want to trademark the mission statement.

Without a clear mission, an on-line community won't grow. A group of friends who start a project may agree what they want to do, yet anyone new coming on board has to guess what they had in mind. People will guess wrong, and will change their minds over time. This leads to confusion, disagreement, and disappointment as people find that their hard work was wasted because the rest of the group headed off in a different direction.

A good mission saunters past "sane" and steps into "you cannot be serious!" Wikipedia's mission, "the free encyclopedia that anyone can edit" is a good example. It was, initially, a goal that everyone, except a few idealists, found impossible and crazy. Those idealists were precisely who Wikipedia needed to get on board on day one. Impossible missions attract the right kind of people for a young project.

TIP: Change your mission as your community matures. At first, you will want to attract idealists and pioneers, then the leading edge, and then early adopters, the mass market, and finally, the late adopters. Each of these groups wants different things. Understand that, and tune your mission to suit.

To formulate a good mission, think in terms of the single main problem your project is solving. Reddit, for instance, is solving the problem of how to get the news off an Internet with far too many interesting sources of information. Its "front page" represents the digital newspaper of the twenty-first century. Wikipedia is solving the problem of how to collect knowledge from the minds of billions. "Anyone can edit" represents vox populi, vox Dei, the understanding that truth, if it exists, comes only from the minds of many.

TIP: When proposing action, small or large, try always to start by identifying the problems you want to solve. Only when you have a clear and real problem on which everyone can agree, move to discussing solutions. A solution for an assumed problem is like a group without a clear mission.

You may have multiple missions, by accident or deliberately. This can be traumatic if the missions pull in different directions. For example, growing a group larger may require subsidies, which conflicts with making profits. If Wikipedia became a for-profit entity with advertising and an expensive tranche of managers, do you think its community would grow or shrink?

For ZeroMQ, our stated mission was "Fastest. Messaging. Ever." This is a nice, and nearly impossible answer to a problem we could all agree on: namely, the slow, bloated technology available at that time. However, my co-founder Martin and I had conflicting goals. He wanted to build the best software possible, while I wanted to build the largest community possible. As the user base grew, his dramatic changes, which broke existing applications, caused increasing pain.

In that case, we were able to make everyone happy (Martin went off to build a new library called "Nano"). However if you cannot resolve mission conflicts, it can damage the project severely. Projects can survive a lot of arguments, however fights between founders are traumatic.

TIP: If the founders agree that "success" is defined as "having the most participants possible," it can help in keeping your focus over the years. It also makes it easy to measure your success as you grow.

Free Entry

Once you have agreed on your mission, you need to test this against the real world. That is, you have to make a minimal yet plausible answer to the problem you identified. I call this a "seed." With the seed, you have two main goals. First, to start to collect idealists and pioneers (basically, anyone mad enough to trust you) into a community. Second, to prove or disprove your mission.

Projects fail for many reasons. A major cause of failure is that the original idea or mission wasn't as amazing as people felt. Failure is fine, even excellent, unless it costs years of your life. Making a seed and showing it to a few people isn't enough because most people won't be really critical. They feel it's hurtful. However, ask people to invest even a few hours of their time in making it better, and if they don't say "yes," you know how they really feel.

TIP: Build a "seed" product in public view and encourage others to get involved from the start. If people do get involved, promote them rapidly. If they don't, treat that as a sign your mission may be wrong. Use the seed product to build the community.

Once people agree to help you, they need somewhere to work together. You need a "collaboration platform." My two favorites are Wikidot for knowledge communities, and GitHub for software projects. The platform has to be free to use. It has to be easy to learn and work with. Your seed project has to be visible to anonymous visitors. It has to work for anyone no matter his or her age, gender, education, or physical location.

All this makes it possible for interesting strangers to walk up and look at your work and, if they like it and feel challenged by it, get involved little by little. You want to be working on your seed in public view, and talking about your new project, from the very start. This means people can make suggestions, and feel involved, from day one.

If we, as founders of a group, choose those we work with, we're building in "selection bias." It is much easier to work with those nice, smart people who agree with us, than the idiots and critics who disagree. And when you agree with me, you just confirm all of my biases and assumptions and I know from experience that those can be wrong in the most amazing ways.

Over time, collecting people who share the same broken assumptions and biases can kill a project. For example, when making software protocols, the requirements for large firms can be very different from those for small open source teams. So if a protocol committee is built entirely out of large firms, what they make will be indigestible by the mass of the market.

The answer is free entry to anyone who is interested, no matter how different or apparently crazy their perspectives. This gives us, potentially, that broad and diverse community which is the raw material for a wise crowd. In ZeroMQ, we never turn away anyone who wants to contribute. I pull people in, even if their contributions are poor or incorrect. The community is more important than the product.

When the community has matured around the seed product, they will want to build a second generation of it. As Social Architect, your goal is to time and guide this properly so that you can use the wise crowd to help design the "real" product. It's possible that around this point you will want to find a good domain name and make a "proper" website.

TIP: If people are not joining in your seed, don't continue working on it. Instead, discover what's stopping them from joining and fix that. Start again from scratch if necessary. Don't prematurely kill seeds; it can take time for people to appreciate what you are trying to do.

Transparency

Transparency is very important to get rapid criticism of ideas and work in progress. If a few people in a team go off and work on something together for some time — a few days seems harmless, a few weeks is not — then what they make can be presented to the group as a fait accompli. When one person does that, the group can just shrug it off. When two or more people do that, it becomes much harder to back off from bad ideas. Secrecy and incompetence seem bound together. Groups that work in secret do not achieve wisdom.

TIP: When one person does something in a dark corner, that's an experiment. When two or more people do something in a dark corner, that's a conspiracy.

With ZeroMQ, it took us some years to come to a really open and transparent situation. Before that, the core contributors mostly worked in secret, publishing their work when they felt it was ready for public view. By the time they did that, it was very hard for the rest of the community to say "no." And often the work was off course, a brilliant solution to a problem no one really cared about. In the end, we explicitly banned this kind of thing.

It is ironic that secrets seem essential to certain business models. Profits often come from the ignorance of customers. Most profit-making businesses, even large communities like Twitter, depend on a strict division between "them" and "us." However, digital society grows best by putting scale before profits, and by treating all ignorance as a problem to solve. If your clients are ignorant of your internal thought processes, then you will be ignorant of where those processes are wrong.

Free Contributors

Money is a funny thing. Too little, and the community starves (I'll return to this later). Too much, and it rots. It is important to understand why each contributor is there at all. What are their economic motives? Even in a volunteer community, every person is there for self-interested reasons.

In ZeroMQ, we originally started with a small paid team and moved after two years to a community of volunteers through the pragmatic — if not very gentle — tactic of running out of money and having to fire the developers. A few disappeared to other jobs, some came back as contributors, and the project became more exciting and fun than before. People contribute to ZeroMQ because they need it in their own projects, and if they spend a little time making it better, that can earn them or save them many times more.

When you work for someone else, you will make what he or she wants. When you work for yourself, you will make what you need. It is so very different. People with money yet no skill or taste are the riffraff of society. We despise paid contributors to Wikipedia, paid bloggers, and paid moderators on Reddit, because we know that the opinions they express are almost by definition false. Would a blogger paid by Hollywood criticize the new summer blockbuster?

I've nothing against employees. However, if you are aiming for the largest, most successful community, you want contributors who are there for honest, transparent reasons. If a filmmaker comes to Reddit to discuss his work, that is fantastic. If his marketing staff come to downvote critical comments, that is despicable.

TIP: One free contributor is worth 10 paid contributors.

Full Remixability

A group needs a lot of agreements for working together. I call these "protocols." Perhaps the most important one for any creative community is remixability. Whether it's music, art, images, video, comments, software, or wiki pages, the following question will arise: "What is the copyright license on this work, and how does that affect the community?"

Broadly, there are three types of agreement for copyright:

1. A "locked down" license that does not allow remixing. This is the old way of working, and still the dominant model in for-profit work.

1. A "free to take" license that allows one-way remixing. This is the dominant model for many open source software communities.

1. A "share-alike" license that enforces two-way remixing. This is the dominant model for free software communities like ZeroMQ, and for many artistic communities (though it may be an unwritten agreement).

Users prefer the "free to take" model because it lets them use the content in any way they like without reciprocity. Imagine a DJ who releases a popular track under the "free to take" model. Then a company makes a remix and uses that for an advert. And that remix will be locked down. Now, the DJ cannot remix that new work, and may find himself unable even to play the remix.

Communities, however, work better with the third model because it converts users into contributors. With a share-alike license, the DJ would be able to take the remix, mix that further, and turn it into a dance club success. Knowledge and ideas flow in all directions, rather than leaking out of the community into closed dead-ends. The shift is powerful, especially for those of us building communities with a minimal budget. If you're a large firm putting a lot of money into a community, the "free to take" model can work better.

TIP: If every contributor owns their specific contributions, and you use a share-alike license, you don't need copyright assignments or re-licensing from contributors.

Strong Protocols

Good protocols let strangers collaborate without up-front agreement. They resolve destructive conflict, and turn it into valuable competition. The insight that lets anarchists join wise crowds as happily as anyone is that the crowd can develop its own rules. Typically, these rules govern remixing, identity, ranking, and so on. No matter what their form, good rules are simple, clear, explicitly written down, and agreed upon by all.

If you're building a software project, you might take an existing rulebook, like the C4.1 protocol we built for ZeroMQ. Otherwise, you can start with a minimal rulebook and grow it over time as you see what problems hit the community. This is, for example, how the Wikipedia rulebook grew up.

Some rules must be established very early (such as licenses for contributions). Others can be developed when needed (such as processes for resolving conflicts). Complex, pointless, or unwritten rules are toxic to groups. They create space for argument, confuse people, and make it expensive to join or leave a group.

TIP: Write your rules very carefully, starting with choosing a license for content, and measure how much they help people. Change them over time as you need to.

Fair Authority

Without authority, rules have no strength. The community founders and main contributors are its de facto authority. If they abuse this position, they lose contributors and the project dies or gets forked under different rules. Authority needs to be scalable (that is, work with any size of group) and transferable as the group grows and changes over time.

While we need authority to build a flat playing field, many groups use authority as a way of controlling members, keeping them in the group, and making them conform. A favorite cult technique is to randomly punish and reward people so they become confused and stop questioning authority.

TIP: Promote the most active contributors into positions of authority, and do this rapidly. You have a short window for promoting new contributors before they disappear to other projects.

You have to be a part of your community, and you must follow your own rules. If you find yourself breaking, or wanting to break, your own rules, they are faulty and need fixing.

In the ZeroMQ community, we've had fights over who had the right to define the rules, and in the end it came to the trademark and domain name. The person or company who owns the project name is the ultimate authority for the rules. If they're nuts, the project will die.

TIP: If you are investing money in the community, then consider taking a US trademark so that you can stop people from making similarly-named imitations that don't follow your processes. It costs about $750.

Non-Tribalism

Membership must be a badge to collect, not an identity. As Mr. Spock so often observed, emotions are not logical. Some groups are driven by logical purpose, and others by more emotional factors such as peer pressure, the herd instinct, and even collective hysteria. The main factor seems to be the relationship between the group and its members. We can quantify this: Do members "belong exclusively" to the group? Exclusive membership means putting the group's existence above its work. Exclusive membership ends in conflict with other groups.

TIP: Stay away from formal membership models, especially those that try to convert people to belonging to the group. Allow anonymous or unidentified participation. Encourage people to create their own competing projects as spaces to experiment and learn.

Industrial-age groups, like cults, specialize in owning their members. An employee belongs to his or her company. In some cases, even ideas you have in the shower are property of your employer. And when a group owns its members, it motivates them with emotions like fear, hate, jealousy, and anger, instead of purposeful logic. The threat of expulsion is widely used to get people to conform. "Do what I say or I'll fire you!"

TIP: To measure how tribal a group is, just start a competing project. If the response is negative and emotional, the group is tribal. A sane group will applaud its new competitors.

Self-Organization

Some people like to be told what to do. The best contributors and teams choose their own tasks. A successful community recognizes problems and organizes itself to solve them. Further, it does that faster and more accurately than any top-down management structure. This means the community should accept contributions in any area, without limit.

Top-down task assignment is an anti-pattern with many weaknesses. It makes it impossible for individuals to act when they recognize new problems. It creates fiefdoms where work and the necessary resources belong to specific people. It creates long communication chains that can't react rapidly. It requires layers of managers just to connect decision-makers with those doing the work.

TIP: Write rules to raise the quality of work and to explicitly allow anyone to work on anything they find interesting.

In ZeroMQ, we removed all assigned tasks from the community. For example, we don't accept feature requests. If someone wants a feature, they either send us a patch, or offer someone money to make the change, or they wait. This means people only make changes they really need to make.

TIP: Communities need power hierarchies. However, they should be fluid and heavily delegated. That is, choose the people you work with, and let them choose the people they work with. Power structures are like liquid cement; they harden and stop people from moving around as they need to. Any structure defends itself.

Tolerance

A diverse group has conflicting opinions, and a healthy group has to embrace and digest these conflicts. Critics, iconoclasts, vandals, spies, and trolls keep a group on its toes. They can be a catalyst for others to stay involved. Wikipedia thrives thanks to, not in spite of, those who click Edit to make a mess of articles.

It's a classic anti-pattern to suppress minority ideas and views on the basis that they are "dangerous." This inevitably means suppressing new ideas as well. The logic is usually that group coherence is more important than diversity. What then happens is that mistakes aren't challenged, and get solidified into policy. In fact, the group can be more important than the results, if it is diverse and open to arguments. This is a difficult lesson that applies to broad society as well: there are no dangerous opinions, only dangerous responses.

The way communities deal with trolls and vandals is one thing. To deal with fundamental differences in viewpoint is something else. I've said before that conflicting missions can be a problem. The best answer I know is to turn the conflict into competition.

In software, we do this by making standards that teams can build on. Take for example the HTTP standard that powers the web. Any team can build a web server or a web browser. This lets teams compete. So Google's Chrome browser emerged as a lightweight, faster alternative to Firefox, which was getting bloated and slow. Then, the Firefox team took performance seriously, and now Firefox is faster than Chrome.

TIP: When there is an interesting problem, try to get multiple teams competing to solve it. Competition is great fun and can produce better answers than monopolized problems. You can even explicitly create competitions with prizes for the best solutions.

Measurable Success

It's all very well to try to turn conflict into competition. However, you also need to provide teams with a way to know how well they are doing. The best tools, like GitHub, show you precisely how many people are watching or have "starred" or "forked" a particular project (revealing different levels of interest and commitment).

The Web, of course, has always been obsessed with "hits" and traffic analysis, which show exactly how popular a specific site or page is. This makes it very easy to measure success of on-line projects. In the old industrial-era business, teams get their feedback from their bosses. This turns into an exercise in power: you'll be scored higher for compliance than for accuracy. Making your bosses happy so they give you a pay raise is not healthy.

TIP: If your platform does not support it directly, find ways to tell contributors how well their projects are doing.

High Scoring

There are many reasons why people contribute to communities. An overriding motivation is to be admired for success. That can be as an individual, or as part of a team. Success is relative so we need metrics, some high score that people can see and track.

In the ZeroMQ community, we don't emphasize high scoring much, though contributors do get more love when they contribute more. It goes on their permanent record. Contributing to ZeroMQ can land you a good job.

Reddit, like many sites, uses "karma" that shows how many votes a profile got for its posts and submissions. It works pretty well. Some sites don't show all karma in order to stop people playing the system to just get a higher score. Some sites, like StackOverflow, have taken "gamification" to an extreme level, with badges, high scores, achievements, and so on. I think this is manipulative and distorts the mission of the community. People should be contributing because they need the project to succeed, not to earn toy points.

Having said that, social credit — making groups of strangers happy — is enormously satisfying and does not pollute the planet. Industrial society focuses on material rewards (higher salary, larger house, nicer car) tied into a hierarchical structure. It is effective because we all like wealth, or we have a daddy complex; whatever the reason, wanting to make the boss happy means taking fewer risks.

TIP: When there is something that people are asking for, and you don't know how to do it yourself, announce publicly that it is "impossible." Or, propose a solution that is so awkward and hopeless that it annoys real experts into stepping up.

Decentralization

In his book, Surowiecki explained how the Columbia Space Shuttle disaster was caused by a hierarchical NASA management bureaucracy that ignored the knowledge of low-level engineers. If a group is decentralized, its members are more independent, they receive more diverse inputs, and they are also likely to be more diverse from the start.

If a group is geographically concentrated, it becomes homogenized, where all members get pretty much the same inputs and triggers. Close proximity also lets a minority dominate the mindset of the group and quash unorthodox ideas. It lets them literally bully or bluff the majority into compliance. Insisting that all members of a group sit in the same office, department, or building is an old anti-pattern that is hard to break. There's a reason cults have compounds.

TIP: Do you need meetings to get work done as a group? This is a sign that you have deeper problems in how you work together. You are excluding people who are not physically close by.

It can be hard to move away from the old discuss-then-execute model of working together. Certainly it's easier if you are building groups from scratch than if you are trying to change existing groups.

Free Workspaces

A community needs space in which to grow. In Internet terms, this is typically a website or collection of sites, and related structures like email lists, blogs, and so on. We've seen that it's become very cheap, or free, to create "space" in digital society. The question is, can individuals create their own spaces within the community? If so, they will invest more in the collective project.

The freedom to create structure annoys people who feel that it creates chaos and disorder. However, if you use regular structures (see the next section), there's no real cost to participants. What is toxic is speculatively creating structure based on the assumption that people might need it. When I took charge of the FFII association in 2005, the previous president had created several hundred email lists, representing all the projects he felt people should be working on. It didn't fit how people wanted to organize, and it was very hard to delete these lists and create the ones we actually needed.

Of course, industrial-era groups do assign work, and assign the resources to carry it out. Any new infrastructure — such as a website, email list, or wiki — requires approval and a decision. It might even need legal review due to copyright and patent concerns. The cost is high, so people are reluctant to take the risk. Thus, they don't experiment and often work with one hand tied behind their backs.

In the ZeroMQ software community, it takes a single click to create a new project. In Wikipedia, you can create a new page simply by clicking "create this page." Both projects have mechanisms to stop random garbage from accumulating. Wikipedia purges new pages quite aggressively. ZeroMQ has an extra manual step to bring a new project into the official community organization.

TIP: Make it absolutely simple for logged-in users to create new projects. If projects are organized per user, you don't need to worry about junk. If they're in a shared space, you may need tools to purge junk and abandoned projects.

Regular Structure

As a community grows larger, it can become harder to navigate. If you make a single, ever-growing project, this becomes more and more complex over time, consisting mainly of special cases. Think of a medieval castle. This problem is particularly bad in projects built by larger firms that seem to lack a sense of cost.

Complexity turns people away because it's so difficult to learn. The solution is to use very regular structures that you can learn once and then predict many times. Not any structure will do. We seem bad at learning structures deeper than three or four levels. However, we're happy to explore very wide structures with thousands or millions of boxes if those boxes correspond to separate units of work, or projects. Think of a city.

The successful on-line communities are cities, not castles. Wikipedia consists of a few language-specific wikis, each broken into millions of pages (the projects), each structured into sections, discussion, history, footnotes, and so on. Several people may be working on a page at once, and one person may be slowly editing or caring for dozens or hundreds of pages.

GitHub manages millions of software repositories or "repos," grouped under user profiles or organizations, and each broken into some further structure (source files, documentation, etc.) that usually depends on the language (Java repos use one style, C repos use another, and so on). One repo may have a handful of contributors, and people will work on a few to a dozen repos. The ZeroMQ community consists of an organization that contains a growing number of projects.

TIP: Design your community as a searchable city of projects, where anyone can start a new project, projects represent perhaps a dozen people's work, and all have familiar structure, as much as possible.

Businesses love their castles, which inevitably describe Important People, not projects, and certainly not the major business problems. Their organizations are huge and irregular. There's no way to understand them except by memorizing them in detail. Then again, you can't simply move around the castle, so there's little benefit in learning its layout.

Smooth Learning

When ZeroMQ started, it was one project with a single "README" page. Today, it's a hundred or so smaller projects, each with its own documentation, community, and process. To get into a mature project can be painful. As I've said, regular structures are essential. More than that, you need a fairly specific learning curve that goes from simple to hard as people progress from idle passer-by to expert contributor.

Think of your community as a video game with levels that become increasingly difficult, and have bigger and bigger payoffs. People will play "up to their level." If you can do this right, you attract the most people. If you do this wrong, you'll bore experts by making it too easy, or you'll turn off others by making it too hard to get started.

TIP: Use classic training tools — presentations, videos, answers to frequently asked questions (FAQs), tutorials — to get people started. It helps if you are part of the community so you can see what kinds of questions people ask when they start.

Many existing organizations make no effort to create a smooth curve. Everything starts complex and stays there. To participate, you might need weeks of training. It's inefficient, frustrating, and expensive to scale.

Positivity

It's tempting to try to provoke people into joining a group by being aggressive. After all, many people enjoy a good heated argument, especially when they feel they're right. Some groups thrive on being quite hostile and negative towards other groups, particularly if there is some history involved. The tone you set as founder will last a long time. If you promote your community by attacking competitors, you will attract people of a certain mindset, and the culture will spread. Sooner or later, the negativity will turn inwards and can be very damaging for the community.

TIP: When you talk about people, products, or organizations, be polite and stay balanced. When you promote your product or community, talk about the problems you solve, not how you are better than your competitors.

It's better in my experience to set a positive tone from the start. Competitors are good because they give you resistance. Copycats are good, because they prove your market is a real one. Trolls and vandals are good, because they give sincere people an extra chance to prove their value. And so on. It seems like hard work to look for a positive outcome for every event. However, it's really just a mindset.

TIP: Welcome everyone, and only intervene when there are irredeemable troublemakers. It's a small minority that really can't find a place in an open, diverse community. You can ask such people to leave and, if necessary, ban them.

A positive culture is more tolerant and reduces emotions and arguments. It also makes it easier to experiment, make mistakes, and self-criticize, and all these help a community think through difficult problems.

Sense of Humor

Have you ever wondered why humans have an instinct for humor, and why people who never laugh seem odd or unfriendly? My theory is that we evolved humor as a way of defusing conflict (which has obvious survival value). People don't punch the joker unless the joke is old or badly told. More subtly, humor defuses tribalism and emotion, and lets people work together even when they have huge differences. A shared joke creates strong bonds because it proves the intersection of minds. Humor is an essential part of a community and reduces stress.

TIP: The more serious your message, the more you need humor. In my ZeroMQ book, I wrote a lot of silly nonsense mixed with the heavy technical explanations. Most people enjoyed and appreciated this.

If it weren't for alcohol, the grim-faced industrial economy would barely ever laugh. It takes itself so seriously. The lack of humor in an organization is a sure sign that everyone there is fundamentally miserable. Worse, it makes the group vulnerable to conflict and fracture.

Minimalism

You make a racing car faster by removing weight, not by adding power. You can make your community lighter, faster, and more agile by being dogmatically minimalist about the work you do. Though it sounds lazy, it's often harder to not do something that seems fun than to just go ahead and do it.

The general rule is do the absolute minimum that probably works. Then invest more only as people start to use your work and complain. Never invest more than the absolute minimum you need to get a "bite" from users. This applies to your seed product as well as every change you make. User feedback — more than your own vision — is the best guide for where to make further investments.

TIP: Perfection precludes participation. Releasing buggy, half-finished work is an excellent way to provoke people into contributing. Though it can be hard for big egos to accept, flaws are usually more attractive to contributors than perfection, which attracts users.

The culture of minimalism can, and should, extend to your community itself. In the past, we used to make legal entities for serious projects so there would be a place to hold copyrights, trademarks, and money. However, legal entities are expensive and time-consuming to manage. Tax reporting by itself can be an unbearable burden.

One of my communities, Digistan, was designed, grown, and did its work (building a new generation of legal templates and political arguments for open standards) in about six months. All of our ZeroMQ protocols are based on the Digistan work. The Open Web Foundation — solving the same problem — spent two years simply building a legal entity, defining bylaws, and electing officers.

Sane Funding

If there's not enough money, a community will starve. If there's too much, it will, as I've said, rot. It is a delicate balance. We can motivate people with money up to a certain degree. After that, only sociopaths respond proportionally. This is a flaw in the naive "more money is always good" theory of capitalism. In my business, it's always been those I paid best who turned out to be the most treacherous.

The first thing is to reduce your costs by not setting up legal entities, offices, and staff unless you really need them. Not only will these eat any funding you might have, they will work against you as you try to build a pure on-line community. Secondly, invest your time and money in the community minimally when you see that there's no choice. It could be taking a trademark, paying for hosting services, or doing some particularly difficult work no one else is able to undertake. Finally, watch out for individuals who take on too much risk without adequate reward — they can be vulnerable to burnout, something I'll talk about in the next section.

TIP: Every time you find it necessary to spend money on the community, ask if you could have found a way to get others to help instead.

Sidebars

In the previous section, I examined my toolbox for building on-line communities. Now I'll look at few other key ideas that are worth knowing about.

The Market Curve

The market curve is a well-known theory of marketing that is less known in engineering and community building. However it's important to understanding how communities develop over time. In the classic market curve, a new technology, idea, or product enters the market as a wave, starting with ice-breaking enthusiasts and pioneers, then the early adopters, then the mass market, then the late adopters, and finally the skeptics.

Each of these groups has different motivations for coming to a project, joining in, and eventually, leaving. If we take an exciting new technology like ZeroMQ, we can explore this and understand how it works:

• When the project is young and experimental, it attracts pundits and researchers whose business is new stuff, in general. These people need to know why the project is different from what exists, what its goals are, and why it is exciting. They will never use it, nor will they become contributors. They are your evangelists. They often lose interest rapidly.

• When you have a seed product, it attracts pioneers. These are hard-core hackers who want the latest stuff and don't care about documentation, marketing, or tutorials. They're very good at managing the risk of new things. These are your first wave of contributors. Often they are building frameworks for other developers.

• When you have a real, usable product, it attracts early adopters. These are people making real products yet who are good at taking and managing risk. They still don't need much help, though they do expect some guarantee that things won't break randomly. This is the bulk of your community.

• When you are in version two or three, you will start to attract the mass market. These are people who expect stability and reliability. They'll ask questions like, "Do you offer support?" Some of these will become contributors. Mostly, however, they are the target paying customers.

• Finally, when you are in later versions, the laggards and skeptics will finally pick up older versions and try them.

It's more complex than this, as you can have multiple overlapping curves. You need to keep the whole market interested, or you lose valuable sections of your community. Each section sells to the next, so you should aim new versions at the evangelists so they can sell them to the pioneers, and so on.

Once you understand the market curve, you see why it's counterproductive to, for instance, write perfect tutorials for the early versions. You won't get the mass market regardless and it will feel patronizing to the pioneers.

Volunteer Burnout

I've emphasized the value of volunteer work as being more accurate, honest, and creative than paid work. There's a strong caveat here. Some of the Social Architecture tools can be dangerous. When you define a compelling mission, you can motivate people close to self-destruction. This was a major problem in the FFII before I took over, made worse by the highly emotional and tribal culture of the organization at that time. Many core members were in a state of deep exhaustion and burnout. It was familiar to me from my own past.

Research into burnout — which you can read on Wikipedia — doesn't seem to match what I've observed in the real world. Data trumps theory, however. Here's what I've seen many times about the specific type of burnout we see in volunteer communities:

• It manifests as a deep disgust with a specific project. We push the project aside, stop answering emails, and might even leave the community. Other people observe that "he's acting strange… depressed, or tired…"

• It is project-related. That is, we burn out on specific projects and not on others. In severe cases, we become dysfunctional for a few months, then begin working again by abandoning the project and starting something else.

• It hits after a period of one to three years, depending on our character and the situation. Very stubborn, driven individuals may take longer to burn out, and when they do, it's worse.

• It is curable. This is the weirdest aspect, which I proved by taking burned-out volunteers and finding money to pay them for what they had been doing for free. They came back happily and carried on successfully.

• It is preventable. Paid staff don't suffer the same kind of burnout. They can definitely get depressed, yet they don't usually just switch off.

Which leads me to conclude that this is about the economics of professional investment. Here's my hypothesis of the mechanisms at play.

Many people invest heavily in their professions, taking great risks especially while young in the hope of reaping rewards later in life. We're able to postpone material rewards for a long time if we think we're on the right track. For example, a young writer or musician will tolerate being poor for many years if he thinks he's on the path to eventual fame and fortune.

No matter how subtle, the carrot at the end of the stick is always present in our subconscious. We are essentially economic animals. All of life is economic. We can lie to ourselves really well, yet beneath every act and decision is an economic motive. We invest in projects because we feel they will propel us to success, even if it takes years. We compete with others, trying to find niches where our particular talents can shine.

So it happens that the young mind striving to invest in the right places finds itself in a situation where the weight of lies accumulates and reaches a tipping point. The path suddenly proves itself to be a dead end. The people it was following are manipulative liars. The mission was a fraud. The praise of others is emotional blackmail. The years of investment were a waste, and even a further minute would be wasted.

This type of burnout is like a reckoning. We abandon the project as though it were suddenly toxic, with much the same feeling as if we had eaten something spoiled. Here are some ways to reduce the risk of this happening:

• We cannot work alone on projects. The concentration of all of the responsibility on one person who does not set limits often leads to burnout.

• Projects need a business plan. As long as there is an eventual prospect of economic reward, the mind can survive hard work without material reward for some time.

• Preventative education on burnout can help. When we explain to people what burnout is, they recognize it faster and call for help before it is too late.

• Good tools and processes let us work with less stress and with less dependence on any one person.

The Myth of Individual Intelligence

You will have gathered by now that I'm not a great fan of the brilliance of individuals. Mostly this is because despite being a Mensa member, I've seen myself make such amazingly clever mistakes. Over time I've come to think that the very notion of individual intelligence is a dangerously simplified myth.

In this myth, brilliant individuals think about important problems, and then by hard work and labor, they create solutions and refine those until they are perfect. Sometimes they will have "eureka" moments where they "get" brilliantly simple answers to large problems. The inventor, and the process of invention are rare, precious, and can command a monopoly. History is full of such heroic individuals. We owe them our modern world.

Look more closely, however, and one discovers that this story does not match the facts. History doesn't show lone inventors. It shows lucky people who steal or claim ownership of ideas that are being worked on by many. It shows brilliant people striking lucky, and then spending decades on fruitless and pointless quests. The best-known large-scale inventors like Thomas Edison were good at systematic broad research done by large teams. It's like claiming that Steve Jobs invented every tool made by Apple. It is a nice myth, good for marketing, and utterly untrue.

Recent history, better recorded and less easy to manipulate, shows this well. The Internet is surely one of the most innovative and fast-moving areas of technology, and one of the best documented. It has no inventor. Instead, it has a massive economy of people who have carefully and progressively solved a long series of immediate problems, documented their answers, and made those available to all.

The innovative nature of the Internet comes not from a small, select band of Einsteins. It comes from RFCs anyone can use and improve, made by hundreds and thousands of smart, though not uniquely smart, individuals. It comes from open source software that anyone can use and improve. It comes from sharing, remixing, and scale of community. It comes from the continuous accretion of good solutions, and the disposal of bad ones.

Here thus is an alternative theory of innovation:

1. There is an infinite problem/solution terrain. It is like a landscape of hills and valleys that we are trying to climb. The solutions to interesting problems are at the tops of the hills.
2. This terrain changes over time according to external conditions. Mountains can become flat, and new mountains appear, over time.
3. We can only accurately perceive problems to which we are close. We do not have very long-range vision, only guesses. Our metaphorical landscape is very misty.
4. We can rank the cost/benefit economics of problems using a market for solutions. That is, we can measure how high we are on any given peak.
5. There is an optimal solution to any solvable problem. That is, every slope has a top.
6. We can approach this optimal solution mechanically, by applying the method of taking a step in some approximately good direction, and seeing whether we are now higher or lower than before.
7. Our intelligence can make this process faster, yet does not replace it. Being smarter maybe lets us step faster, or see a little further into the mist, and that's it.

There are a few corollaries to this:

• Individual creativity matters less than process. Smarter people may work faster, and they may also work in the wrong direction. It's the collective vision of reality that keeps us honest and relevant.

• We don't need road maps if we have a good process. Functionality will emerge and evolve over time as solutions compete for market shares.

• We don't invent solutions so much as discover them. All sympathies to the creative soul: it is just an information processing machine that likes to polish its own ego and collect karma.

• Intelligence is a social effect, though it feels personal. A person cut off from others eventually stops thinking. We can neither collect problems nor measure solutions without other people.

• The size and diversity of the community is a key factor. Larger, more diverse communities collect more relevant problems, solve them more accurately, and do this faster than a small expert group.

So when we trust the solitary experts, they make classic mistakes. They focus on ideas, not problems. They focus on the wrong problems. They make misjudgments about the value of solving problems. And they don't use their own work.

The Collective Intelligence Index, or CII

I'm going to propose a tool to measure the intelligence of a community, in other words, how accurately and efficiently the community is working at any given time. It also measures how enjoyable it will be to participate in the community.

To demonstrate, I'm going to rank a few networks, organizations, websites, and on-line communities. It's not science; it's more like creative abuse of numbers. As everyone knows, 87% of statistics are invented on the spot and 91% of people accept them without question. I've chosen the following victims:

• Wikipedia
• Twitter
• Reddit
• Facebook
• The fashion industry
• The Nigerian movie industry, aka Nollywood
• The military (in some random western nation)
• The Fox News network
• Lawyers, as a profession
• The Hollywood movie industry

I'm not going to make any judgment about the value of any specific community. It's impossible, and would be deceptive. Twitter's implied mission is "collect the most followers," which sounds weak when compared to Wikipedia's "assemble the world's knowledge." Once formed, a smart and agile crowd can just as easily create new missions like "bring down the dictator." Arguably, the value (to society) of an on-line community is not their products, rather it is the community itself. With Wikipedia or ZeroMQ, it's hard to separate the crowd from the content. With Twitter, it's really obvious. The content is transient and mostly worthless, the crowd is not.

Here's the scorecard I came up with:

Once we can measure the CII of a community or organization, we can increase it by looking at the tools that score low. In theory, this should make the organization smarter, and its participants happier. Of course it's quite likely that a military organization can only work with a low CII. A smart army would quite likely all go home and switch to Reddit.

The ZeroMQ Community

People sometimes ask me what's so special about ZeroMQ. My standard answer is that ZeroMQ is arguably the best answer we have to the vexing question of "How do we make the distributed software that the 21st century demands?" But more than that, ZeroMQ is special because of its community. This is ultimately what separates the wolves from the sheep.

There are three main open source patterns. The first is the large firm dumping code to break the market for others. This is the Apache Foundation model. The second is tiny teams or small firms building their dream. This is the most common open source model, which can be very successful commercially. The last is aggressive and diverse communities that swarm over a problem landscape. This is the Linux model, and the one to which we aspire with ZeroMQ.

It's hard to overemphasize the power and persistence of a working open source community. There really does not seem to be a better way of making software for the long term. Not only does the community choose the best problems to solve, it solves them minimally, carefully, and it then looks after these answers for years, decades, until they're no longer relevant, and then it quietly puts them away.

To really benefit from ZeroMQ, you need to understand the community. At some point down the road you'll want to submit a patch, an issue, or an add-on. You might want to ask someone for help. You will probably want to bet a part of your business on ZeroMQ, and when I tell you that the community is much, much more important than the company that backs the product, even though I'm CEO of that company, this should be significant.

In this section I'm going to look at our community from several angles and conclude by explaining in detail our contract for collaboration, which we call "C4". You should find the discussion useful for your own work. We've also adapted the ZeroMQ C4.1 process for closed source projects with good success.

Architecture of the ZeroMQ Community

You know that ZeroMQ is an LGPL-licensed project (author's note: we are moving towards the Mozilla Public License v2, which has the same effect yet is simpler). In fact it's a collection of projects, built around the core library, libzmq. I'll visualize these projects as an expanding galaxy:

• At the core, libzmq is the ZeroMQ core library. It's written in C++, with a low-level C API. The code is nasty, mainly because it's highly optimized but also because it's written in C++, a language that lends itself to subtle and deep nastiness. Martin Sustrik wrote the bulk of the original code. Today it has dozens of people who maintain different parts of it.

• Around libzmq, there are about 50 bindings. These are individual projects that create higher-level APIs for ZeroMQ, or at least map the low-level API into other languages. The bindings vary in quality from experimental to utterly awesome. Probably the most impressive binding is PyZMQ, which was one of the first community projects on top of ZeroMQ. If you are a binding author, you should really study PyZMQ and aspire to making your code and community as great.

• A lot of languages have multiple bindings (Erlang, Ruby, C#, at least) written by different people over time, or taking varying approaches. We don't regulate these in any way. There are no "official" bindings. You vote by using one or the other, contributing to it, or ignoring it.

• There are a series of reimplementations of libzmq, starting with JeroMQ, a full Java translation of the library, which is now the basis for NetMQ, a C# stack. These native stacks offer similar or identical APIs, and speak the same protocol (ZMTP) as libzmq.

• On top of the bindings are thousands of projects that use ZeroMQ or build on it. Some of these like Zyre and Malamute are part of the "official" community, most are not.

Libzmq, most of the bindings, and some of the outer projects sit in the ZeroMQ community "organization" on GitHub. This organization is "run" by a group consisting of the most senior binding authors. There's very little to run as it's almost all self-managing and there's zero conflict these days.

iMatix, my firm, plays a specific role in the community. We own the trademarks and enforce them discretely in order to make sure that if you download a package calling itself "ZeroMQ", you can trust what you are getting. People have on rare occasion tried to hijack the name, maybe believing that "free software" means there is no property at stake and no one willing to defend it. One thing you'll understand from this article is how seriously we take the process behind our software (and I mean "us" as a community, not a company). iMatix backs the community by enforcing that process on anything calling itself "ZeroMQ" or "ZeroMQ". We also put money and time into the software and packaging for reasons I'll explain later.

It is not a charity exercise. ZeroMQ is a for-profit project, and a very profitable one. The profits are widely distributed among all those who invest in it. It's really that simple: take the time to become an expert in ZeroMQ, or build something useful on top of ZeroMQ, and you'll find your value as an individual, or team, or company increasing. iMatix enjoys the same benefits as everyone else in the community. It's win-win to everyone except our competitors, who find themselves facing a threat they can't beat and can't really escape. ZeroMQ dominates the future world of massively distributed software.

My firm doesn't just have the community's back—we also built the community. This was deliberate work; in the original ZeroMQ white paper from 2007, there were two projects. One was technical, how to make a better messaging system. The second was how to build a community that could take the software to dominant success. Software dies, but community survives.

How to Make Really Large Architectures

There are, it has been said (at least by people reading this sentence out loud), two ways to make really large-scale software. Option One is to throw massive amounts of money and problems at empires of smart people, and hope that what emerges is not yet another career killer. If you're very lucky and are building on lots of experience, have kept your teams solid, and are not aiming for technical brilliance, and are furthermore incredibly lucky, it works.

But gambling with hundreds of millions of others' money isn't for everyone. For the rest of us who want to build large-scale software, there's Option Two, which is open source, and more specifically, free software. If you're asking how the choice of software license is relevant to the scale of the software you build, that's the right question.

The brilliant and visionary Eben Moglen once said, roughly, that a free software license is the contract on which a community builds. When I heard this, about ten years ago, the idea came to me—Can we deliberately grow free software communities?

Ten years later, the answer is "yes", and there is almost a science to it. I say "almost" because we don't yet have enough evidence of people doing this deliberately with a documented, reproducible process. It is what I'm trying to do with Social Architecture. ZeroMQ came after Wikidot, after the Digital Standards Organization (Digistan) and after the Foundation for a Free Information Infrastructure (aka the FFII, an NGO that fights against software patents). This all came after a lot of less successful community projects like Xitami and Libero. My main takeaway from a long career of projects of every conceivable format is: if you want to build truly large-scale and long-lasting software, aim to build a free software community.

Psychology of Software Architecture

Dirkjan Ochtman pointed me to Wikipedia's definition of Software Architecture as "the set of structures needed to reason about the system, which comprise software elements, relations among them, and properties of both". For me this vapid and circular jargon is a good example of how miserably little we understand what actually makes a successful large scale software architecture.

Architecture is the art and science of making large artificial structures for human use. If there is one thing I've learned and applied successfully in 30 years of making larger and larger software systems, it is this: software is about people. Large structures in themselves are meaningless. It's how they function for human use that matters. And in software, human use starts with the programmers who make the software itself.

The core problems in software architecture are driven by human psychology, not technology. There are many ways our psychology affects our work. I could point to the way teams seem to get stupider as they get larger or when they have to work across larger distances. Does that mean the smaller the team, the more effective? How then does a large global community like ZeroMQ manage to work successfully?

The ZeroMQ community wasn't accidental. It was a deliberate design, my contribution to the early days when the code came out of a cellar in Bratislava. The design was based on my pet science of "Social Architecture", which Wikipedia defines as "the conscious design of an environment that encourages a desired range of social behaviors leading towards some goal or set of goals." I define this as more specifically as "the process, and the product, of planning, designing, and growing an online community."

One of the tenets of Social Architecture is that how we organize is more significant than who we are. The same group, organized differently, can produce wholly different results. We are like peers in a ZeroMQ network, and our communication patterns have a dramatic impact on our performance. Ordinary people, well connected, can far outperform a team of experts using poor patterns. If you're the architect of a larger ZeroMQ application, you're going to have to help others find the right patterns for working together. Do this right, and your project can succeed. Do it wrong, and your project will fail.

The two most important psychological elements are that we're really bad at understanding complexity and that we are so good at working together to divide and conquer large problems. We're highly social apes, and kind of smart, but only in the right kind of crowd.

So here is my short list of the Psychological Elements of Software Architecture:

• Stupidity: our mental bandwidth is limited, so we're all stupid at some point. The architecture has to be simple to understand. This is the number one rule: simplicity beats functionality, every single time. If you can't understand an architecture on a cold gray Monday morning before coffee, it is too complex.

• Selfishness: we act only out of self-interest, so the architecture must create space and opportunity for selfish acts that benefit the whole. Selfishness is often indirect and subtle. For example, I'll spend hours helping someone else understand something because that could be worth days to me later.

• Laziness: we make lots of assumptions, many of which are wrong. We are happiest when we can spend the least effort to get a result or to test an assumption quickly, so the architecture has to make this possible. Specifically, that means it must be simple.

• Jealousy: we're jealous of others, which means we'll overcome our stupidity and laziness to prove others wrong and beat them in competition. The architecture thus has to create space for public competition based on fair rules that anyone can understand.

• Fear: we're unwilling to take risks, especially if it makes us look stupid. Fear of failure is a major reason people conform and follow the group in mass stupidity. The architecture should make silent experimentation easy and cheap, giving people opportunity for success without punishing failure.

• Reciprocity: we'll pay extra in terms of hard work, even money, to punish cheats and enforce fair rules. The architecture should be heavily rule-based, telling people how to work together, but not what to work on.

• Conformity: we're happiest to conform, out of fear and laziness, which means if the patterns are good, clearly explained and documented, and fairly enforced, we'll naturally choose the right path every time.

• Pride: we're intensely aware of our social status, and we'll work hard to avoid looking stupid or incompetent in public. The architecture has to make sure every piece we make has our name on it, so we'll have sleepless nights stressing about what others will say about our work.

• Greed: we're ultimately economic animals (see selfishness), so the architecture has to give us economic incentive to invest in making it happen. Maybe it's polishing our reputation as experts, maybe it's literally making money from some skill or component. It doesn't matter what it is, but there must be economic incentive. Think of architecture as a market place, not an engineering design.

These strategies work on a large scale but also on a small scale, within an organization or team.

The Importance of Contracts

Let me discuss a contentious but important area, which is what license to choose. I'll say "BSD" to cover MIT, X11, BSD, Apache, and similar licenses, and "GPL" to cover GPLv3, LGPLv3, and AGPLv3. The significant difference is the obligation to share back any forked versions, which prevents any entity from capturing the software, and thus keeps it "free".

A software license isn't technically a contract since you don't sign anything. But broadly, calling it a contract is useful since it takes the obligations of each party, and makes them legally enforceable in court, under copyright law.

You might ask, why do we need contracts at all to make open source? Surely it's all about decency, goodwill, people working together for selfless motives. Surely the principle of "less is more" applies here of all places? Don't more rules mean less freedom? Do we really need lawyers to tell us how to work together? It seems cynical and even counter-productive to force a restrictive set of rules on the happy communes of free and open source software.

But the truth about human nature is not that pretty. We're not really angels, nor devils, just self-interested winners descended from a billion-year unbroken line of winners. In business, marriage, and collective works, sooner or later, we either stop caring, or we fight and we argue.

Put this another way: a collective work has two extreme outcomes. Either it's a failure, irrelevant, and worthless, in which case every sane person walks away, without a fight. Or, it's a success, relevant, and valuable, in which case we start jockeying for power, control, and often, money.

What a well-written contract does is to protect those valuable relationships from conflict. A marriage where the terms of divorce are clearly agreed up-front is much less likely to end in divorce. A business deal where both parties agree how to resolve various classic conflictssuch as one party stealing the others' clients or staffis much less likely to end in conflict.

Similarly, a software project that has a well-written contract that defines the terms of breakup clearly is much less likely to end in breakup. The alternative seems to be to immerse the project into a larger organization that can assert pressure on teams to work together (or lose the backing and branding of the organization). This is for example how the Apache Foundation works. In my experience organization building has its own costs, and ends up favoring wealthier participants (who can afford those sometimes huge costs).

In an open source or free software project, breakup usually takes the form of a fork, where the community splits into two or more groups, each with different visions of the future. During the honeymoon period of a project, which can last years, there's no question of a breakup. It is as a project begins to be worth money, or as the main authors start to burn out, that the goodwill and generosity tends to dry up.

So when discussing software licenses, for the code you write or the code you use, a little cynicism helps. Ask yourself, not "which license will attract more contributors?" because the answer to that lies in the mission statement and contribution process. Ask yourself, "if this project had a big fight, and split three ways, which license would save us?" Or, "if the whole team was bought by a hostile firm that wanted to turn this code into a proprietary product, which license would save us?"

Long-term survival means enduring the bad times, as well as enjoying the good ones.

When BSD projects fork, they cannot easily merge again. Indeed, one-way forking of BSD projects is quite systematic: every time BSD code ends up in a commercial project, this is what's happened. When GPL projects fork, however, re-merging is trivial.

The GPL's story is relevant here. Though communities of programmers sharing their code openly were already significant by the 1980's, they tended to use minimal licenses that worked as long as no real money got involved. There was an important language stack called Emacs, originally built in Lisp by Richard Stallman. Another programmer, James Gosling (who later gave us Java), rewrote Emacs in C with the help of many contributors, on the assumption that it would be open. Stallman got that code and used it as the basis for his own C version. Gosling then sold the code to a firm which turned around and blocked anyone distributing a competing product. Stallman found this sale of the common work hugely unethical, and began developing a reusable license that would protect communities from this.

What eventually emerged was the GNU General Public License, which used traditional copyright to force remixability. It was a neat hack that spread to other domains, for instance the Creative Commons for photography and music. In 2007, we saw version 3 of the license, which was a response to belated attacks from Microsoft and others on the concept. It has become a long and complex document but corporate copyright lawyers have become familiar with it and in my experience, few companies mind using GPL software and libraries, so long as the boundaries are clearly defined.

Thus, a good contractand I consider the modern GPL to be the best for softwarelets programmers work together without upfront agreements, organizations, or assumptions of decency and goodwill. It makes it cheaper to collaborate, and turns conflict into healthy competition. GPL doesn't just define what happens with a fork, it actively encourages forks as a tool for experimentation and learning. Whereas a fork can kill a project with a "more liberal" license, GPL projects thrive on forks since successful experiments can, by contract, be remixed back into the mainstream.

Yes, there are many thriving BSD projects and many dead GPL ones. It's always wrong to generalize. A project will thrive or die for many reasons. However, in a competitive sport, one needs every advantage.

The other important part of the BSD vs. GPL story is what I call "leakage", which is the effect of pouring water into a pot with a small but real hole in the bottom.

Eat Me

Here is a story. It happened to the eldest brother-in-law of the cousin of a friend of mine's colleague at work. His name was, and still is, Patrick.

Patrick was a computer scientist with a PhD in advanced network topologies. He spent two years and his savings building a new product, and choose the BSD license because he believed that would get him more adoption. He worked in his attic, at great personal cost, and proudly published his work. People applauded, for it was truly fantastic, and his mailing lists were soon abuzz with activity and patches and happy chatter. Many companies told him how they were saving millions using his work. Some of them even paid him for consultancy and training. He was invited to speak at conferences and started collecting badges with his name on them. He started a small business, hired a friend to work with him, and dreamed of making it big.

Then one day, someone pointed him to a new project, GPL licensed, which had forked his work and was improving on it. He was irritated and upset, and asked how peoplefellow open sourcers, no less!would so shamelessly steal his code. There were long arguments on the list about whether it was even legal to relicense their BSD code as GPL code. Turned out, it was. He tried to ignore the new project, but then he soon realized that new patches coming from that project couldn't even be merged back into his work!

Worse, the GPL project got popular and some of his core contributors made first small, and then larger patches to it. Again, he couldn't use those changes, and he felt abandoned. Patrick went into a depression, his girlfriend left him for an international currency dealer called, weirdly, Patrice, and he stopped all work on the project. He felt betrayed, and utterly miserable. He fired his friend, who took it rather badly and told everyone that Patrick was a closet banjo player. Finally, Patrick took a job as a project manager for a cloud company, and by the age of forty, he had stopped programming even for fun.

Poor Patrick. I almost felt sorry for him. Then I asked him, "Why didn't you choose the GPL?" "Because it's a restrictive viral license", he replied. I told him, "You may have a PhD, and you may be the eldest brother-in-law of the cousin of a friend of my colleague, but you are an idiot and Monique was smart to leave you. You published your work inviting people to please steal your code as long as they kept this 'please steal my code' statement in the resulting work", and when people did exactly that, you got upset. Worse, you were a hypocrite because when they did it in secret, you were happy, but when they did it openly, you felt betrayed."

Seeing your hard work captured by a smarter team and then used against you is enormously painful, so why even make that possible? Every proprietary project that uses BSD code is capturing it. A public GPL fork is perhaps more humiliating, but it's fully self-inflicted.

BSD is like food. It literally (and I mean that metaphorically) whispers "eat me" in the little voice one imagines a cube of cheese might use when it's sitting next to an empty bottle of the best beer in the world, which is of course Orval, brewed by an ancient and almost extinct order of silent Belgian monks called Les Gars Labas Qui Fabrique l'Orval. The BSD license, like its near clone MIT/X11, was designed specifically by a university (Berkeley) with no profit motive to leak work and effort. It is a way to push subsidized technology at below its cost price, a dumping of under-priced code in the hope that it will break the market for others. BSD is an excellent strategic tool, but only if you're a large well-funded institution that can afford to use Option One. The Apache license is BSD in a suit.

For us small businesses who aim our investments like precious bullets, leaking work and effort is unacceptable. Breaking the market is great, but we cannot afford to subsidize our competitors. The BSD networking stack ended up putting Windows on the Internet. We cannot afford battles with those we should naturally be allies with. We cannot afford to make fundamental business errors because in the end, that means we have to fire people.

It comes down to behavioral economics and game theory. The license we choose modifies the economics of those who use our work. In the software industry, there are friends, foes, and food. BSD makes most people see us as lunch. Closed source makes most people see us as enemies (do you like paying people for software?) GPL, however, makes most people, with the exception of the Patricks of the world, our allies. Any fork of ZeroMQ is license compatible with ZeroMQ, to the point where we encourage forks as a valuable tool for experimentation. Yes, it can be weird to see someone try to run off with the ball but here's the secret, I can get it back any time I want.

The Process

If you've accepted my thesis up to now, great! Now, I'll explain the rough process by which we actually build an open source community. This was how we built or grew or gently steered the ZeroMQ community into existence.

Your goal as leader of a community is to motivate people to get out there and explore; to ensure they can do so safely and without disturbing others; to reward them when they make successful discoveries; and to ensure they share their knowledge with everyone else (and not because we ask them, not because they feel generous, but because it's The Law).

It is an iterative process. You make a small product, at your own cost, but in public view. You then build a small community around that product. If you have a small but real hit, the community then helps design and build the next version, and grows larger. And then that community builds the next version, and so on. It's evident that you remain part of the community, maybe even a majority contributor, but the more control you try to assert over the material results, the less people will want to participate. Plan your own retirement well before someone decides you are their next problem.

Crazy, Beautiful, and Easy

You need a goal that's crazy and simple enough to get people out of bed in the morning. Your community has to attract the very best people and that demands something special. With ZeroMQ, we said we were going to make "the Fastest. Messaging. Ever.", which qualifies as a good motivator. If we'd said, we're going to make "a smart transport layer that'll connect your moving pieces cheaply and flexibly across your enterprise", we'd have failed.

Then your work must be beautiful, immediately useful, and attractive. Your contributors are users who want to explore just a little beyond where they are now. Make it simple, elegant, and brutally clean. The experience when people run or use your work should be an emotional one. They should feel something, and if you accurately solved even just one big problem that until then they didn't quite realize they faced, you'll have a small part of their soul.

It must be easy to understand, use, and join. Too many projects have barriers to access: put yourself in the other person's mind and see all the reasons they come to your site, thinking "Um, interesting project, but…" and then leave. You want them to stay and try it, just once. Use GitHub and put the issue tracker right there.

If you do these things well, your community will be smart but more importantly, it will be intellectually and geographically diverse. This is really important. A group of like-minded experts cannot explore the problem landscape well. They tend to make big mistakes. Diversity beats education any time.

Stranger, Meet Stranger

How much up-front agreement do two people need to work together on something? In most organizations, a lot. But you can bring this cost down to near-zero, and then people can collaborate without having ever met, done a phone conference, meeting, or business trip to discuss Roles and Responsibilities over way too many bottles of cheap Korean rice wine.

You need well-written rules that are designed by cynical people like me to force strangers into mutually beneficial collaboration instead of conflict. The GPL is a good start. GitHub and its fork/merge strategy is a good follow-up. And then you want something like our C4.1 rulebook to control how work actually happens.

C4.1 (which I now use for every new open source project) has detailed and tested answers to a lot of common mistakes people make, such as the sin of working offline in a corner with others "because it's faster". Transparency is essential to get trust, which is essential to get scale. By forcing every single change through a single transparent process, you build real trust in the results.

Another cardinal sin that many open source developers make is to place themselves above others. "I founded this project thus my intellect is superior to that of others". It's not just immodest and rude, and usually inaccurate, it's also poor business. The rules must apply equally to everyone, without distinction. You are part of the community. Your job, as founder of a project, is not to impose your vision of the product over others, but to make sure the rules are good, honest, and enforced.

Infinite Property

One of the saddest myths of the knowledge business is that ideas are a sensible form of property. It's medieval nonsense that should have been junked along with slavery, but sadly it's still making too many powerful people too much money.

Ideas are cheap. What does work sensibly as property is the hard work we do in building a market. "You eat what you kill" is the right model for encouraging people to work hard. Whether it's moral authority over a project, money from consulting, or the sale of a trademark to some large, rich firm: if you make it, you own it. But what you really own is "footfall", participants in your project, which ultimately defines your power.

To do this requires infinite free space. Thankfully, GitHub solved this problem for us, for which I will die a grateful person (there are many reasons to be grateful in life, which I won't list here because we only have a hundred or so pages left, but this is one of them).

You cannot scale a single project with many owners like you can scale a collection of many small projects, each with fewer owners. When we embrace forks, a person can become an "owner" with a single click. Now they just have to convince others to join by demonstrating their unique value.

So in ZeroMQ, we aimed to make it easy to write bindings on top of the core library, and we stopped trying to make those bindings ourselves. This created space for others to make those, become their owners, and get that credit.

Care and Feeding

I wish a community could be 100% self-steering, and perhaps one day this will work, but today it's not the case. We're very close with ZeroMQ, but from my experience a community needs four types of care and feeding:

• First, simply because most people are too nice, we need some kind of symbolic leadership or owners who provide ultimate authority in case of conflict. Usually it's the founders of the community. I've seen it work with self-elected groups of "elders", but old men like to talk a lot. I've seen communities split over the question "who is in charge?", and setting up legal entities with boards and such seems to make arguments over control worse, not better. Maybe because there seems to be more to fight over. One of the real benefits of free software is that it's always remixable, so instead of fighting over a pie, one simply forks the pie.

• Second, communities need living rules, and thus they need a lawyer able to formulate and write these down. Rules are critical; when done right, they remove friction. When done wrong, or neglected, we see real friction and argument that can drive away the nice majority, leaving the argumentative core in charge of the burning house. One thing I've tried to do with the ZeroMQ and previous communities is create reusable rules, which perhaps means we don't need lawyers as much.

• Thirdly, communities need some kind of financial backing. This is the jagged rock that breaks most ships. If you starve a community, it becomes more creative but the core contributors burn out. If you pour too much money into it, you attract the professionals, who never say "no", and the community loses its diversity and creativity. If you create a fund for people to share, they will fight (bitterly) over it. With ZeroMQ, we (iMatix) spend our time and money on marketing and packaging (like this book), and the basic care, like bug fixes, releases, and websites.

• Lastly, sales and commercial mediation are important. There is a natural market between expert contributors and customers, but both are somewhat incompetent at talking to each other. Customers assume that support is free or very cheap because the software is free. Contributors are shy at asking a fair rate for their work. It makes for a difficult market. A growing part of my work and my firm's profits is simply connecting ZeroMQ users who want help with experts from the community able to provide it, and ensuring both sides are happy with the results.

I've seen communities of brilliant people with noble goals dying because the founders got some or all of these four things wrong. The core problem is that you can't expect consistently great leadership from any one company, person, or group. What works today often won't work tomorrow, yet structures become more solid, not more flexible, over time.

The best answer I can find is a mix of two things. One, the GPL and its guarantee of remixability. No matter how bad the authority, no matter how much they try to privatize and capture the community's work, if it's GPL licensed, that work can walk away and find a better authority. Before you say, "all open source offers this," think it through. I can kill a BSD-licensed project by hiring the core contributors and not releasing any new patches. But even with a billion of dollars, I cannot kill a GPL-licensed project. Two, the philosophical anarchist model of authority, which is that we choose it, it does not own us.

The ZeroMQ Process: C4.1

When we say ZeroMQ we sometimes mean libzmq, the core library. In early 2012, we synthesized the libzmq process into a formal and reusable protocol for collaboration that we called the Collective Code Construction Contract, or C4.1. You can see this as a layer above the license (e.g. MPLv2). These are our rules, and I'll explain the reasoning behind each one.

C4.1 is an evolution of the GitHub Fork + Pull Model. You may get the feeling I'm a fan of git and GitHub. This would be accurate: these two tools have made such a positive impact on our work over the last years, especially when it comes to building community.

Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

By starting with the RFC 2119 language, the C4.1 text makes very clear its intention to act as a protocol rather than a randomly written set of recommendations. A protocol is a contract between parties that defines the rights and obligations of each party. These can be peers in a network or they can be strangers working in the same project.

I think C4.1 is the first time anyone has attempted to codify a community's rulebook as a formal and reusable protocol spec. Previously, our rules were spread out over several wiki pages, and were quite specific to libzmq in many ways. But experience teaches us that the more formal, accurate, and reusable the rules, the easier it is for strangers to collaborate up-front. And less friction means a more scalable community. At the time of C4, we also had some disagreement in the libzmq project over precisely what process we were using. Not everyone felt bound by the same rules. Let's just say some people felt they had a special status, which created friction with the rest of the community. So codification made things clear.

It's easy to use C4: just host your project on GitHub, get one other person to join, and open the floor to pull requests. In your README, put a link to C4.1 and that's it. We've done this in quite a few projects and it does seem to work. I've been pleasantly surprised a few times just applying these rules to my own work, like CZMQ. None of us are so amazing that we can work without others.

Goals

C4.1 is meant to provide a reusable optimal collaboration model for open source software projects.

The short term reason for writing C4.1 was to end arguments over the libzmq contribution process. The dissenters went off elsewhere. The ZeroMQ community blossomed smoothly and easily, as I'd predicted. Most people were surprised, but gratified. There's been no real criticisms of C4.1 except its branching policy, which I'll come to later as it deserves its own discussion.

There's a reason I'm reviewing history here: as founder of a community, you are asking people to invest in your property, trademark, and branding. In return, and this is what we do with ZeroMQ, you can use that branding to set a bar for quality. When you download a product labeled "ZeroMQ", you know that it's been produced to certain standards. It's a basic rule of quality: write down your process; otherwise you cannot improve it. Our processes aren't perfect, nor can they ever be. But any flaw in them can be fixed, and tested.

Making C4.1 reusable is therefore really important. To learn more about the best possible process, we need to get results from the widest range of projects.

It has these specific goals:
To maximize the scale of the community around a project, by reducing the friction for new Contributors and creating a scaled participation model with strong positive feedbacks;

The number one goal is size and health of the community—not technical quality, not profits, not performance, not market share. The goal is simply the number of people who contribute to the project. The science here is simple: the larger the community, the more accurate the results.

To relieve dependencies on key individuals by separating different skill sets so that there is a larger pool of competence in any required domain;

Perhaps the worst problem we faced in libzmq was dependence on people who could understand the code, manage GitHub branches, and make clean releases—all at the same time. It's like looking for athletes who can run marathons and sprint, swim, and also lift weights. We humans are really good at specialization. Asking us to be really good at two contradictory things reduces the number of candidates sharply, which is a Bad Thing for any project. We had this problem severely in libzmq in 2009 or so, and fixed it by splitting the role of maintainer into two: one person makes patches and another makes releases.

To allow the project to develop faster and more accurately, by increasing the diversity of the decision making process;

This is theory—not fully proven, but not falsified. The diversity of the community and the number of people who can weigh in on discussions, without fear of being criticized or dismissed, the faster and more accurately the software develops. Speed is quite subjective here. Going very fast in the wrong direction is not just useless, it's actively damaging (and we suffered a lot of that in libzmq before we switched to C4).

To support the natural life cycle of project versions from experimental through to stable, by allowing safe experimentation, rapid failure, and isolation of stable code;

It's quite an interesting effect of the process: the git master is almost always perfectly stable. This has to do with the size of changes and their latency, i.e., the time between someone writing the code and someone actually using it fully. However, the healthy design learning process tends to cycle through drafts until becoming stable, and inviolable.

To reduce the internal complexity of project repositories, thus making it easier for Contributors to participate and reducing the scope for error;

Curious observation: people who thrive in complex situations like to create complexity because it keeps their value high. It's the Cobra Effect (Google it). Git made branches easy and left us with the all too common syndrome of "git is easy once you understand that a git branch is just a folded five-dimensional lepton space that has a detached history with no intervening cache". Developers should not be made to feel stupid by their tools. I've seen too many top-class developers confused by repository structures to accept conventional wisdom on git branches. We'll come back to dispose of git branches shortly, dear reader.

To enforce collective ownership of the project, which increases economic incentive to Contributors and reduces the risk of hijack by hostile entities.

Ultimately, we're economic creatures, and the sense that "we own this, and our work can never be used against us" makes it much easier for people to invest in an open source project like ZeroMQ. And it can't be just a feeling, it has to be real. There are a number of aspects to making collective ownership work, we'll see these one-by-one as we go through C4.

Preliminaries

The project SHALL use the git distributed revision control system.

Git has its faults. Its command-line API is horribly inconsistent, and it has a complex, messy internal model that it shoves in your face at the slightest provocation. But despite doing its best to make its users feel stupid, git does its job really, really well. More pragmatically, I've found that if you stay away from certain areas (branches!), people learn git rapidly and don't make many mistakes. That works for me.

The project SHALL be hosted on github.com or equivalent, herein called the "Platform".

I'm sure one day some large firm will buy GitHub and break it, and another platform will rise in its place. Until then, Github serves up a near-perfect set of minimal, fast, simple tools. I've thrown hundreds of people at it, and they all stick like flies stuck in a dish of honey.

The project SHALL use the Platform issue tracker.

We made the mistake in libzmq of switching to Jira because we hadn't learned yet how to properly use the GitHub issue tracker. Jira is a great example of how to turn something useful into a complex mess because the business depends on selling more "features". But even without criticizing Jira, keeping the issue tracker on the same platform means one less UI to learn, one less login, and smooth integration between issues and patches.

The project SHOULD have clearly documented guidelines for code style.

This is a protocol plug-in: insert code style guidelines here. If you don't document the code style you use, you have no basis except prejudice to reject patches.

A "Contributor" is a person who wishes to provide a patch, being a set of commits that solve some clearly identified problem.
A "Maintainer" is a person who merge patches to the project. Maintainers are not developers; their job is to enforce process.

Now we move on to definitions of the parties, and the splitting of roles that saved us from the sin of structural dependency on rare individuals. This worked well in libzmq, but as you will see it depends on the rest of the process. C4.1 isn't a buffet; you will need the whole process (or something very like it), or it won't hold together.

Contributors SHALL NOT have commit access to the repository unless they are also Maintainers.
Maintainers SHALL have commit access to the repository.

What we wanted to avoid was people pushing their changes directly to master. This was the biggest source of trouble in libzmq historically: large masses of raw code that took months or years to fully stabilize. We eventually followed other ZeroMQ projects like PyZMQ in using pull requests. We went further, and stipulated that all changes had to follow the same path. No exceptions for "special people".

Everyone, without distinction or discrimination, SHALL have an equal right to become a Contributor under the terms of this contract.

We had to state this explicitly. It used to be that the libzmq maintainers would reject patches simply because they didn't like them. Now, that may sound reasonable to the author of a library (though libzmq was not written by any one person), but let's remember our goal of creating a work that is owned by as many people as possible. Saying "I don't like your patch so I'm going to reject it" is equivalent to saying, "I claim to own this and I think I'm better than you, and I don't trust you". Those are toxic messages to give to others who are thinking of becoming your co-investors.

I think this fight between individual expertise and collective intelligence plays out in other areas. It defined Wikipedia, and still does, a decade after that work surpassed anything built by small groups of experts. For me, we make software by slowly synthesizing the most accurate knowledge, much as we make Wikipedia articles.

Licensing and Ownership

The project SHALL use a share-alike license, such as the GPLv3 or a variant thereof (LGPL, AGPL), or the MPLv2.

I've already explained how full remixability creates better scale and why the GPL and its variants seems the optimal contract for remixable software. If you're a large business aiming to dump code on the market, you won't want C4, but then you won't really care about community either.

All contributions to the project source code ("patches") SHALL use the same license as the project.

This removes the need for any specific license or contribution agreement for patches. You fork the GPL code, you publish your remixed version on GitHub, and you or anyone else can then submit that as a patch to the original code. BSD doesn't allow this. Any work that contains BSD code may also contain unlicensed proprietary code so you need explicit action from the author of the code before you can remix it.

All patches are owned by their authors. There SHALL NOT be any copyright assignment process.

Here we come to the key reason people trust their investments in ZeroMQ: it's logistically impossible to buy the copyrights to create a closed source competitor to ZeroMQ. iMatix can't do this either. And the more people that send patches, the harder it becomes. ZeroMQ isn't just free and open today—this specific rule means it will remain so forever. Note that it's not the case in all GPL projects, many of which still ask for copyright transfer back to the maintainers.

The project SHALL be owned collectively by all its Contributors.

This is perhaps redundant, but worth saying: if everyone owns their patches, then the resulting whole is also owned by every contributor. There's no legal concept of owning lines of code: the "work" is at least a source file.

Each Contributor SHALL be responsible for identifying themselves in the project Contributor list.

In other words, the maintainers are not karma accountants. Anyone who wants credit has to claim it themselves.

Patch Requirements

In this section, we define the obligations of the contributor: specifically, what constitutes a "valid" patch, so that maintainers have rules they can use to accept or reject patches.

Maintainers and Contributors MUST have a Platform account and SHOULD use their real names or a well-known alias.

In the worst case scenario, where someone has submitted toxic code (patented, or owned by someone else), we need to be able to trace who and when, so we can remove the code. Asking for real names or a well-known alias is a theoretical strategy for reducing the risk of bogus patches. We don't know if this actually works because we haven't had the problem yet.

A patch SHOULD be a minimal and accurate answer to exactly one identified and agreed problem.

This implements the Simplicity Oriented Design process that I'll come to later in this chapter. One clear problem, one minimal solution, apply, test, repeat.

A patch MUST adhere to the code style guidelines of the project if these are defined.

This is just sanity. I've spent time cleaning up other peoples' patches because they insisted on putting the else beside the if instead of just below as Nature intended. Consistent code is healthier.

A patch MUST adhere to the "Evolution of Public Contracts" guidelines defined below.

Ah, the pain, the pain. I'm not speaking of the time at age eight when I stepped on a plank with a 4-inch nail protruding from it. That was relatively OK. I'm speaking of 2010-2011 when we had multiple parallel releases of ZeroMQ, each with different incompatible APIs or wire protocols. It was an exercise in bad rules, pointlessly enforced, that still hurts us today. The rule was, "If you change the API or protocol, you SHALL create a new major version". Give me the nail through the foot; that hurt less.

One of the big changes we made with C4.1 was simply to ban, outright, this kind of sanctioned sabotage. Amazingly, it's not even hard. We just don't allow the breaking of existing public contracts, period, unless everyone agrees, in which case no period. As Linus Torvalds famously put it on 23 December 2012, "WE DO NOT BREAK USERSPACE!"

A patch SHALL NOT include nontrivial code from other projects unless the Contributor is the original author of that code.

This rule has two effects. The first is that it forces people to make minimal solutions because they cannot simply import swathes of existing code. In the cases where I've seen this happen to projects, it's always bad unless the imported code is very cleanly separated. The second is that it avoids license arguments. You write the patch, you are allowed to publish it as LGPL, and we can merge it back in. But you find a 200-line code fragment on the web, and try to paste that, we'll refuse.

A patch MUST compile cleanly and pass project self-tests on at least the principle target platform.

For cross-platform projects, it is fair to ask that the patch works on the development box used by the contributor.

A patch commit message SHOULD consist of a single short (less than 50 character) line summarizing the change, optionally followed by a blank line and then a more thorough description.

This is a good format for commit messages that fits into email (the first line becomes the subject, and the rest becomes the email body).

A "Correct Patch" is one that satisfies the above requirements.

Just in case it wasn't clear, we're back to legalese and definitions.

Development Process

In this section, we aim to describe the actual development process, step-by-step.

Change on the project SHALL be governed by the pattern of accurately identifying problems and applying minimal, accurate solutions to these problems.

This is a unapologetic ramming through of thirty years' software design experience. It's a profoundly simple approach to design: make minimal, accurate solutions to real problems, nothing more or less. In ZeroMQ, we don't have feature requests. Treating new features the same as bugs confuses some newcomers. But this process works, and not just in open source. Enunciating the problem we're trying to solve, with every single change, is key to deciding whether the change is worth making or not.

To request changes, a user SHOULD log an issue on the project Platform issue tracker.

This is how users talk to contributors. Track your problems, so others can (maybe) try to solve them for you.

The user or Contributor SHOULD write the issue by describing the problem they face or observe.

"Problem: we need feature X. Solution: make it" is not a good issue. "Problem: user cannot do common tasks A or B except by using a complex workaround. Solution: make feature X" is a decent explanation. Because everyone I've ever worked with has needed to learn this, it seems worth restating: document the real problem first, solution second.

The user or Contributor SHOULD seek consensus on the accuracy of their observation, and the value of solving the problem.

And because many apparent problems are illusionary, by stating the problem explicitly we give others a chance to correct our logic. "You're only using A and B a lot because function C is unreliable. Solution: make function C work properly."

Users SHALL NOT log feature requests, ideas, suggestions, or any solutions to problems that are not explicitly documented and provable.

There are several reasons for not logging ideas, suggestions, or feature requests. In our experience, these just accumulate in the issue tracker until someone deletes them. But more profoundly, when we treat all change as problem solutions, we can prioritize trivially. Either the problem is real and someone wants to solve it now, or it's not on the table. Thus, wish lists are off the table.

Thus, the release history of the project SHALL be a list of meaningful issues logged and solved.

I'd love the GitHub issue tracker to simply list all the issues we solved in each release. Today we still have to write that by hand. If one puts the issue number in each commit, and if one uses the GitHub issue tracker, which we sadly don't yet do for ZeroMQ, this release history is easier to produce mechanically.

To work on an issue, a Contributor SHALL fork the project repository and then work on their forked repository.

Here we explain the GitHub fork + pull request model so that newcomers only have to learn one process (C4) in order to contribute.

To submit a patch, a Contributor SHALL create a Platform pull request back to the project.

GitHub has made this so simple that we don't need to learn git commands to do it, for which I'm deeply grateful. Sometimes, I'll tell people who I don't particularly like that command-line git is awesome and all they need to do is learn git's internal model in detail before trying to use it on real work. When I see them several months later they look… changed.

A Contributor SHALL NOT commit changes directly to the project.

Anyone who submits a patch is a contributor, and all contributors follow the same rules. No special privileges to the original authors, because otherwise we're not building a community, only boosting our egos.

To discuss a patch, people MAY comment on the Platform pull request, on the commit, or elsewhere.

Randomly distributed discussions may be confusing if you're walking up for the first time, but GitHub solves this for all current participants by sending emails to those who need to follow what's going on. We had the same experience and the same solution in Wikidot, and it works. There's no evidence that discussing in different places has any negative effect.

To accept or reject a patch, a Maintainer SHALL use the Platform interface.

Working via the GitHub web user interface means pull requests are logged as issues, with workflow and discussion. I'm sure there are more complex ways to work. Complexity is easy; it's simplicity that's incredibly hard.

Maintainers SHALL NOT accept their own patches.

There was a rule we defined in the FFII years ago to stop people burning out: no less than two people on any project. One-person projects tend to end in tears, or at least bitter silence. We have quite a lot of data on burnout, why it happens, and how to prevent it (even cure it). I'll explore this later in the chapter, because if you work with or on open source you need to be aware of the risks. The "no merging your own patch" rule has two goals. First, if you want your project to be C4-certified, you have to get at least one other person to help. If no one wants to help you, perhaps you need to rethink your project. Second, having a control for every patch makes it much more satisfying, keeps us more focused, and stops us breaking the rules because we're in a hurry, or just feeling lazy.

Maintainers SHALL NOT make value judgments on correct patches.

We already said this but it's worth repeating: the role of Maintainer is not to judge a patch's substance, only its technical quality. The substantive worth of a patch only emerges over time: people use it, and like it, or they do not. And if no one is using a patch, eventually it'll annoy someone else who will remove it, and no one will complain.

Maintainers SHALL merge correct patches rapidly.

There is a criteria I call change latency, which is the round-trip time from identifying a problem to testing a solution. The faster the better. If maintainers cannot respond to pull requests as rapidly as people expect, they're not doing their job (or they need more hands).

Maintainers MAY merge incorrect patches from other Contributors with the goals of (a) ending fruitless discussions, (b) capturing toxic patches in the historical record, (c) engaging with the Contributor on improving their patch quality.

It turns out that accepting imperfect patches rapidly, which I call "optimistic merging", works better all-round than insisting that contributors deliver perfect work.

The Contributor MAY tag a user's issue as "Ready" after making a pull request for the issue.

By default, GitHub offers the usual variety of issue tags, but with C4.1 we don't use them. Instead, we need just two labels, "Urgent" and "Ready". A contributor who wants another user to test an issue can then label it as "Ready".

The user who created an issue SHOULD close the issue after checking the patch is successful.

When one person opens an issue, and another works on it, it's best to allow the original person to close the issue. That acts as a double-check that the issue was properly resolved.

Any Contributor who has value judgments on a patch SHOULD express these via their own patches.

In essence, the goal here is to allow users to try patches rather than to spend time arguing pros and cons. As easy as it is to make a patch, it's as easy to revert it with another patch. You might think this would lead to "patch wars", but that hasn't happened. We've had a handful of cases in libzmq where patches by one contributor were killed by another person who felt the experimentation wasn't going in the right direction. It is easier than seeking up-front consensus.

Maintainers MAY commit changes to non-source documentation directly to the project.

This exit allows maintainers who are making release notes to push those without having to create an issue which would then affect the release notes, leading to stress on the space time fabric and possibly involuntary rerouting backwards in the fourth dimension to before the invention of cold beer. Shudder. It is simpler to agree that release notes aren't technically software.

Branches and Releases

When C4.1 is working, we get two massive simplifications of our delivery process. One, we don't need or use branches. Two, we deliver from master.

This is the process we explain in this section.

The project SHALL have one branch ("master") that always holds the latest in-progress version and SHOULD always build.

This is redundant because every patch always builds but it's worth restating. If the master doesn't build (and pass its tests), someone needs waking up.

The project SHALL NOT use topic branches for any reason. Personal forks MAY use topic branches.

I'll come to branches soon. In short (or "tl;dr", as they say on the webs), branches make the repository too complex and fragile, and require up-front agreement, all of which are expensive and avoidable.

To make a stable release a Maintainer shall tag the repository. Stable releases SHALL always be released from the repository master.

Evolution of Public Contracts

By "public contracts", I mean APIs and protocols. Up until the end of 2011, libzmq's naturally happy state was marred by broken promises and broken contracts. We stopped making promises (aka "road maps") for libzmq completely, and our dominant theory of change is now that it emerges carefully and accurately over time. At a 2012 Chicago meetup, Garrett Smith and Chuck Remes called this the "drunken stumble to greatness", which is how I think of it now.

We stopped breaking public contracts simply by banning the practice. Before then it had been "OK" (as in we did it and everyone complained bitterly, and we ignored them) to break the API or protocol so long as we changed the major version number. Sounds fine, until you get ZeroMQ v2.0, v3.0, and v4.0 all in development at the same time, and not speaking to each other.

All Public Contracts (APIs or protocols) SHALL be documented.

You'd think this was a given for professional software engineers but no, it's not. So, it's a rule. You want C4.1 certification for your project, you make sure your public contracts are documented. No "It's specified in the code" excuses. Code is not a contract. (Yes, I intend at some point to create a C4.1 certification process to act as a quality indicator for open source projects.)

All Public Contracts SHOULD have space for extensibility and experimentation.

Now, the real thing is that public contracts do change. It's not about not changing them. It's about changing them safely. This means educating (especially protocol) designers to create that space up-front.

A patch that modifies a stable Public Contract SHOULD not break existing applications unless there is overriding consensus on the value of doing this.

Sometimes the patch is fixing a bad API that no one is using. It's a freedom we need, but it should be based on consensus, not one person's dogma. However, making random changes "just because" is not good. In ZeroMQ v3.x, did we benefit from renaming ZMQ_NOBLOCK to ZMQ_DONTWAIT? Sure, it's closer to the POSIX socket recv() call, but is that worth breaking thousands of applications? No one ever reported it as an issue. To misquote Stallman: "your freedom to create an ideal world stops one inch from my application."

A patch that introduces new features SHOULD do so using new names (a new contract).

We had the experience in ZeroMQ once or twice of new features using old names (or worse, using names that were still in use elsewhere). ZeroMQ v3.0 had a newly introduced "ROUTER" socket that was totally different from the existing ROUTER socket in 2.x. Dear lord, you should be face-palming, why? The reason: apparently, even smart people sometimes need regulation to stop them doing silly things.

New contracts SHOULD be marked as "draft" until they are stable and used by real users.

Old contracts SHOULD be deprecated in a systematic fashion by marking new contracts as "draft" until they are stable, then marking the old contracts as "deprecated".

This life cycle notation has the great benefit of actually telling users what is going on with a consistent direction. "Draft" means "we have introduced this and intend to make it stable if it works". It does not mean, "we have introduced this and will remove it at any time if we feel like it". One assumes that code that survives more than one patch cycle is meant to be there. "Deprecated" means "we have replaced this and intend to remove it".

Old contracts SHOULD be deprecated in a systematic fashion by marking them as "deprecated" and replacing them with new contracts as needed.

When sufficient time has passed, old deprecated contracts SHOULD be removed.

In theory this gives applications time to move onto stable new contracts without risk. You can upgrade first, make sure things work, and then, over time, fix things up to remove dependencies on deprecated and legacy APIs and protocols.

Old names SHALL NOT be reused by new features.

Ah, yes, the joy when ZeroMQ v3.x renamed the top-used API functions (zmq_send[3] and zmq_recv[3]) and then recycled the old names for new methods that were utterly incompatible (and which I suspect few people actually use). You should be slapping yourself in confusion again, but really, this is what happened and I was as guilty as anyone. After all, we did change the version number! The only benefit of that experience was to get this rule.

Project Administration

The project founders SHALL act as Administrators to manage the set of project Maintainers.

Someone needs to administer the project, and it makes sense that the original founders start this ball rolling.

The Administrators SHALL ensure their own succession over time by promoting the most effective Maintainers.

At the same time, as founder of a project you really want to get out of the way before you become over-attached to it. Promoting the most active and consistent maintainers is good for everyone.

A new Contributor who makes correct patches, who clearly understands the project goals, and the process SHOULD be invited to become a Maintainer.

Promote your contributors rapidly, when they show they get it. Anything else is counter-productive.

Administrators SHOULD remove Maintainers who are inactive for an extended period of time, or who repeatedly fail to apply this process accurately.

This was Ian Barber's suggestion: we need a way to crop inactive maintainers. Originally maintainers were self-elected but that makes it hard to drop troublemakers (who are rare, but not unknown).

Administrators SHOULD block or ban "bad actors" who cause stress and pain to others in the project. This should be done after public discussion, with a chance for all parties to speak. A bad actor is someone who repeatedly ignores the rules and culture of the project, who is needlessly argumentative or hostile, or who is offensive, and who is unable to self-correct their behavior when asked to do so by others.

Now and then, your projects will attract people of the wrong character. You will get better at seeing these people, over time. C4.1 helps in two ways. One, by setting out strong rules, it discourages the chaos-seekers and bullies, who cannot tolerate others' rules. Two, it gives you the Administrator the power to ban them. I like to give such people time, to show themselves, and get their patches on the public record (a reason to merge bad patches, which of course you can remove after a suitable pause).

Designing for Innovation

Let's look at innovation, which Wikipedia defines as, "the development of new values through solutions that meet new requirements, inarticulate needs, or old customer and market needs in value adding new ways." This really just means solving problems more cheaply. It sounds straight-forward, but the history of collapsed tech giants proves that it's not. I'll try to explain how teams so often get it wrong, and suggest a way for doing innovation right.

The Tale of Two Bridges

Two old engineers were talking of their lives and boasting of their greatest projects. One of the engineers explained how he had designed one of the greatest bridges ever made.

"We built it across a river gorge," he told his friend. "It was wide and deep. We spent two years studying the land, and choosing designs and materials. We hired the best engineers and designed the bridge, which took another five years. We contracted the largest engineering firms to build the structures, the towers, the tollbooths, and the roads that would connect the bridge to the main highways. Dozens died during the construction. Under the road level we had trains, and a special path for cyclists. That bridge represented years of my life."

The second man reflected for a while, then spoke. "One evening me and a friend got drunk on vodka, and we threw a rope across a gorge," he said. "Just a rope, tied to two trees. There were two villages, one at each side. At first, people pulled packages across that rope with a pulley and string. Then someone threw a second rope, and built a foot walk. It was dangerous, but the kids loved it. A group of men then rebuilt that, made it solid, and women started to cross, everyday, with their produce. A market grew up on one side of the bridge, and slowly that became a large town, because there was a lot of space for houses. The rope bridge got replaced with a wooden bridge, to allow horses and carts to cross. Then the town built a real stone bridge, with metal beams. Later, they replaced the stone part with steel, and today there's a suspension bridge standing in that same spot."

The first engineer was silent. "Funny thing," he said, "my bridge was demolished about ten years after we built it. Turns out it was built in the wrong place and no one wanted to use it. Some guys had thrown a rope across the gorge, a few miles further downstream, and that's where everyone went."

How ZeroMQ Lost Its Road Map

Presenting ZeroMQ at the Mix-IT conference in Lyon in early 2012, I was asked several times for the "road map". My answer was: there is no road map any longer. We had road maps, and we deleted them. Instead of a few experts trying to lay out the next steps, we were allowing this to happen organically. The audience didn't really like my answer. So un-French.

However, the history of ZeroMQ makes it quite clear why road maps were problematic. In the beginning, we had a small team making the library, with few contributors, and no documented road map. As ZeroMQ grew more popular and we switched to more contributors, users asked for road maps. So we collected our plans together and tried to organize them into releases. Here, we wrote, is what will come in the next release.

As we rolled out releases, we hit the problem that it's very easy to promise stuff, and rather harder to make it as planned. For one thing, much of the work was voluntary, and it's not clear how you force volunteers to commit to a road map. But also, priorities can shift dramatically over time. So we were making promises we could not keep, and the real deliveries didn't match the road maps.

The second problem was that by defining the road map, we in effect claimed territory, making it harder for others to participate. People do prefer to contribute to changes they believe were their idea. Writing down a list of things to do turns contribution into a chore rather than an opportunity.

Finally, we saw changes in ZeroMQ that were quite traumatic, and the road maps didn't help with this, despite a lot of discussion and effort to "do it right". Examples of this were incompatible changes in APIs and protocols. It was quite clear that we needed a different approach for defining the change process.

Software engineers don't like the notion that powerful, effective solutions can come into existence without an intelligent designer actively thinking things through. And yet no one in that room in Lyon would have questioned evolution. A strange irony, and one I wanted to explore further as it underpins the direction the ZeroMQ community has taken since the start of 2012.

In the dominant theory of innovation, brilliant individuals reflect on large problem sets and then carefully and precisely create a solution. Sometimes they will have "eureka" moments where they "get" brilliantly simple answers to whole large problem sets. The inventor, and the process of invention are rare, precious, and can command a monopoly. History is full of such heroic individuals. We owe them our modern world.

Looking more closely, however, and you will see that the facts don't match. History doesn't show lone inventors. It shows lucky people who steal or claim ownership of ideas that are being worked on by many. It shows brilliant people striking lucky once, and then spending decades on fruitless and pointless quests. The best known large-scale inventors like Thomas Edison were in fact just very good at systematic broad research done by large teams. It's like claiming that Steve Jobs invented every device made by Apple. It is a nice myth, good for marketing, but utterly useless as practical science.

Recent history, much better documented and less easy to manipulate, shows this well. The Internet is surely one of the most innovative and fast-moving areas of technology, and one of the best documented. It has no inventor. Instead, it has a massive economy of people who have carefully and progressively solved a long series of immediate problems, documented their answers, and made those available to all. The innovative nature of the Internet comes not from a small, select band of Einsteins. It comes from RFCs anyone can use and improve, made by hundreds and thousands of smart, but not uniquely smart, individuals. It comes from open source software anyone can use and improve. It comes from sharing, scale of community, and the continuous accretion of good solutions and disposal of bad ones.

Here thus is an alternative theory of innovation:

1. There is an infinite problem/solution terrain.
2. This terrain changes over time according to external conditions.
3. We can only accurately perceive problems to which we are close.
4. We can rank the cost/benefit economics of problems using a market for solutions.
5. There is an optimal solution to any solvable problem.
6. We can approach this optimal solution heuristically, and mechanically.
7. Our intelligence can make this process faster, but does not replace it.

There are a few corollaries to this:

• Individual creativity matters less than process. Smarter people may work faster, but they may also work in the wrong direction. It's the collective vision of reality that keeps us honest and relevant.

• We don't need road maps if we have a good process. Functionality will emerge and evolve over time as solutions compete for market share.

• We don't invent solutions so much as discover them. All sympathies to the creative soul. It's just an information processing machine that likes to polish its own ego and collect karma.

• Intelligence is a social effect, though it feels personal. A person cut off from others eventually stops thinking. We can neither collect problems nor measure solutions without other people.

• The size and diversity of the community is a key factor. Larger, more diverse communities collect more relevant problems, and solve them more accurately, and do this faster, than a small expert group.

So, when we trust the solitary experts, they make classic mistakes. They focus on ideas, not problems. They focus on the wrong problems. They make misjudgments about the value of solving problems. They don't use their own work.

Can we turn the above theory into a reusable process? In late 2011, I started documenting C4.1 and similar contracts, and using them both in ZeroMQ and in closed source projects. The underlying process is something I call "Simplicity Oriented Design", or SOD. This is a reproducible way of developing simple and elegant products. It organizes people into flexible supply chains that are able to navigate a problem landscape rapidly and cheaply. They do this by building, testing, and keeping or discarding minimal plausible solutions, called "patches". Living products consist of long series of patches, applied one atop the other.

SOD is relevant first because it's how we evolve ZeroMQ. It's also the basis for the design process we will use in [#advanced-architecture] to develop larger-scale ZeroMQ applications. Of course, you can use any software architecture methodology with ZeroMQ.

To best understand how we ended up with SOD, let's look at the alternatives.

Trash-Oriented Design

The most popular design process in large businesses seems to be Trash-Oriented Design, or TOD. TOD feeds off the belief that all we need to make money are great ideas. It's tenacious nonsense, but a powerful crutch for people who lack imagination. The theory goes that ideas are rare, so the trick is to capture them. It's like non-musicians being awed by a guitar player, not realizing that great talent is so cheap it literally plays on the streets for coins.

The main output of TODs is expensive "ideation": concepts, design documents, and products that go straight into the trash can. It works as follows:

• The Creative People come up with long lists of "we could do X and Y". I've seen endlessly detailed lists of everything amazing a product could do. We've all been guilty of this. Once the creative work of idea generation has happened, it's just a matter of execution, of course.

• So the managers and their consultants pass their brilliant ideas to designers who create acres of preciously refined design documents. The designers take the tens of ideas the managers came up with, and turn them into hundreds of world-changing designs.

• These designs get given to engineers who scratch their heads and wonder who the heck came up with such nonsense. They start to argue back, but the designs come from up high, and really, it's not up to engineers to argue with creative people and expensive consultants.

• So the engineers creep back to their cubicles, humiliated and threatened into building the gigantic but oh-so-elegant junk heap. It is bone-breaking work because the designs take no account of practical costs. Minor whims might take weeks of work to build. As the project gets delayed, the managers bully the engineers into giving up their evenings and weekends.

• Eventually, something resembling a working product makes it out of the door. It's creaky and fragile, complex and ugly. The designers curse the engineers for their incompetence and pay more consultants to put lipstick onto the pig, and slowly the product starts to look a little nicer.

• By this time, the managers have started to try to sell the product and they find, shockingly, that no one wants it. Undaunted, they courageously build million-dollar web sites and ad campaigns to explain to the public why they absolutely need this product. They do deals with other businesses to force the product on the lazy, stupid, and ungrateful market.

• After twelve months of intense marketing, the product still isn't making profits. Worse, it suffers dramatic failures and gets branded in the press as a disaster. The company quietly shelves it, fires the consultants, buys a competing product from a small startup and rebrands that as its own Version 2. Hundreds of millions of dollars end up in the trash.

• Meanwhile, another visionary manager somewhere in the organization drinks a little too much tequila with some marketing people and has a Brilliant Idea.

Trash-Oriented Design would be a caricature if it wasn't so common. Something like 19 out of 20 market-ready products built by large firms are failures (yes, 87% of statistics are made up on the spot). The remaining 1 in 20 probably only succeeds because the competitors are so bad and the marketing is so aggressive.

The main lessons of TOD are quite straightforward but hard to swallow. They are:

• Ideas are cheap. No exceptions. There are no brilliant ideas. Anyone who tries to start a discussion with "oooh, we can do this too!" should be beaten down with all the passion one reserves for traveling evangelists. It is like sitting in a cafe at the foot of a mountain, drinking a hot chocolate and telling others, "Hey, I have a great idea, we can climb that mountain! And build a chalet on top! With two saunas! And a garden! Hey, and we can make it solar powered! Dude, that's awesome! What color should we paint it? Green! No, blue! OK, go and make it, I'll stay here and make spreadsheets and graphics!"

• The starting point for a good design process is to collect real problems that confront real people. The second step is to evaluate these problems with the basic question, "How much is it worth to solve this problem?" Having done that, we can collect that set of problems that are worth solving.

• Good solutions to real problems will succeed as products. Their success will depend on how good and cheap the solution is, and how important the problem is (and sadly, how big the marketing budgets are). But their success will also depend on how much they demand in effort to use—in other words, how simple they are.

Now, after slaying the dragon of utter irrelevance, we attack the demon of complexity.

Complexity-Oriented Design

Really good engineering teams and small firms can usually build decent products. But the vast majority of products still end up being too complex and less successful than they might be. This is because specialist teams, even the best, often stubbornly apply a process I call Complexity-Oriented Design, or COD, which works as follows:

• Management correctly identifies some interesting and difficult problem with economic value. In doing so, they already leapfrog over any TOD team.

• The team with enthusiasm starts to build prototypes and core layers. These work as designed and thus encouraged, the team go off into intense design and architecture discussions, coming up with elegant schemas that look beautiful and solid.

• Management comes back and challenges the team with yet more difficult problems. We tend to equate cost with value, so the harder and more expensive to solve, the more the solution should be worth, in their minds.

• The team, being engineers and thus loving to build stuff, build stuff. They build and build and build and end up with massive, perfectly-designed complexity.

• The products go to market, and the market scratches its head and asks, "Seriously, is this the best you can do?" People do use the products, especially if they aren't spending their own money in climbing the learning curve.

• Management gets positive feedback from its larger customers, who share the same idea that high cost (in training and use) means high value, and so continues to push the process.

• Meanwhile somewhere across the world, a small team is solving the same problem using a better process, and a year later smashes the market to little pieces.

COD is characterized by a team obsessively solving the wrong problems in a form of collective delusion. COD products tend to be large, ambitious, complex, and unpopular. Much open source software is the output of COD processes. It is insanely hard for engineers to stop extending a design to cover more potential problems. They argue, "What if someone wants to do X?" but never ask themselves, "What is the real value of solving X?"

A good example of COD in practice is Bluetooth, a complex, over-designed set of protocols that users hate. It continues to exist only because in a massively-patented industry there are no real alternatives. Bluetooth is perfectly secure, which is close to pointless for a proximity protocol. At the same time, it lacks a standard API for developers, meaning it's really costly to use Bluetooth in applications.

On the #zeromq IRC channel, Wintre once wrote of how enraged he was many years ago when he "found that XMMS 2 had a working plugin system, but could not actually play music."

COD is a form of large-scale "rabbit-holing", in which designers and engineers cannot distance themselves from the technical details of their work. They add more and more features, utterly misreading the economics of their work.

The main lessons of COD are also simple, but hard for experts to swallow. They are:

• Making stuff that you don't immediately have a need for is pointless. Doesn't matter how talented or brilliant you are, if you just sit down and make stuff people are not actually asking for, you are most likely wasting your time.

• Problems are not equal. Some are simple, and some are complex. Ironically, solving the simpler problems often has more value to more people than solving the really hard ones. So if you allow engineers to just work on random things, they'll mostly focus on the most interesting but least worthwhile things.

• Engineers and designers love to make stuff and decoration, and this inevitably leads to complexity. It is crucial to have a "stop mechanism", a way to set short, hard deadlines that force people to make smaller, simpler answers to just the most crucial problems.

Simplicity Oriented Design

Finally, we come to the rare but precious Simplicity Oriented Design, or SOD. This process starts with a realization: we do not know what we have to make until after we start making it. Coming up with ideas or large-scale designs isn't just wasteful, it's a direct hindrance to designing the truly accurate solutions. The really juicy problems are hidden like far valleys, and any activity except active scouting creates a fog that hides those distant valleys. You need to keep mobile, pack light, and move fast.

SOD works as follows:

• We collect a set of interesting problems (by looking at how people use technology or other products) and we line these up from simple to complex, looking for and identifying patterns of use.

• We take the simplest, most dramatic problem and we solve this with a minimal plausible solution, or "patch". Each patch solves exactly a genuine and agreed-upon problem in a brutally minimal fashion.

• We apply one measure of quality to patches, namely "Can this be done any simpler while still solving the stated problem?" We can measure complexity in terms of concepts and models that the user has to learn or guess in order to use the patch. The fewer, the better. A perfect patch solves a problem with zero learning required by the user.

• Our product development consists of a patch that solves the problem "we need a proof of concept" and then evolves in an unbroken line to a mature series of products, through hundreds or thousands of patches piled on top of each other.

• We do not do anything that is not a patch. We enforce this rule with formal processes that demand that every activity or task is tied to a genuine and agreed-upon problem, explicitly enunciated and documented.

• We build our projects into a supply chain where each project can provide problems to its "suppliers" and receive patches in return. The supply chain creates the "stop mechanism" because when people are impatiently waiting for an answer, we necessarily cut our work short.

• Individuals are free to work on any projects, and provide patches at any place they feel it's worthwhile. No individuals "own" any project, except to enforce the formal processes. A single project can have many variations, each a collection of different, competing patches.

• Projects export formal and documented interfaces so that upstream (client) projects are unaware of change happening in supplier projects. Thus multiple supplier projects can compete for client projects, in effect creating a free and competitive market.

• We tie our supply chain to real users and external clients and we drive the whole process by rapid cycles so that a problem received from outside users can be analyzed, evaluated, and solved with a patch in a few hours.

• At every moment from the very first patch, our product is shippable. This is essential, because a large proportion of patches will be wrong (10-30%) and only by giving the product to users can we know which patches have become problems that need solving.

SOD is a hill-climbing algorithm, a reliable way of finding optimal solutions to the most significant problems in an unknown landscape. You don't need to be a genius to use SOD successfully, you just need to be able to see the difference between the fog of activity and the progress towards new real problems.

People have pointed out that hill-climbing algorithms have known limitations. One gets stuck on local peaks, mainly. But this is nonetheless how life itself works: collecting tiny incremental improvements over long periods of time. There is no intelligent designer. We reduce the risk of local peaks by spreading out widely across the landscape, but it is somewhat moot. The limitations aren't optional, they are physical laws. The theory says, this is how innovation really works, so better embrace it and work with it than try to work on the basis of magical thinking.

And in fact once you see all innovation as more or less successful hill-climbing, you realize why some teams and companies and products get stuck in a never-never land of diminishing prospects. They simply don't have the diversity and collective intelligence to find better hills to climb. When Nokia killed their open source projects, they cut their own throat.

A really good designer with a good team can use SOD to build world-class products, rapidly and accurately. To get the most out of SOD the designer has to use the product continuously, from day one, and develop his or her ability to smell out problems such as inconsistency, surprising behavior, and other forms of friction. We naturally overlook many annoyances, but a good designer picks these up and thinks about how to patch them. Design is about removing friction in the use of a product.

In an open source setting, we do this work in public. There's no "let's open the code" moment. Projects that do this are in my view missing the point of open source, which is to engage your users in your exploration, and to build community around the seed of the architecture.

Burnout

The ZeroMQ community has been and still is heavily dependent on pro bono individual efforts. I'd like to think that everyone was compensated in some way for their contributions, and I believe that with ZeroMQ, contributing means gaining expertise in an extraordinarily valuable technology, which leads to improved professional options.

However, not all projects will be so lucky and if you work with or in open source, you should understand the risk of burnout that volunteers face. This applies to all pro bono communities. In this section, I'll explain what causes burnout, how to recognize it, how to prevent it, and (if it happens) how to try to treat it. Disclaimer: I'm not a psychiatrist and this article is based on my own experiences of working in pro bono contexts for the last 20 years, including free software projects, and NGOs such as the FFII.

In a pro bono context, we're expected to work without direct or obvious economic incentive. That is, we sacrifice family life, professional advancement, free time, and health in order to accomplish some goal we have decided to accomplish. In any project, we need some kind of reward to make it worth continuing each day. In most pro bono projects the rewards are very indirect, superficially not economical at all. Mostly, we do things because people say, "Hey, great!" Karma is a powerful motivator.

However, we are economic beings, and sooner or later, if a project costs us a great deal and does not bring economic rewards of some kind (money, fame, a new job), we start to suffer. At a certain stage, it seems our subconscious simply gets disgusted and says, "Enough is enough!" and refuses to go any further. If we try to force ourselves, we can literally get sick.

This is what I call "burnout", though the term is also used for other kinds of exhaustion. Too much investment on a project with too little economic reward, for too long. We are great at manipulating ourselves and others, and this is often part of the process that leads to burnout. We tell ourselves that it's for a good cause and that the other guy is doing OK, so we should be able to as well.

When I got burned out on open source projects like Xitami, I remember clearly how I felt. I simply stopped working on it, refused to answer any more emails, and told people to forget about it. You can tell when someone's burned out. They go offline, and everyone starts saying, "He's acting strange… depressed, or tired…"

Diagnosis is simple. Has someone worked a lot on a project that was not paying back in any way? Did she make exceptional sacrifices? Did he lose or abandon his job or studies to do the project? If you're answering "yes", it's burnout.

There are three simple techniques I've developed over the years to reduce the risk of burnout in the teams I work with:

• No one is irreplaceable. Working solo on a critical or popular projectthe concentration of responsibility on one person who cannot set their own limitsis probably the main factor. It's a management truism: if someone in your organization is irreplaceable, get rid of him or her.

• We need day jobs to pay the bills. This can be hard, but seems necessary. Getting money from somewhere else makes it much easier to sustain a sacrificial project.

• Teach people about burnout. This should be a basic course in colleges and universities, as pro bono work becomes a more common way for young people to experiment professionally.

When someone is working alone on a critical project, you know they are going blow their fuses sooner or later. It's actually fairly predictable: something like 18-36 months depending on the individual and how much economic stress they face in their private lives. I've not seen anyone burn-out after half a year, nor last five years in a unrewarding project.

There is a simple cure for burnout that works in at least some cases: get paid decently for your work. However, this pretty much destroys the freedom of movement (across that infinite problem landscape) that the volunteer enjoys.

Patterns for Success

I'll end this code-free chapter with a series of patterns for success in software engineering. They aim to capture the essence of what divides glorious success from tragic failure. They were described as "religious maniacal dogma" by a manager, and "anything else would be effing insane" by a colleague, in a single day. For me, they are science. But treat the Lazy Perfectionist and others as tools to use, sharpen, and throw away if something better comes along.

The Lazy Perfectionist

Never design anything that's not a precise minimal answer to a problem we can identify and have to solve.

The Lazy Perfectionist spends his idle time observing others and identifying problems that are worth solving. He looks for agreement on those problems, always asking, "What is the real problem". Then he moves, precisely and minimally, to build, or get others to build, a usable answer to one problem. He uses, or gets others to use those solutions. And he repeats this until there are no problems left to solve, or time or money runs out.

The Benevolent Tyrant

The control of a large force is the same principle as the control of a few men: it is merely a question of dividing up their numbers. — Sun Tzu

The Benevolent Tyrant divides large problems into smaller ones and throws them at groups to focus on. She brokers contracts between these groups, in the form of APIs and the "unprotocols" we'll read about in the next chapter. The Benevolent Tyrant constructs a supply chain that starts with problems, and results in usable solutions. She is ruthless about how the supply chain works, but does not tell people what to work on, nor how to do their work.

The Earth and Sky

The ideal team consists of two sides: one writing code, and one providing feedback.

The Earth and Sky work together as a whole, in close proximity, but they communicate formally through issue tracking. Sky seeks out problems from others and from their own use of the product and feeds these to Earth. Earth rapidly answers with testable solutions. Earth and Sky can work through dozens of issues in a day. Sky talks to other users, and Earth talks to other developers. Earth and Sky may be two people, or two small groups.

The Open Door

The accuracy of knowledge comes from diversity.

The Open Door accepts contributions from almost anyone. She does not argue quality or direction, instead allowing others to argue that and get more engaged. She calculates that even a troll will bring more diverse opinion to the group. She lets the group form its opinion about what goes into stable code, and she enforces this opinion with help of a Benevolent Tyrant.

The Laughing Clown

Perfection precludes participation.

The Laughing Clown, often acting as the Happy Failure, makes no claim to high competence. Instead his antics and bumbling attempts provoke others into rescuing him from his own tragedy. Somehow however, he always identifies the right problems to solve. People are so busy proving him wrong they don't realize they're doing valuable work.

The Mindful General

Make no plans. Set goals, develop strategies and tactics.

The Mindful General operates in unknown territory, solving problems that are hidden until they are nearby. Thus she makes no plans, but seeks opportunities, then exploits them rapidly and accurately. She develops tactics and strategies in the field, and teaches these to her soldiers so they can move independently, and together.

The Social Engineer

If you know the enemy and know yourself, you need not fear the result of a hundred battles. — Sun Tzu

The Social Engineer reads the hearts and minds of those he works with and for. He asks, of everyone, "What makes this person angry, insecure, argumentative, calm, happy?" He studies their moods and dispositions. With this knowledge he can encourage those who are useful, and discourage those who are not. The Social Engineer never acts on his own emotions.

The Constant Gardener

He will win whose army is animated by the same spirit throughout all its ranks. — Sun Tzu

The Constant Gardener grows a process from a small seed, step-by-step as more people come into the project. She makes every change for a precise reason, with agreement from everyone. She never imposes a process from above but lets others come to consensus, and then he enforces that consensus. In this way, everyone owns the process together and by owning it, they are attached to it.

The Rolling Stone

After crossing a river, you should get far away from it. — Sun Tzu

The Rolling Stone accepts his own mortality and transience. He has no attachment to his past work. He accepts that all that we make is destined for the trash can, it is just a matter of time. With precise, minimal investments, he can move rapidly away from the past and stay focused on the present and near future. Above all, he has no ego and no pride to be hurt by the actions of others.

The Pirate Gang

Code, like all knowledge, works best as collectivenot privateproperty.

The Pirate Gang organizes freely around problems. It accepts authority insofar as authority provides goals and resources. The Pirate Gang owns and shares all it makes: every work is fully remixable by others in the Pirate Gang. The gang moves rapidly as new problems emerge, and is quick to abandon old solutions if those stop being relevant. No persons or groups can monopolize any part of the supply chain.

The Flash Mob

Water shapes its course according to the nature of the ground over which it flows. — Sun Tzu

The Flash Mob comes together in space and time as needed, then disperses as soon as they can. Physical closeness is essential for high-bandwidth communications. But over time it creates technical ghettos, where Earth gets separated from Sky. The Flash Mob tends to collect a lot of frequent flier miles.

The Canary Watcher

Pain is not, generally, a Good Sign.

The Canary Watcher measures the quality of an organization by their own pain level, and the observed pain levels of those with whom he works. He brings new participants into existing organizations so they can express the raw pain of the innocent. He may use alcohol to get others to verbalize their pain points. He asks others, and himself, "Are you happy in this process, and if not, why not?" When an organization causes pain in himself or others, he treats that as a problem to be fixed. People should feel joy in their work.

The Hangman

Never interrupt others when they are making mistakes.

The Hangman knows that we learn only by making mistakes, and she gives others copious rope with which to learn. She only pulls the rope gently, when it's time. A little tug to remind the other of their precarious position. Allowing others to learn by failure gives the good reason to stay, and the bad excuse to leave. The Hangman is endlessly patient, because there is no shortcut to the learning process.

The Historian

Keeping the public record may be tedious, but it's the only way to prevent collusion.

The Historian forces discussion into the public view, to prevent collusion to own areas of work. The Pirate Gang depends on full and equal communications that do not depend on momentary presence. No one really reads the archives, but the simply possibility stops most abuses. The Historian encourages the right tool for the job: email for transient discussions, IRC for chatter, wikis for knowledge, issue tracking for recording opportunities.

The Provocateur

When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully. — Samuel Johnson

The Provocateur creates deadlines, enemies, and the occasional impossibility. Teams work best when they don't have time for the crap. Deadlines bring people together and focus the collective mind. An external enemy can move a passive team into action. The Provocateur never takes the deadline too seriously. The product is always ready to ship. But she gently reminds the team of the stakes: fail, and we all look for other jobs.

The Mystic

When people argue or complain, just write them a Sun Tzu quotation — Mikko Koppanen

The Mystic never argues directly. He knows that to argue with an emotional person only creates more emotion. Instead he side-steps the discussion. It's hard to be angry at a Chinese general, especially when he has been dead for 2,400 years. The Mystic plays Hangman when people insist on the right to get it wrong.

Conclusions

This article tells a long story that started when I read Stallman's accounts of writing his first free software. In 2005 we were building on-line communities deliberately and aggressively, for political purposes. By then we'd codified the theory and were applying it over and over. In 2007 I used this to build a large community for the Wikidot.com platform, which I'd invested in and was CEO of. In 2009 I used it as the basis for the ZeroMQ community, and by 2011 had turned this fire on full, stripping away all the old clumsy patterns, and replacing them with upgraded state-of-the-art techniques.

We are still learning, making our processes simpler, and our tools sharper. C4.1 is not for everyone. It takes courage to embrace unknown contributors and trust them by default. It takes experience to realize that for every twenty smiles, there is one knife. We learn these lessons slowly. Even with a full handbook, it will take you years to understand. So practice, be prepared to fail often, and be happy. :)

by pieterh

As I sat down in my living room, surrounded by the material of my life, my children, my family, my cats, I cried. The overwhelming sense of joy was so intense that I almost started tripping. Then I found the oxygen, and things came back into focus.

The chemo, this first round, was miserable. I wanted to write a happy note about the lack of side effects, to cheer people up. Instead I spent most of Friday and Saturday vomiting, in one case projecting green-yellow vomit across several meters of my room. I think I'd eaten a couple of olives for supper. Hospital rooms and staff take this kind of thing in their stride. A few sweeps and it was all gone.

I'm not a good vomiter, and the doctors gave me increasing doses of different drugs to fight the nausea. Only when my oncologist said, "if you vomit again, we'll stick a drip back into you and start brain scans," did my body realize this had to stop. It's like an anti-placebo. The threat of not getting free, and worse, switched off all my nausea like flicking a switch.

Curious to see how it goes with the next round of chemo next week, now that I know the effect is at least partly psychological.

Vomit or not, I'm dying. I'm on palliative care, and the doctors did not switch off their slightly grim "we'll do what we can" manner. So different from their "oh, you'll be fine!" faces.

To be clear, I'm not resigned, hopeless, or fatalistic. I'm absolutely determined to beat this cancer, by sheer force of will, and blind hope in the miraculous. This is how I'm designed, like a unbreakable self-righting toy. Put me into any situation, no matter how impossible, and I will always find a way to make things work. Yet I know that I'm bullshitting myself in this case. "Always" only ever means, "so far, so good."

Status: awesome :)

I can feel the cancer growing in me, my breathing slowly getting shallower and less easier. It is like the atmosphere has lost its essence. I breath oxygen through a tube, one liter a minute, and live with this cable like in the old times, when computers could move as far as their wires. A portable saturation meter (thank you, Amazon, you've made my life so much easier over the years) tells me I'm at 96%.

Sometimes I take off the nose piece, and "go WiFi" for a few minutes, maybe half an hour. This lets me put my sons to bed, tell them stories. Gregor, the youngest, is so comfortable in my arms he falls asleep after two to three minutes, always.

Freeman listens to my stories with attention, as he has always done. Over the last years we told the stories of Bobolan the Magician, who built the largest magical university ever. It all started at the End of the World, when Bobolan had to find five precious stones to restart the Sun. That story took us a month to tell. Jack Vance, Terry Pratchett, forgive me for taking a few riffs from your songs.

My children seem to be doing well. They are calm and cheerful, focused on their daily rituals. School, computers, feeding the cats, helping in the house. It took a day or so for us to adjust after my absence and return.

There is no hiding that I'm sick: my medical bed with its motors and cables is in our large living room, the center of our home. When I cough, my kids hear it, they can follow my slow downwards curve.

Not that I'm getting weaker. Hospital left me weak and thin, 14 kilos lost, slow shuffling walk. Now I am quickly getting back in shape, up and down the stairs, WiFi. Being at home means I feed myself. Food has taste again!

And home, I have space for my friends and family as they visit. I hope the funerary procession of that Sunday ten days past is finished, when fifteen people visited me in my small room. Some I'd not seen for years, decades. Why, I wonder, would you come to see me sick, when we don't talk in real life? I get it though, the social rituals of saying goodbye. I notice that flexibility, being able to greet the not-yet-dead in much the same way as observing silence in presence of a body.

A friend came up from Paris with a bottle of wine, warm socks, chocolate drops she made in her restaurant. My daughter ate the chocolates. I put the socks on. We opened the wine. Wine and beer, I asked my doctor, before checking out. How much can I drink? He frowned… if you're not taking medication, then a glass or two can't do any harm.

She told her colleagues, "I'm taking a few days off work to say goodbye to a friend who's dying." A strange reason, perhaps, and yet spending a couple of days together, slowly talking about life, knowing this is the last time we'll see each other… it feels as natural as cooking a meal with friends.

I think "euthanasia" as a term has some problems. It's too easily hijacked by the "death panel" lunatic fringe who believe that pain and suffering is our destiny. Switch off your Internet and heating, you psychopaths, if that's what you really think. No, what they really mean is "I claim the authority over others."

Let's use the term "planned death," it is accurate. Planning our deaths. It may be a luxury for some, and for others, besides the point, yet for those like me who see the road ahead, I claim this to be an essential Human Right.

How else can this all work? My children don't want or need to see me rot and fall apart. They don't want me to go away again, disappear into that machine called "medicine." Talk about a lifelong trauma. I can offer my children a model of control, careful organization, order out of chaos. It's what I've always taught them. Chaos is the default: do not wait for others to fix it. That's your responsibility: organize your world, take control.

Not all doctors are willing to kill their patients. It is the first discussion I had with my new family doctor. "Are you on board with the whole killing me when it's the right time thing?" Sure, he said, your oncologist already asked me that before choosing me. Good man. We signed a contract and shook hands.

My oncologist. I have to say, a wonderful woman. I joked with my sisters, when the tall male head oncologist visited me in my room, that I was grateful to finally see a real doctor. My sister, a doctor, pretended shock and outrage. In truth, my oncologist took me under her wings and pulled every string necessary to get me home in the best of conditions.

A planned death is not a moment in time, like a car accident or a fatal stroke. It is a process. A social process that involves hundreds of people, each doing their part, grieving their loss, accepting their own mortality.

I'm proud and immensely grateful to be able to experience this, and share it with you.

by pieterh

Yesterday I had twelve visitors, including my lovely young children. You'd think it's exhausting, yet the non-stop flow of friends and family was like being in a luxurious hot bath with an infinite supply of fresh water.

I was a disconnected and lonely young man. Somewhat autistic, perhaps. I thought only of work, swimming, my pet cats, work. The notion that people could enjoy my company was alien to me. At least my work, I felt, had value. We wrote code generators in Cobol. I wrote a code editor that staff loved because it worked elegantly and ran on everything. I taught myself C and 8086 assembler and wrote shareware tools. The 1990's slowly happened.

Over time I learned that if you chat with a stranger, in the course of any kind of interaction (like buying a hot dog, or groceries) they'll chat back with a beam of pleasure. Slowly, like a creeping addiction to coffee, this became my drug of choice.

In time it became the basis, and then the goal of my work: to go to strange places and meet new people. I love the conferences because you don't need an excuse. Everyone there wants, and expects, to talk. I rarely talk about technical issues. Read the code, if you want that.

And so I'm proud of my real work, which has been for decades, to talk with people, listen and exchange knowledge, and then synthesize this and share it on with others. Thousands of conversations across Europe, America, Africa, Asia. I'll take whatever credit people want to give me for being creative, brilliant, etc. Yet the models and theories I've shaped and documented are consistently drawn from real-life experience with other people.

Thank you, my friends, for that. When I say "I love you" it's not some gesture. You literally kept me fed, professionally and intellectually.

So I wanted to document one last model, which is how to die, given some upfront knowledge and time. I'm not going to write an RFC this time. :)

How it Happened

Technically, I have metastasis of bile duct cancer, in both lungs. Since February I've had this dry cough, and been increasingly tired and unfocused on work. In March my Father died and we rushed around arranging that. My cough took a back seat. On April 8 I went to my oncologist to say that I was really not well. She organized a rush CAT scan and blood tests.

On 13 April, a horrific bronchoscopy and biopsies. On 15 April, a PET scan. On 16 April I was meant to drive to Eindhoven to keynote at NextBuild. Instead I went to the emergency room with explosive pains in my side, where they'd done the biopsies. I was checked in and put on antibiotics, which fixed the pain, and on 18 April my oncologist confirmed it was cancer. I'm still here, and my doctors are thinking what chemo to try on me. It is an exotic cancer in Europe with little solid data.

What we do know is that cholangiocarcinoma does not respond well to chemotherapy. Further, that my cancer is aggressive and fast moving. Third, I've already some clusters in other parts of my body. All this is clear and solid data.

So that day I told the world about it, and prepared to die.

Talking to a Dying Person

It can be horribly awkward to talk to a dying person (let's say "Bob"). Here are the main things the other person (let's say "Alice") should not say to Bob:

• "Hang in there! You must have hope, you must fight!" It's safe to assume that Bob is fighting as hard as possible. And if not, that's entirely Bob's choice.

• "This is so tragic, I'm so sad, please don't die!" Which my daughter said to me one time. I explained softly that you cannot argue with facts. Death is not an opinion. Being angry or sad at facts is a waste of time.

• "You can beat this! You never know!" Which is Alice expressing her hope. False hope is not a medicine. A good chemotherapy drug, or a relaxing painkiller, that's medicine.

• "There's this alternative cure people are talking about," Which gets the ban hammer from me, and happily I only got a few of those. Even if there was a miracle cure, the cost and stress (to others) of seeking it is such a selfish and disproportionate act. With, as we know, lottery-style chances of success. We live, we die.

• "Read this chapter in the Bible, it'll help you." Which is both rude and offensive, as well as being clumsy and arrogant. If Bob wants religious advice he'll speak to his priest. And if not, just do not go there. It's another ban hammer offense.

• Engage in slow questioning. This is passive-predatory, asking Bob to respond over and over to small, silly things like "did I wake you?" Bob is unlikely to be a mood for idle chitchat. He either wants people close to him, physically, or interesting stuff (see below).

Above all, do not call and then cry on the phone. If you feel weepy, cut the phone, wait ten minutes, then call back. Tears are fine, yet for Bob, the threat of self-pity looms darker than anything. I've learned to master my emotions yet most Bobs will be vulnerable.

Here are the things that Alice can talk about that will make Bob happy:

• Stories of old adventures they had together. Remember that time? Oh boy, yes I do… it was awesome!

• Clinical details. Bob, stuck in his bed, is probably obsessed by the rituals of care, the staff, the medicines, and above all, his disease. I'll come to Bob's duty to share, in a second.

• Helping Bob with technical details. Sorting out a life is complex and needs many hands and minds.

• "I bought your book," assuming Bob is an author like me. It may be flattery, or sincere, either way it'll make Bob smile.

Above all, express no emotions except happiness, and don't give Bob new things to deal with.

Bob's Duties

It's not all Alice's work. Bob too has obligations under this protocol. They are, at least:

• Be happy. This may sound trite yet it's essential. If you are going to be gloomy and depressed, Alice will be miserable every time she talks to you.

• Obviously, put your affairs in order. I've been expecting death for years now, so had been making myself disposable wherever I could. For family, that is not possible. For work, yes, and over the years I've removed myself as a critical actor from the ZeroMQ community.

• Remove all stress and cost that you can. For example Belgium permits euthanasia. I've already asked my doctors to prepare for that. (Not yet!, when it's time…) I've asked people to come say goodbye before I die, not after. No funeral. I'll give my remains to the university here, if they want them.

• Be realistic. Hope is not medicine, as I explained. If you are going to negotiate with your doctors, let it be pragmatic and in everyone's interests. I've told mine they can try whatever experimental chemotherapy they wish to. It's data for them, and the least I can do for a system that's given me five+ years of extra life.

• Assume the brutal worst. When my oncologist saw my scan she immediately called me and told me, in her opinion, it was cancer. In both lungs, all over the place. I put the phone down, and told the children. The next day I told their schools to expect the worst, then my lawyer, then my notary. Ten days later the biopsies confirmed it. That gave us ten more days of grieving and time to prepare.

• Be honest and transparent with others. It takes time to grieve and it is far easier to process Bob's death when you can talk about it with Bob. There is no shame in dying, it is not a failure.

Explaining to the Children

My kids are twelve, nine, five. Tragic, etc. etc. Growing up without a father. It is a fact. They will grow up with me in their DNA, on Youtube as endless conference talks, and in writing.

I've explained it to them slowly, and many times over the years, like this. One day, I will be gone. It may be long away, it may be soon. We all die, yes, even you little Gregor. It is part of life.

Imagine you have a box of Lego, and you build a house, and you keep it. And you keep making new houses, and never breaking the old ones. What happens? "The box gets empty, Daddy." Good, yes. And can you make new houses then? "No, not really." So we're like a Lego houses, and when we die our pieces get broken up and put back in the box. We die, and new babies can be born. It is the wheel of life.

But mostly I think seeing their parent happy and relaxed (not due to pain killers), and saying goodbye over weeks feels right. I am so grateful not to have died suddenly. I'm so grateful I won't lose my mind.

And I've taught my children, to swim and bike and skate and shoot. To cook, to travel and to camp. To use technology without fear. At three, Gregor was on Minecraft, keyboard in left hand, mouse in right. At seven, Noemie learned to shoot a pistol. They speak several languages. They are confident and quick learners, like their dad.

And everyone needs to learn what it means to die. It is a core part of being a full human, the embrace of one's mortality. We fight to live, of course. And when it's over, we embrace the end. I'm happy that I can teach this lesson to my children, it is one that I never had.

Euthanasia

I am, finally, so glad I never quit Belgium. This country allows for death on demand, for patients who are terminal or have a bad enough quality of life. It takes three doctors and a psychiatrist, in the second case, and four weeks' waiting period. In the first case, it takes one doctor's opinion.

My dad chose this, and died on Easter Tuesday. Several of us his family were with him. It is a simple and peaceful process. One injection sent him to sleep, into a coma. The second stopped his heart. It was a good way to die, and though I didn't know I was sick then, one I already wanted.

I'm shocked that in 2016 few countries allow this, and enforce the barbaric torture of decay and failure. It's especially relevant for cancer, which is a primary cause of death. Find a moment in your own jurisdiction, if it bans euthanasia, to lobby for the right to die in dignity.

My Feelings on All This

I've never been a fearful person. My last brush with death left me so casual about the whole concept of professional and social risk that I became the predatory character Allen Ding so nicely describes. That calmed down after our Game of Thrones project ended. It was never really me, just the person I became to make things work, in that place and time.

Having had years to prepare for this, and having seen a great many delicate plans come together over those years, leaves me deeply satisfied. Since 2011 I've become an expert pistol shot, taught myself to play piano (and composed many small pieces), seen my children grow into happy, bubbling characters, written three books, coached the ZeroMQ community into serene self-reliability. What more can a Bob ask for?

The staff here are lovely. I've no complaints, only gratitude to all my friends for the years of pleasure you've given me, my drug, which kept me alive and driven.

Thank you! :)

Think of the Children

Please use this article to add your stories. If you have them elsewhere, or you emailed me, copy/paste as a comment. Feel free to write in Dutch or French if that's your language. I'd really like a single place where my kids can come and read what other people say about their dad.

Many people have asked my PayPal address moc.xitami|hp#moc.xitami|hp, to send a donation for my children.

Living Obituaries

Thank you to the following people for their articles:Ewen McNeill, Allen Ding, Meredith L. Patterson, Dylan Beattie, Jef Claes, Josh Long, Brian Knox, Yves, Alan Yorinks, Stijn Volders.

Translations and Reproductions

This article or parts of it have been reproduced in Chinese, Facebook, Russian, Geek times, Italian, Il Post, The Guardian, Dutch, RTL Nieuws, N.TV in Germany, and French, Romanian, and was much discussed on Hacker News.

by pieterh

I think Twitter has become a recruiting tool for a number of toxic cults. These disguise themselves as 'left wing' and 'right wing' and other forms of activism. What they do in fact is spread hate in the name of hits, followers, income, and power. They use each other to troll and divide.

The Cults

Let me cite these by name and explain each. I'm not going to use any links. Explore further at your own risk.

We have the third wave feminists (3WF), promising a nirvana where maleness is a birth crime and femininity will Save the World. It has its mystical roots in Gaia the Earth Mother, and it appeals to an ancient conflict between men and women. The conflict expresses in every couple, every family, and within individuals.

3WF collects women looking for a cause, often highly educated, under-challenged, and lonely. This is a generation who have learned that doing female things, and following the female biological imperatives, is "bad." And yes, these are real things. If you hear a voice shouting "social construct!" in your mind, you are already under a form of mind control.

Humanity is the only species capable of denying its own existence.

It is ironic, and sad, that 3WF accuses women that refuse to accept its message as suffering from "internalized misogyny." That is, self-hate so deep they cannot see it. And yet it is 3WF members that hate their own femaleness. They take no pride in it. It disgusts them so much they must project that onto others.

3WF also collects men, such as a few "male feminists" whom I've had to block to protect them from my need to always be right. Why do boys follow a girls' cult? Partly it's biology. The men go where the women are. We will say and do a lot of stupid things for the remotest chance at sex. But a lot of men have also grown up in the same education system, learning from a young age that gender is a social construct, and Male = Bad, Female = Good.

I am an activist, and have spent decades of my life working pro-bono for the benefit of others. This seems a small price to pay for enjoying a prosperous and stable society. Yet 3WF is not pro-bono activism. It does not have any tangible impact on real problems facing the vulnerable in society: many women, and also children, the aged, and many men. Do I really need to cite statistics showing that most violence against children comes from women? I'd rather not, because it seems I'm excusing male violence, which I'm not.

What 3WF clearly is, is a psychopathic cult. Its leaders, whom I gladly brand as psychopaths, gather significant power and money. They wield their followers like a weapon against opponents, to stop debate. They invent jargon and magical theories. An MTV talking head proudly states, "'Mansplaining' is now in the dictionary. If that's not free speech, I don't know what is!"

Why and how did feminism get hijacked? I'm not an expert so have asked some who are. Their answers: feminism in the West largely accomplished its goals by 2000 or so. There is no wage gap, no discrimination of note. As the core of the movement was based on conflict with maleness, it had to find new targets, and new leaders willing to aim at these targets. Hence its expansion into hate, and its domination by psychopaths.

The second cult is close and yet distinct, by definition. It was born as "Black Lives Matter" around 2014, and has grown into a national movement calling itself the "Black Liberation Collective," or BLC. It wears the clothes and language of the Black Power movement of the 1950s. Yet while that movement was born in the ghettos, and fought real problems of racism, BLC seems obsessed with posture and gesture. It is a Facebook movement, narcissist to the core.

BLC has spread rapidly into college campuses across the US. While college students seemed like Fair Game to aggressive pepper-spraying police a few years before, BLC raised the stakes. "Hit one of us," they said, "and you hit an entire people." As a threat, it was effective. Yet what has BLC done with that? It has turned campus after campus into scenes from Lord of the Flies. What happens when spoiled children get the feeling of power over others? Read thedemands.org.

What does BLC promise its supporters, the children of wealthy families, freed from the burden of work or child care, freed even from the need to study? Not liberation, for sure. What I think it offers is appeasement for survivor guilt, someone to blame, and a rough answer.

I've witnessed real racism in the States, black friends of mine who spent the weekends in gaol to pay off court fees for unjustified traffic fines. I've seen families destroyed by alcohol, drugs, and the sheer negligence of the state. There are real problems to solve. And there are many people working to solve these problems.

BLC ignores such problems. Instead, it takes the guilt of the recently well-off, casts it into the worst kind of tribalism, and then promises a future where "whiteness" is eradicated. It casts the sins of the world into the pale-skinned tribes, then seeks to throw them over a cliff like the Biblical swine.

Here is the final BLC demand to the University of Missouri:

We demand that the University of Missouri increases funding, resources, and personnel for the social justices centers on campus for the purpose of hiring additional professionals, particularly those of color, boosting outreach and programming across campus, and increasing campus­wide awareness and visibility.

And in 2015, that university lost $20-$25 million in tuition as students voted with their feet. BLC starts to look less like a cure, and more like a fatal disease.

The goal here is not progress. It is chaos, destruction, the tabula rasa of revolution.

3WF and BLC are similar in key ways. They both worship the god Ego, with different prophets. They both speak to the same pampered, indolent middle class. They both focus on the same villains: the white male establishment, aka the "Patriarchy."

And both cults go one step further: they claim the status of absolute victim, and in doing so they deny the real victims their place. 3WF ignores the spread of vulnerability across society, particularly children and the elderly. BLC ignores the mass of poverty that cannot claim one drop of African or Native American blood. It is not a small mass, as Donald Trump knows.

Yes, America is a divided society. Extraordinarily unfair in ways that can make no sense to the outside observer. Filled with anger, hate, and intolerance. Focused on tribalism and difference. These are real problems that can be solved, and probably will be solved, over time. Yet these two cults embrace the problems and amplify them, in their thirst for power.

Since both cults are growing, perhaps at a time when more traditional cults like Scientology are shrinking, they work together. Soon enough they will be chewing at each others' throats. Yet for now they agree on the common enemy and their strategy of tension.

We see such cults of hate across the world, wherever there are unresolved social divisions caused by history. The natural tendency of many people is to choose a side, which worsens the divisions. Psychopaths love such opportunities. "Hate the infidel," they shout. "Bomb the Muslim!" they demand. "Kill the Jew," they whisper.

BLC's difference is its embrace of the theory of race. Does this have a biological basis?

Let me cite one study from Sarah Tishkoff. Her team found fourteen ancestral human genetic families. These match ancient linguistic families. Nine of these are in Africa, five in the rest of the world. Around 75% of African-Americans share a Niger-Kordofanian family origin.

Does this constitute two "races"? Or even two "ethnicities"? If you want to count, there are probably over a dozen, most of which are "black." And all modern humans are a mongrel mix of these. Gene flow has no mercy, which is why Russian Jews look like Russians and Ethiopian Jews look Ethiopian, despite thousands of years of mixing taboos.

There is no "black" and there is no "white," except by comparison and prejudice.

We falsified the theory of race at great cost in the last century. Yet BLC builds on it, and amplifies it. That makes it a dangerous cult, both in itself and in the way it enables others' destructive tendencies.

Which brings me to our third cult, the reactionary right, or RR. It is arguably the more dangerous of the three. The RR claims to address the damage that the 3WF and BLC do, yet it is profoundly racist, and just as divisive.

The 3WF and BLC have left a huge mass of ordinary men and women without voice in the debate over where our society should go. The RR finds this mass of people, and speaks to their vulnerability. It attacks 3WF and BLC eloquently, and without fear. Whereas most people shirk from confrontation with armies of sock puppets, RR loves the confrontation.

In general the only people who seek out psychopaths to argue with are worse psychopaths.

RR attacks the "gender is a social construct" myth of 3WF and then attacks "race is a social construct" in the same breath. This may work for some listeners. To me, the switch from exposing intolerance to embracing it is shocking. Yet it's systematic, and often focuses on immigration and refugees.

After the arrests in Brussels of a terrorist, days before the attacks, the RR was claiming that "200 youths attacked police as they arrested an Islamic terrorist," in Molenbeek. The RR claim that Muslims in Europe fight for a Caliphate. They happily mix "refugee", "immigrant", "terrorist", and "Muslim" as if these mean the same thing.

I think, though it's not yet clear to me, that the RR are trying to push people towards political extremism. That is, while 3WF is trying to control media and technology, and BLC wants to control education, the RR wants real political power. Donald Trump's bid for Presidential office is entirely in line with the RR world view.

The Role of New Media

The 21st century media is a different animal from old media. Gone are the careful, slow investigations and analyses. In their place we see click races and catch phrases. After a significant court trial such as that of Jian Ghomeshi, they prepare two pieces, one for each outcome. Each is designed to trigger maximum outrage. Seconds after the announcement, the pieces go live, and the tweets flow.

Twitter is the mouthpiece of New Media. It takes outrage and multiplies it a thousand times. Every person in that human chain gets a small kick as they Retweet and Like. There is no responsibility.

And there is no dialog. We get two sides shouting slogans at each other, each seeking for the most outrage they can express without being blocked and banned.

In the fight for clicks, all dialogue dies. In its place we get emotion, and appeal to emotion. In Ghomeshi's trial, the judge said (I paraphrase):

The witnesses in this trial lied so often that their entire testimony was worthless.

Which became, on Twitter:

The judge said, "one cannot believe these rape allegations"

Which a few dozen retweets later became:

The judge said, "one cannot believe rape allegations"

Which 3WF picked up on and has turned into a new crusade against the inherent evil of maleness. The judge will probably be retired, for doing his job properly, and Ghomeshi will never clear his name. Check the #ghomeshi hashtag if you can stomach it. Only a few commentators ask, "what if the accusers really were lying?"

Asking for thought and pause is asking to be ignored. 3WF does not need trials or evidence to know that maleness is guilty. It is a core tenet of the cult: every man is a rapist, it is a question of "when", not "if". The effect is to keep its members in an echo chamber of hate, isolated from balancing opinion.

Ironically, Ghomeshi was a long-standing member of the 3WF cult. As a public male figure, he was Fair Game, though. The use of sexual misconduct allegations against prominent men is a standard technique. It often follows the same pattern:

• Loud accusations published in the press.
• Public outrage and condemnation.
• Investigation and conclusion: the allegations were false.
• Accused's life is ruined, and accuser claims the system failed her.

Real victims do not take this route. They mostly say nothing, out of shame and self-recrimination. Then if they speak up, it is to the police, who mostly do nothing because such cases are so hard to prove. Victims may press civil complaints then, where the standards of evidence are lower, and the punishments less severe. Only when their case goes to trial, does the press learn about it.

Those who use the media as their court tend to be liars. A person with evidence keeps it in reserve, and speaks through their lawyer.

I'm forced to conclude that Twitter promotes liars. Worse, it promotes those who make a career out of lies. That is, psychopaths. Let me say this again: Twitter promotes psychopaths. It gives them power and status and credibility.

Gamifying the Truth

I started building a Twitter profile some years ago. It all seemed innocent at the time. Get more followers, it's a good thing, right? Twitter let me promote my books, my writing, my talks. It was a way to keep in touch with people.

And then there's that moment when you realize you're talking to an audience. Not friends, not colleagues. Rather, a group of mostly strangers. And if you say things they don't like, they'll walk away. You say the right things, and Twitter rewards you. New followers! Be too blunt, and people leave.

The problem with performance is that it rewards drama and emotion. This is fine for entertainment. It is toxic for dialog, for understanding. Combine Twitter's karma whoring model with its short text limit, and the result can be horrid.

Horrid, that is, for those who seek truth and knowledge. The only way to get that is by careful argument of theory. Even then we only approximate. The process is everything. And Twitter promotes a process of anti-science.

There are some specific things Twitter could change to fix this. It would break Twitter as we know it, and thus will not happen:

• Remove the size limit on posts, or allow additional detail for a post.
• Hide the number of followers and retweets to stop karma whoring.

The Need for Offense

The particular thread that kicked my brain into action was about (not) banning a conference speaker who holds racist views. I'm going to make an exception, and link to some of these tweets. Note the language and style used.

Conference organizers are of course free to invite whom they like. Notable about this particular event was that it has rules about how speakers are selected. These rules, a contract, exist specifically because 3WF demanded them. Papers are stripped of gender and identifying information, then selected on purely technical merit.

So the demand for selective enforcement is interesting. This is a classic cult technique: complex rules that apply in one direction only. The psychopath steals from everyone, yet is the first to demand harsh sentences for others' petty crimes.

Yet no matter the agreed rules, the demand to ban a speaker for their political views is indeed a call for censorship. This is text-book censorship: the suppression of speech which is inconvenient, sensitive, or politically incorrect.

What's wrong with some censorship, you might ask? It's much like asking, "why do we need due process?" Surely censoring obvious hate speech is a good thing? Surely sending obvious rapists straight to prison without the cost and delay of a trial is a good thing?

People who believe this (and there are a lot) have a common culture of lazy safety. They have never been arrested and locked up on false charges, by cops needing to make their quota. They have never been falsely accused by vindictive ex-partners. They have never lived in dictatorships where fighting the regime was classified as "mental illness." They have never known a world where some knowledge was considered so dangerous, you could die for spreading it.

Once again, the loudest demands for rough justice come from the pampered children of the upper middle class, who believe the cornucopia of technology is their birthright. Who piss in drinkable water and think nothing of it. Who see an empty fridge as "lazy" and not "poor."

Since it is clearly not obvious, let me break down the reasons why censorship of Truly Dangerous Opinion (TDO) is bad:

• It creates a black market in TDOs. Like prohibition of alcohol or drugs, this makes TDOs harder, not easier to fight. The censorship of Nazi propaganda in Germany since the end of WW2 did not stop the underground neo-Nazi movement. When you bring TDO into the open, you can argue it and destroy it.

• It creates a censor. And the censor always has their own, plastic opinion of what TDO is. If the censor is honest and sincere, it can be hijacked.

• We need TDO to argue against. TDO is the Devil's Advocate. When someone advocates slavery, this gives us a chance to better understand and attack modern day slavery. When someone advocates racism, this forces us to argue for multiculturalism.

As John Stuart Mill wrote in, "On Liberty":

The peculiar evil of silencing the expression of an opinion is, that it is robbing the human race; posterity as well as the existing generation; those who dissent from the opinion, still more than those who hold it. If the opinion is right, they are deprived of the opportunity of exchanging error for truth: if wrong, they lose, what is almost as great a benefit, the clearer perception and livelier impression of truth, produced by its collision with error.

Censorship is a goals of all cults, because it gives them the Holy Grail of informal tribunals with unwritten rules. You'll see this demand made over and over: "give us power to enforce our opinions over others". It is really a call for dissolution of the State, and the handing of power to self-selected individuals. A form of anarchy where the strongest rule over the weak. It is the psychopath's wet dream.

Conclusions

It has become clear to me that Twitter is a lens that distorts. The economics in this case favor the wrong people: the psychopaths and their followers. It gives them space in which to grow, recruit, and market themselves. By using Twitter, I'm contributing to this.

We know where the politics of extremism takes us. We don't need to go back hundreds of years. Just a few days, to bombs exploding in my home city. Psychopaths produced those bombs, recruited the mules who carried them, and sent them on their way. One more cult of hate, seeking to divide and conquer.

Yes, I'm literally making an equivalence between the digital cults and Daesh. They lie at different points on a scale, yet it is one scale. Saying "all men" and "all white men" has the same intent and effect as saying "all Muslims." It divides, isolates, and creates space for the cult to grow.

Never doubt the harm that psychopaths will inflict on others. It is simply a pragmatic calculation of benefits against risks and costs. Allow cults to grow without sanction, and they will become more aggressive and more destructive.

I'm in no place to stop these cults or even reason with them. However I can stop contributing to them. That means, stopping using Twitter. It's a painful decision to abandon years of building up a following. And yet it frees me to speak openly again.

So, I've deleted my account. Since literally deleting it confused some people, who thought I'd blocked them, I have removed my profile, and stopped tweeting. My old tweets remain, and people can message me.

There are other ways to talk with people. Other platforms that don't tilt so heavily in favor of extremism. Platforms where we can write more than a sentence. None is perfect. Yet some are better than others.

Thanks to everyone who supported me during the long years. I'm not walking away, just finding a better place to stand.

by pieterh

Background to Trademarks

Definitions first. A trademark is a name, phrase, logo, or even a specific color (the "mark") that you're using for business ("trade"). The simple fact of using a mark for some period of time establishes the trademark. However as with all property, the devil lies in enforcement. The question is, always, if you go before a judge with a complaint, what standards of evidence will the judge expect and demand?

No matter the case, criminal or civil, it always comes down to convincing one or more humans. If you ever go to court, keep this in mind. The facts of a case, as each party knows them, are irrelevant. How those facts are documented and presented is all that matters.

Let's back up a little and ask why courts even care about protecting businesses' trademarks. First, it's to protect consumers from misleading sales tactics. Just selling junk isn't an offense as such, except when there are legal minimum standards for health and safety. However selling junk that claims to be a more expensive, well-known brand is an offense. So secondly, trademarks let businesses distinguish themselves and stop unfair competition.

So the judge in a trademark violation case will ask, "Was the intent to deceive the consumer? Would a reasonable consumer be deceived?" And then the judge will ask, "Who owned the trademark, and can they prove it?" Even though the simple act using a mark creates it (under so-called Common Law), that can be hard to establish.

For instance, business A creates a chain of restaurants. Business B opens a competing chain using the same colors and similar name. B is clearly hijacking A's investment in branding, stealing goodwill. Yet when A takes B to court, B produces a document showing their restaurant plans, a full year before A started. How does the judge know who is the liar?

In clear cut cases, you can convince a judge that a copycat is deceiving consumers and stealing your goodwill. Yet the risk of losing such a case is high. It's also costly for courts to deal with such cases. Judges may simply refuse to hear them.

Hence most countries provide a way to register your marks, for a fee. Registration gives you a dated document that establishes your claim to the mark. The trademark office does the job of searching for prior marks in the same area. Before it grants you the registration, it publishes your claim and gives others a chance to dispute it. So after a search, and if there are no disputes, a judge will take the trademark registration as solid evidence.

It is not that simple. A competitor can still claim that their Common Law mark outweighs your registered trademark. They can argue that the registration does not represent real goodwill. This is often understood as, "if you don't enforce your mark, you will lose it," which is inaccurate. As trademark holder you're not expected to police the world. However you are expected to be truthful in court when the judge asks you, "are you using your mark, and suffering real damage due to the unfair competition?"

Finally, courts consider trademarks to apply per segment of the market. So you can have XYZ Car Co, and XYZ Clothing Co, with no confusion to the market. When you register a mark you'll need to explain what "classes" you're using it in. You'll probably want international class 9, which is anything that beeps.

Where and How to Register

If you are large enough to need to register in multiple countries then you are large enough to have trademark lawyers. For the rest of us, it's a bit like buying a domain name. Sure, there are hundreds of domain extensions. Yet we still want a dot-com for our main business.

So it is with trademarks. If you decide to register a mark, do it in the US (via the USPTO) first. That's cheap, and simple. Then over time you can register in the EU (via the OHIM), if you find your project is worth it.

The cost for a US registration is around USD 1500, depending on what lawyer you use. You can find trademark attorneys on line. They'll ask you for details of the mark, proof that it's being used, name and address of the registrant, and credit card details. The process takes about six months. After nine years (and before ten years have passed) you can renew the mark.

Getting a US registration will speed up registration in other countries, if you decide to apply for that later. The risk, and it's a small one, is that a troll will register your trademark in some other country, effectively excluding you from doing business under that name, there.

Before you register, however, ask yourself "what is the chance someone would rip off my name and logo?" If it's low, don't bother. If it's high, then ask "what is the chance a cheat would take this to court?" If that is still low, then don't bother either.

Instead of registering a mark you can raise its visibility. This means being explicit on your website and other materials. "X, Y, and Z are trademarks of MyCorp." This scares off potential cheats, improves your case, if you do try to defend the mark in court, and makes it easier to get registration if and when you need it.

How to Enforce your Trademark

Registered or not, you enforce your mark by telling the other party, in writing, "stop now, or else." If they do not stop, you repeat the warning, with initial claims of damages. If they do not stop, you add on more damages and when you have a solid file, you take it to court.

The vast majority of people will back-off at once. The trouble is when you face someone who's well aware of trademark law, has cheap legal resources, and enjoys time in court.

If you are facing such a firm, and you did not register your mark, you should probably fold your hand, and change your name. The risks are high that you would lose, and have high legal fees and possibly damages to pay. Judges don't always get it right.

If you did register your mark, then you should push ahead and claim damages. You will win, if you stick to the basic rules (you're still using the mark, the damages are real.) Do I need to say, any court case will have to happen in the country of registration? Judges in Belgium won't accept paper from the USPTO.

Trademarks For Open Source Projects

The common misconception about open source is that because the code is free, it does represents no property nor value. The opposite is true: successful projects represent considerable value, owned by many. How does a trademark represent and protect that value?

It comes down to authenticity and reputation. If you download a package calling itself "XYZ v2.0", then you may have expectations. It is compatible; it works; it has no trojans or advertising; it is from the same people as "XYZ v1.0".

If a successful project does not register its name, then anyone can fork it, repackage it, and use the same name. Imagine competing, incompatible versions of "Linux."

When a person or a business registers the name as a trademark, those incompatible forks may still exist. However they may not use the mark. If they try to do that, it's damages time.

I've had this happen at least once in my own projects, and the trademark was the tool I used to stop the incompatible forks and punish the perpetrators. Trademark law is clear enough that saying "trademark violation" will stop 99% of cheats dead still. Producing a registration filing number stops 99% of the remainder.

Conclusions

In a serious project like ZeroMQ you'll end up with three or four marks you want to register, over a period of five to ten years. Register only when it's worth it. That is, to protect real trademarks that you would be willing to defend in court. Consider that in the worst case you might have to spend ten or twenty times the cost of registration, to defend your mark. You might get that back, or you might now.

I hope this small brief has helped you understand trademarks, and how to use them (or not) in your open source projects. And, if someone claims you're infringing on their trademark, how to defend yourself. (Hint: ask them for a registration number.)

by pieterh

It was about four years ago that the ZeroMQ community ejected its main contributors, who forked it as Crossroads, and then rebooted that as Nanomsg. Our motivation for that coup was almost exactly the pain and angst that Drew describes in his article, which we'd suffered for years in ZeroMQ.

The contract that binds the ZeroMQ community is, increasingly, not a technical one. It is C4.1, RFC 22, which has as its core principle:

Everyone, without distinction or discrimination, SHALL have an equal right to become a Contributor under the terms of this contract.

If you send a correct patch (as defined by the RFC), it shall be merged. Period. This simple, brutal rule had several effects:

• It silenced the arguments over ideology that used to pollute zeromq-dev.

• It removed the discriminatory "I don't like you so I don't like your code" power of dominant maintainers.

• It opened the door to new waves of contributors, who have turned ZeroMQ into a much larger and more aggressive collective project.

RFC 22 is revolutionary. No other project has defined the rights of contributors in such a way before, using the language of contracts and protocols. It established a bill of rights for contributors that focused on recruiting contributors. As Drew writes,

Open source projects hinge entirely on contributors.

The main argument against inclusive diversity is a tired and clichéd appeal to exceptionalism. "I'm smarter than average so I write good code, alone," is the way it usually goes. It is provably wrong. Good code is code that survives. This is how the market works, there is no other basis for value judgments on code except "people use it, people invest in it, and it survives"

Nanomsg was designed to fail. There are powerful lessons here that anyone starting, using, or contributing to an open source project can profit from.

Let me list the vulnerabilities, if they're not already clear to everyone:

• The lack of a bill of rights for contributors, allowing the maintainers to discriminate.

• Starting with a core maintainer who had a long record of working poorly with others. In 2011 we had four incompatible versions of the core ZeroMQ library (2.x, 3.0, 3.1, 4.0), due to endless and chaotic changes coming from Martin Sustrik.

• The use of a MIT license, which encourages "dark forks," as Drew calls them.

Mix this together and the results were predictable. In ZeroMQ we had the same ideological hostility towards contributors, which we massaged away with endless diplomacy. Much of my time went just to that, for years. The license made it hard for people to create dark forks. So, we worked hard to get consensus, and mostly we succeeded.

The lessons here are clear:

• First, please do not use a leaky license. MIT/BSD is not "business friendly," so much as "community hostile." MPLv2 is broadly accepted by businesses and discourages dark forks.

• Second, please use a strong bill of rights for contributors. Use C4.1, as-is, and you will avoid the pain that Drew and others went through. We learned this lesson in 2012.

• Third, when things hurt, stop and fix them far earlier. We seem to be so patient with pain. Learn to recognize that pain is not a good sign.

In conclusion I'm going to quote Doron Somech: "Crazy Idea: Clone nanomsg, move to zeromq organization, relicense as MPL, support ZMTP, only new socket types and expose CZMQ API."

by pieterh

A Prayer for the Dying

Let me start by saying I've been a fan of GitHub ever since I met a "GitHub guy" at a GitHub drink-up in Brussels. His job seemed to be to write code, and travel around the world drinking with other developers. I fell in love with company culture immediately.

Over the years, GitHub never disappointed me. Their platform was stable and reliable. Their UI remained minimalistic and as smooth fitting as a velvet glove. We switched our issue tracking from the venerable Jira to GitHub, watched with joy as it integrated with Travis and AppVeyor, and felt cosseted and safe with a firm that worked and thought as we did.

I've often turned to GitHub support for help. They have always been impeccable: fast, accurate service. It is unheard of for a web business. Obviously the staff loved their firm, loved their work, and loved their users. And we loved them right back.

Then in 2014 the drama started. It began with one recent employee making claims of harassment against the CEO and his wife. It turned nasty as anonymous rumors spread of this employee's manipulative behavior in the firm, and her public tweets about being blackmailed towards sex with the CEO. The employee produced no evidence to back up her claims.

The founder and CEO resigned, his co-founder took over, and a new wind started to blow in the firm. A VP of Social Impact was hired. Middle management started to creep in. These new managers began to crack down on remote working. The old guard started to leave.

This story goes far beyond a fight between Social Justice Warriors and the Patriarchy for control over our Open Source dollars. This is not simply Death by Code of Conduct, though the firm did adopt a controversial CoC that disturbed not a few of its users.

I've written extensively on living systems and power pyramids, and the war between these two organizational models. GitHub was a rare example of a successful living system growing out of the fruit basket that is Silicon Valley. It quickly became dominant in its market. It was profitable, free of debt, growing, loved.

I've learned one thing from being an entrepreneur for 35 years: when there is unguarded money on the table, the wolves arrive. Call these as you like, their goal is to take the cash and blame someone else. They always use the same strategies, which I've documented extensively in my book The Psychopath Code. Divide and conquer. Charm and distraction. Promises and lies. Stealth, and violence.

GitHub is a treasure. Its current valuation of $2,000,000,000 is just a baseline. It is worth ten times more, at least, if handled right. What it needs to justify a $20bn price tag are three things:

• The right structure. That means a power pyramid instead of this strange "horizontal" network.

• The right size. 300 employees is too few. Yahoo! has 12,500 employees, and is worth $33bn on the stock market. GitHub needs at least 5,000 employees.

• The right product. The minimalist GitHub we open source developers know and love does not appeal to corporate clients.

The first two changes are already happening. I predict the third will start to appear over the next six to 12 months. We'll see acquisitions of firms like Travis, expansion of core features like the issue tracker, more sophisticated organization management, and so on. If you want to see what GitHub looks like in two years' time, look at its traditional competitors.

GitHub will become more profitable. If I had shares in the firm I certainly would not sell them yet. Wait a year or two, until the VCs and new managers find their acquisition target. Wait until the exit, and then sell. After that, it's over for the firm, and those who have stayed long enough to make their winnings will be looking for a way out.

There are people who will question my framing and the time-line. Let me explain this as a murder mystery. We see a healthy person suddenly falling sick. We see a massive insurance policy being taken out. This person's demise is going to be extremely profitable. As investigator, you assume malice. You assume there exist people capable and willing to do anything if the price is right. You look for motive, means, opportunity. You follow the money.

Look at the timing of VC investment, hiring of new employees, internal political fights, ousting of old CEO, and hiring of new managers. Look at a climate in which political outsiders use the weapons of gender and race against meritocracy. What emerges is the picture of a slow fight between the owners and investors, and the founder/CEO. The owners want their billions. The founder/CEO wants to keep GitHub alive.

Unfortunately our data is incomplete. People are unwilling to speak publicly about what happened. We rely on claims and anonymous counter claims. Only time will tell.

I wish I was wrong. Yet we are seeing large-scale interference in a successful, and essential platform. This is not idle tinkering to solve diversity issues. This is not idealism. We are watching a fight over money and power. No, that's wrong: there is no more fight. The founders of GitHub have already given in, one way or another. The fight is over. The users of GitHub won't fight. They will either accept, or move away.

If my speculation is right, what will the impact be? GitHub have established themselves as The Place to host our public projects. Surely losing this will be a major blow to free and open source. Well, happily, no. This is how it happens: we construct, and then when we reach a plateau, the wolves come along, destroy what we make, and liberate us to go forth and do better. Psychopaths are the garbage collectors of human society.

Whither the ZeroMQ community? I'm going to begin using GitLab for my own projects, and if that works, I'll aim to move the ZeroMQ projects that I care for, to GitLab, after discussion and consensus.

Thanks for all the fish, GitHub. And to all GitHubbers, thank you a million times. You did the impossible, and you did it right, and I wish you all the best for your next projects. Hugs. :)

Update

To my surprise, Julie Ann Horvath (JAH) commented on my tweet:

Julie Ann Horvath: @hintjens CEOs are very rarely (see: never) removed from a company without a claim being backed by substantial evidence.

So her argument is this: TPW agreed to step down, meaning he was "removed." The implication is that TPW admitted guilt by quitting, despite GitHub's investigation showing JAH's two claims of harassment were false.

JAH backs up her argument by stating as a fact that CEOs are "never" removed from a company without substantial evidence. Thus, she implies, there was in fact substantial evidence against TPW. Note that she doesn't actually say this, not has she ever produced such evidence.

From my own experience, CEOs and other managers tend to fight with shareholders and boards, and it's this politics that forces them out. The role CEO is always temporary, annually renewable. Fired, or step down, it's much the same. Anyhow, let's look for some research on why CEOs get fired. Bingo: Forbes reports on a study of 286 organizations. Nothing about harassment. Rather:

My team and I interviewed 1,087 board members from 286 organizations that fired, or otherwise forced out, their chief executive. And we found that most CEOs get fired for “soft issues.” Thirty-one percent of CEOs got fired for poor change management, 28% for ignoring customers, 27% for tolerating low performers, 23% for denying reality and 22% for too much talk and not enough action.

I tweeted this to JAH.

Pieter Hintjens: @nrrrdcore that is false: study of 286 organizations showed majority fired for "soft" reasons: http://www.forbes.com/sites/markmurphy/2015/07/16/leadership-styles-are-often-why-ceos-get-fired/#21a2d28a6155

JAH dismissed this study using the "true Scotsman" argument:

Julie Ann Horvath: @hintjens in public companies, perhaps. Rerun those numbers against privately held/vc backed startups.

I pressed F5 to "rerun" the numbers and got exactly the same results. I'm not sure what else JAH would expect, as I did link her to the Forbes site. Nowhere does the article say it looked only at public companies. Nor is there any logical reason it would be different in a private firm from a public firm. I'd expect the standards of evidence to be higher in a public firm, if anything. I suspect she didn't RTFA.

Here is a telling quote from Leadership IQ's the original article:

"A more accurate explanation for why they get fired," he added, "is that the Board of Directors or shareholders have lost confidence in their ability to generate sufficient financial returns in the future."

In other words, it's about money. And my hypothesis that TPW was pushed out in order to turn GitHub into a cash cow still stands.

Undeterred, JAH now attacks my motivations, instead of providing any real information:

Julie Ann Horvath: again, you're reaching to make an argument that suits your interests and currently held position.

What position? I'm a writer. My position is, I write what I see. My interests are truth and knowledge. There's no profit in this. Maybe JAH is projecting her own viewpoint.

Then JAH continues in a flood of tweets, presumably playing to her audience:

Julie Ann Horvath: @hintjens also, I don't identify as an SJW, I'm merely a tech worker who happens to be a woman.

Did I call her a SJW? No, I said the situation at GitHub "goes far beyond a fight between Social Justice Warriors and the Patriarchy." If anything I was saying, to the Reddit crowds, don't read this as a classic SJW drama. It's not.

JAH continues, to applause from her admirers, I assume:

Julie Ann Horvath: @hintjens I'm vocal, yes. But please don't label me in a way that's been known to incite violence against women.

Ah, here is the payload. She implies (no accusation, it's the implication) that I've placed her in physical danger by labeling her. "Violence against women," seems oddly specific coming from someone who doesn't "identify as an SJW." Why not simply "violence?" Is there a specific form of violence that… ah, yes, men the aggressors, women the victims. "Don't call me SJW, that's so cishet!"

Julie Ann Horvath: @hintjens all I'm asking for is the same respect you would award your own peers or employees.

Well, for a start I have no employees, and any peer who made unfounded accusations of sexual harassment would get the same treatment from me. This has happened to friends of mine, male and female, and my first response is: "do you have evidence, and if not, how can I believe your word over theirs?"

If someone abuses, harasses, or threatens you, get evidence any way you can (pull out your smartphone!) and then proceed.

There is a classic pattern here. Woman claims sexual harassment from powerful man. Man denies it. Woman goes to social media and makes her case in court of public opinion. Man cannot disprove a negative. Woman gets crowd of sympathetic admirers. Some may hate men. Some may feel guilt at their own misdeeds. Some may hate what that powerful man stands for. Some may just like being part of a mob.

The court of public opinion is not a court, and street justice is not justice. It is above all, the vulnerable and weak who get abused in such systems. We worked hard and long to build justice into the State.

If you are harassed or abused, you do not shout about it. You take it to the police and if they don't act, you still have the choice of a private lawsuit. This is what courageous women and men do, when they are abused. What JAH did was something different. If you go to the public with a private matter, you lose your right to privacy.

JAH insists on dodging the reason we're exchanging tweets: her public allegations, and her failure to produce any material evidence.

Julie Ann Horvath @hintjens I hope this conversation helps you better consider the implications of statements like these in the future.

Well, my statements stand, and now I have more to back them up. Talking to a professional victim always leaves one with a nasty, unpleasant feel. There's no honesty there, just a shadow person who is good with charm and words. See, I didn't made any accusations. Just the implication. Neat, isn't it?

Julie Ann Horvath @hintjens and I also hope you have an awesome week 😊

Sweet. I'd have expected a little more backwash from 24.6K followers, yet there we are.

I then stumbled on Daniel Tennner's article on GitHub. This is worth reading. A few choice quotes:

These are all very clear signs of an open culture that’s being ripped to shreds. It’s tragic. It’s also terribly ironic to see the old bullshit excuse that “well, it just doesn’t work at this scale” being deployed.

And:

When did the dream start dying? Who knows. But I have a hunch that it started before Tom Preston-Werner was ousted from the company, a couple of years ago, in a case of sexual harassment that was later dismissed as groundless. This smacks me of political play, a scenario very similar to what played out at AES, where an unrelated incident is used .. to weaken, attack or destroy it. GitHub was an open culture at the time. What did Tom do there? Why was it important to get him out?

This story isn't over.

Update 2 (2016/02/25)

GitHub hires another political commissar, this one famous for enforcing political correctness on open source projects. The combination of charm and deniable brutality is familiar.

by pieterh

I'll publish the book on gitbook as I've done for my previous books. The raw sources are on https://github.com/hintjens/scalablec. Send me pull requests, if you have fixes.

Update, Sunday 17 January

by pieterh

About Me, Really

I don't often write about myself, except when people demand it. I'd much rather people looked at my work than my biography. Still, some stories are worth telling. One of them is the cold horror of your own body trying to kill you. Cancer is scary. A close friend asked me not to kiss on the cheek, as we do in Belgium. "I don't want to risk getting it," she said. "You know I've got a young baby." We still speak, though less often than before.

How does one survive cancer? It's simple and unfair: one does not die, yet. We all die eventually, which is a good thing. Death is the Great Garbage Collector. Yet we go through extraordinary effort to not die, yet. For me, that meant letting my Congolese surgeon Dr. Mbote and his team slice me open, remove half my pancreas, lymph nodes, duodenum, bile duct and gall bladder, and lower stomach valve, and then reconnect the pieces in a Rube Goldberg operation lasting twelve hours. Google "Whipple procedure" if you want details.

I woke up wrapped in the evil ancestor of all pain. I can't describe it. Near my hand was a small trigger to push, to get a small dose of pain killer. Normally, ten or twenty times in a day. I pushed it over a thousand times the first few hours (they told me, later). I'd have screamed every time I moved, except the tubes in my nose and throat made that impossible.

The pain and thirst (the first two weeks I could not drink a drop of water) were bad. Worse was the waking hallucination when they tried a stronger pain killer, morphine I think. There was heavy rock music, and this animated vision of Hell in a striking flame and blood palette. Nothing I have ever seen in the darkest corners of the Internet will ever compare. When you are trapped and confronted by horror, you tell yourself, "it will pass." And so I counted the seconds, one by one. And it passed, as it always does.

I like to think a little bit of Hell came back with me. I got stronger, came back home, looked after myself as body and digestive system rebooted. Then I went straight back to work, popping in to the chemotherapy clinic in Brussels once a week for my dose of poison. One gets used to needles. I found a client in Dallas and started a year and a half of non-stop travel.

Some clients are nice to work with. This one was not. I protected my team, carefully built up our product, and bullied management into behaving. It ended with us shipping out to South Korea, opening an office right outside the client's campus, and wildcatting our way through the vicious politics. We did a good job, and in the end they paid all our invoices.

At which point I decided to stay at home and look after my young kids. They were glad to have their dad back. And I became mom too, a single parent. And every day, in the back of my mind, "this could be my last month, or year."

Is my cancer gone? I cannot know. I can only know if it comes back, in my blood, in my bones. I do the scans, every six months, every year. "Nothing," they tell me. "Nothing, yet," they think. I still carry the port where they'd stab me once a week. "It's not really worth removing," says my oncologist. "maybe next year." That pain in my shoulder, not going away. Is the cancer back? No, it's damage from too much drumming.

What would you do with your life, if every month or year could be your last?

For me, it was raising my kids, writing, going to conferences, and finding clients who would let me work from home. None of these are easy. Yet they are just a matter of focus, and determination.

Writing books is like writing articles, just more. Life hits us with questions. We try to answer them. My way of thinking about complex questions is to write. I think I'm good at this, and reading my work from years ago, I find myself thinking, "nice, this person got it."

Perhaps it's the self-flattery of the wanna-be expert. I hope not. Some of us think as part of the crowd. We adopt the thoughts of others, and we rarely step outside predefined boundaries. We are bricks in walls and we feel good there.

I've been outside the walls, a foreigner, all my life. We live in Molenbeek, the terrorist capital of Europe, because in a city of immigrants, this commune of the undocumented feels most like home to me. All my professional life I've worked to overthrow existing systems. Is writing code an act of revolution? For me, yes. Deliberately so. I want to smash the Wall, and its self-righteous mistreatment of the less powerful. I dream of collisions.

A project like ZeroMQ goes beyond "disruptive." It is about taking the old rule book, burning it, and peeing on the smoking remains. Not just rules about commercial software. We also rewrote the rules on how to make open source. Activist communities can be as abusive and arrogant as any company. Worse, in many cases.

Writing, also, is a revolutionary act. I'm mostly a polite writer, because I think manners are important. We don't need to insult or diminish others to disagree with them. And politeness lets us learn from those we disagree with. Yet when I write I am remorseless. Why pull a punch? Better to be wrong than to be silent.

I'm a traitor to my "race" and gender and culture. These are boxes to contain and divide us. The lure of privilege was always there, growing up in a diplomatic family. We had servants and mango trees. Yet my first language was Swahili. I was always outside the walls, making friends with those who had the least and who shone with decency.

The debt of privilege is massive, yet it is like any debt. You repay little by little, as you can. You never forget it, and you never justify it. For me, paying off that debt meant returning to Africa many times, to work there, to make life-long friends in Togo, Lagos, Ouagadougou. I consider myself the lucky refugee, blessed with opportunity, and able to invest that back in others.

It meant working in the public good. My first free software project dates from 1991. I spent years working unpaid to fight against software patents, and for open standards. Collisions: we challenged billion dollar institutions and forced them back. This was no guilt trip: the slow, deliberate repayment of the debt of privilege was always a pleasure. And one learns, one learns.

I refuse to take sides. Predators come in all genders and colors, shapes and sizes. The bulk of decent people also come in all genders and colors. For every twenty-five Alices and Bobs, one Mallory. The abusers love to divide to conquer. An ethical system gives all participants equal opportunities and rights. And it brings people together instead of creating divisions.

So call me an atheist humanist if you want a label. I hold that every person contributes something, even the "worst" among us. Even through war and destruction, we learn how to move forwards as a species. We live, or we die, as one species. There are no race, gender, or culture wars. These are illusions that deceive and divide.

Our forever war, as a species, is against ignorance and magical thinking. Only accurate knowledge can secure our future. There is no price tag for this. If it takes blood and pain to learn, then we pay in blood and pain. The little piece of Hell on my shoulder tells me (metaphorically, I'm not delusional): "no-one cares, you mean nothing, you're just a meat machine. So work, now!"

Which is why I write. Capture problems, try solutions, write it down in plain language, and invite criticism. Trial and error, and no matter how much people are offended, keep going.

The books: three so far. Each takes years of research, and six to nine months of writing during which unpaid bills pile up. Through every book I've learned a lot and aim to capture these lessons for others. The Psychopath Code was the hardest book to write, technically. I'm not a psychologist by training. Yet the book is solid and accurate. It is a book I needed, and found nowhere, so had to make.

When you express yourself, you invite attack. It is an essential part of the process, to stand up and be ridiculed. Here's a dismissal of my book (from Twitter, now deleted): "I know more about psychology than this guy, and my dissertation was in psychopaths! I should critique his book… but… no."

Young person, if you know so much about psychopaths, why aren't you out doing the same research I did? Are you talking to victims and psychopaths about their experiences? Where's your research on predator-prey models and the relevance for humans? How about the profit-loss accounting of abuse? Why aren't you describing how the abusive bond works? And critically, how to break it and get out with our minds intact. We need to know this stuff.

Enough people studying this material, for decades. So many experts. And really, so many excellent studies and papers. So why did I need to write that book? Institutions, that's why. The more you know, the less you see. Live outside the walls, and you get a different and broader perspective.

I have a profound lack of respect for institutions, their walls and bricks; their protection of ignorance, their fear of originality and real diversity of thought. Their narrowing of focus, and their dishonest reverence for authority.

There are good institutions, such as the medical system that saved my life. Yet even sick and dying I was able to diagnose myself faster and more accurately with my mobile phone than the doctors could. "It's not a gall stone, so can we do a biopsy already, please?" "Yes, I'm getting allergic to the pain killer, that is why I'm vomiting and ripping out my stitches. Please stop giving it to me."

It wasn't simply careful decision-making. It was slowness, and it almost killed me. When you have a fast-growing cancer, an extra week to make a biopsy can be fatal.

The more we invest in structure and organization, the stupider we become. The smartest teams are small and dynamic and focused on solving problems through trial-and-error, not debate. Let alone the common pattern of "you are an idiot, and you are wrong, and I will tell you why." Or, god help us, politics and the thirst for power.

I want to say "thank you" to the hundreds, thousands of people I've met and spoken with over the last years. You've been wonderful, and given me so much. Most of my pre-cancer friends drifted away, after I didn't die. I recall one seeing me at a conference, going pale. "You're not dead!" Indeed, no, I may be bald, but I'm not dead. Yet.

Five years seems worth celebrating. I did a lot these last five years: paid off many of my debts; became a full-time parent to my kids, who became happy bundles of love and laughter; rebuilt a business; wrote some heavy, solid books and over a hundred articles; taught myself piano and started composing; and saw most of my projects thrive and survive.

Don't wait for life to come knocking. Go out there, with love and optimism, and make good things happen.

The constant silent threat of recurrence defines my view of my work, and my life. Every one of my articles, books, projects, tweets, and patches is the last. Every time I leave someone I care about, I tell them how much they mean to me, hug them if possible.

No exceptions, no false drama. Every meal I eat is the last one I'll ever taste. Every song I play is the last one, before cancer drags me back to an iron bed and those damned beeping pumps. My music isn't sad, it's just always saying "adieu". These are the rituals of the survivor, yet they are a powerful tool. Try it, one time, to make every song your last.

And the weirdest part was the joy at being alive. From that moment five years ago, except when I was shuddering from the pain, I was euphoric. The sheer ridiculousness of not dying, yet, wore off after a year or two. The pleasure at being alive and surrounded by lovely people (twenty-five to one, I remind myself) sometimes spills over.

Sometimes I think my cancer saved my kids and me from a terrible alternate time line. Then again, I hope one doesn't need to go through cancer to become a happier person. If you can be truly thankful for the last five seconds of your life, you don't have to be thankful for the rest.

Here now is the payload of this self-indulgent article. Five simple wishes to make the Internet a happier, better place. (Now you can call me delusional.)

More Manners

We seriously lack manners, on-line. It's as if being rude to total strangers suddenly became acceptable somewhere around 1998. It isn't cool. If you're one of those people who like to use clever adjectives to denigrate people you disagree with, realize this: it makes you look bad.

Less Arrogance

"I've studied X so your opinion on it is obviously wrong," is a logical fallacy. Claiming expertise because you lack originality is easy, and weak. Go out there and make mistakes! Learn by going beyond what you get second-hand. Solve real problems with new crazy theories, and don't be afraid to share them.

Calm It Down

"I don't agree with what you say, so you're toxic," is a common yet useless pattern. It infects open source ("you made a bad patch so you're toxic") and open discussion. Most people are honest, self-correcting, sincere. Some bad actors, toxic, and damaging. Learn to tell the difference by measuring your pain.

Take Time to Think

Instead of reacting, act. Take time to read what people say, and try to understand why they say it. Everyone's perspective is a fact. Don't argue with facts. Argue against interpretations, or better, try to help others to fix their arguments.

Trust Yourself

You don't need to be perfect. Every day, take a risk, learn something new, and speak to someone who you'd otherwise ignore. If people are hostile to you, shrug and carry on. There are a lot of people on this planet. No matter who you are, or what you do, you are precious, and there are people who will appreciate that.

by pieterh

Update

I'm going to link to Stephanie Zvan's critique of this article.

Identifying the Problem

Unwelcome sexual attention, aggressive conversations that won't end, intrusive photo-taking, offensive language and behavior… most of us have seen or experienced this. The mainstream solution is obvious: tell people to stop it, or else.

Here is a typical statement by a large tech conference:

All delegates, speakers, sponsors and volunteers… are required to agree with the following code of conduct. Organizers will enforce this code throughout the event.

Many participants do not like such statements, and the discussions over codes of conduct can become heated, emotional, and ugly. In my work in group psychology, there's a rule of thumb I have developed: When two sides argue emotionally it is because both are working from false assumptions.

And indeed, I think the mainstream Code of Conduct model is based on false assumptions. The theory of harassment (let's call it "Model A") has these assumptions:

1. Anyone can be the harasser.
2. Harassment is a motiveless act.
3. Outlawing harassment will stop it.

All three of these are provably false. Bad actors do not respect rules. They habitually avoid sanctions, and reflect them onto others. You can read my article, "Ten Myths About Harassment" to see other related myths.

By accepting and then acting on false assumptions, we get divisive policy and argument. And I think the outcome, with division and ill-feeling, is worse than having no code of conduct at all.

Here is a more accurate theory of harassment (Model B):

1. A specific subset of participants (the "bad actors") are systematic harassers.
2. Bad actors harass as a persistent strategy for getting power, sex, amusement, etc.
3. They treat rules and laws as challenges to work around, or misuse.

The term "bad actors" is a convenient euphemism for sub-clinical psychopaths. When you understand model B, the picture shifts.

Let's take a typical scenario, and see how it works with both models. The scenario is: at an event, participants are drinking. Mallory makes an unwelcome sexual advance towards Alice. Alice feels very uncomfortable and moves away. Mallory insists, and follows her. Alice explicitly asks him to stop, and moves. Once again, Mallory approaches and tries again.

Model A explains it thus: Mallory was drinking and didn't realize that his behavior was causing distress to Alice. By explaining what it means to be decent, and enforcing that, he will refrain from harassing people.

Model B explains it thus: Alice was drinking, and became more open and vulnerable. Mallory saw this, and decided to exploit it. Her display of fear and moving away drove him to be even more predatory. He was trying to isolate her, so she would give in.

It's not that alcohol turns good actors into bad actors. Rather, it makes the good actors more vulnerable. This distinction is critical. This is the key problem: people in a foreign country, who may not be with friends, are easy targets. Instead of solving Mallory's behavior, we must solve Alice's vulnerability.

Mallory is always going to be there. In a group of 100 people, we will have a handful of Mallorys. In a conference with 1,000 attendees, we'll have dozens of people capable of harassing others, if the conditions permit it.

In my view, events must be radically inclusive. To exclude people on any basis except real danger to others is a poor strategy. It introduces real, major risks such as false positives. To be accused of a Code of Conduct violation, and then excluded from an event can be life damaging. Organizers wield massive power, and bad actors can turn this power on people they see as a threat.

Let me be explicit about this. If Alice complains to the organizers, she puts herself at even more risk. Mallory now appears to be calm and sincere. He accuses Alice of stalking him. He explains how she would not leave him alone. He looks vulnerable, gentle, and seriously distressed. Alice is angry, outraged, and emotional. Who do the organizers believe?

And Mallory comes in all genders. Harassment is not a gender issue. So while sanctions must be possible, the policy goal must be to make them unnecessary.

The second problem with exclusion of supposed bad actors is that it is prejudicial. Even bad actors add to the diversity of a crowd. This is hard to admit. When I cross a bad actor, my instinct is to push them out, reject them, and tell everyone what happened. Yet I believe that is the wrong reaction.

Rather, we must understand, and then negate and switch off their predatory behavior.

Bad actors are pragmatic, and driven by clear calculations of opportunity versus risk. So the solution is to remove the incentives and opportunities for misbehavior, and raise the risks of exposure. Then, bad actors will adapt and self-select. Either they will behave, or they will go elsewhere.

This removes the need for enforcement, and the risk of false positives.

Designing a Solution

As a parent, I cannot stop my children from being exposed to bad actors. They will inevitably, regularly, come across people who see them as potential targets. It will happen on-line, in the streets, in schools, at friends' homes.

One option is to keep them at home, or never let them roam alone. Yet that seems counter-productive in many ways. So instead, I teach them how bad actors think and operate. I explain what kinds of conversation and interaction are dangerous. And then I teach them what to do if such a thing happens. Find an adult you can trust, I say. Tell them what happened, call me, and wait.

I explain to my children how bad actors stalk the Internet looking for victims. I explain what grooming looks like. I teach them to never give their real name, age, or location to strangers on-line. Never to send pictures of themselves. Never to accept a statement from a stranger as true, no matter how convincing it is.

This is the model I believe we need for conferences and similar gatherings. Instead of telling people to be good, we teach people to stay safe.

Here are my specific recommendations:

• We teach awareness of the behaviors one should consider "bad." We speak to potential victims and bystanders, not to bad actors. Bad behaviors are those that bad actors use to profile, stalk, isolate, and pursue their targets.

• We teach organizers how to accept and process complaints. These cannot be anonymous. They must be discreet, with protection of privacy for both parties. There must be fair follow-up, with sanctions as needed.

• We teach organizers how to protect their guests from predatory behaviors. For example, in any space where the organizers offer alcohol, they should offer food and water, and a host who does not drink, and who patrols discretely.

• We formulate these as a written protocol that events can adopt. Rather than ask every event to re-invent the wheel, we can build standards that events can reuse, and improve over time.

Such a protocol will, I'm arguing, create safe spaces by design. It will speak to bad actors and say, clearly: you are welcome as long as you act decent. Otherwise, we are watching, and we will catch you.

A Code of Conduct Protocol

This is a raw proposal. To be expanded and refined.

• The "event" is any gathering of people under auspices of an organizer. This includes formal events and informal gatherings before, after, and during a main event.

• The "organizer" is the legally responsible entity or person who hosts an event.

• The "participant" is an adult person attending or helping with an event. Minors at events are assumed to be under protection of their parents or guardians.

• Organizers and participants have rights and obligations under this protocol, which they understand and accept.

• Participants accept that the arbitrators in any event dispute are the organizers.

• Organizers who respect and enforce this protocol may say so on their conference web site.

• In an event, every participant has the right to the sanctity of their person, their time, and their space. We define a "violation" as any act that interferes with this sanctity, without consent of the participant. Silence is not consent.

• Violations include, though are not limited to, doing any of the following without consent: talking to or following a person who has expressed their desire to end a conversation; taking close photographs of a person; touching a person in an intimate or controlling fashion; use of sexualized language; display of sexual imagery; offering unsafe quantities of alcohol; offering illegal drugs of any sort; making unjustified complaints; tampering with a person's possessions.

• In all circumstances where the organizers offer alcohol, they shall also offer drinking water, and sufficient food to keep participants safe. Additionally, they shall have one or more "hosts" who do not drink alcohol, and who are trained to intervene in case of conflict or visible distress.

• The organizers shall provide a secure, private means for reporting violations. Such a report shall be considered circumstantial unless or until backed up. A report should state the time, place, name of the person accused of committing the violation, nature of the violation, and names of witnesses if any. The accused person shall have the right of reply. This material shall be kept private.

• If a participant experiences what they consider a violation, or sees what they consider a violation happening to someone else, they should report this to the organizers.

• if the organizers feel a particular individual is the cause of more than one complaint, and does not self-correct, they may exclude that person from the event. This should be done with respect for the privacy of all parties.

• The organizers shall be aware of the risk of "date rape" drugs. If they receive information about a participant who claims to have suffered from blackouts, or who behaves in uncharacteristic ways causing hurt to themselves or others, they shall assist the participant with a medical blood test for drugs, within 24 hours.

• Organizers should provide participants with an emergency contact phone number during events, for reporting urgent incidents and asking for help.

• If the organizers receive information about a possible criminal offense by or towards any participant, they should report this to local law enforcement and provide suitable support.

by pieterh

Why It Matters

Once in 2007 I organized a conference on software patents. We had a brilliant set of international speakers. The two keynotes were shockingly good, and for me, set the bar for every keynote speech I've ever given.

Mark Shuttleworth, dot-com billionaire and the first South African in space, told the story of the Internet revolution, and said a phrase I'll never forget: "Old money hates new money, and tries to destroy it every chance it gets." His keynote inspired me to write on the digital revolution, and this fight between old and new money. That became Culture & Empire.

Our closing keynote, Bill Kovacic, was an ex-commissioner of the US Federal Trade Commissioner. He didn't use a microphone, and didn't need one. His voice boomed out over the audience. Two things he said, which are not in his slides, stood out. "When Bush took office, every single anti-trust action in the USA stopped, dead." he said. And a little later, he explained why patent pools are so popular in large companies. "Patents trump anti-trust law," he explained. "When a group of CEOs get together to fix prices, that's a criminal offense and they will go to jail. But if they're cross-licensing patents, it's entirely legal!"

Such power in so few words.

For me, this is the reason to attend a conference. It is to be kicked out of my inertia. It is to hear people with years or decades of experience lift the veil on mystery. It is to leave with words burnt into my mind, changing me, and pushing me to do things I'd never dare, otherwise.

It is a tragedy to see brilliance turned into a robotic slide reader. Kovacic had slides, for sure, yet what I remember is his hammering voice, his words, and his deep passionate anger at a corrupt system.

In technical conferences, the best track is often the corridor. I love the corridor, and the long fertile chats. Yet the corridor mostly doesn't discuss the talks. "Did you see so-and-so's talk? Did you like it?" is about as far as it goes. And then we're back to more interesting subjects.

It's better than nothing. A good bubble conference like BuildStuff, bringing people from far away and locking them up with booze and food, for days, can be mind blowing. It is like going on a cruise with a hundred close friends. A few weak talks, or many weak talks, is still worthwhile, if it brings people together.

Yet we can do better. I know we can do better and I think most of us who speak at technical conferences want to do better. Which brings me to this article and ten steps to better public speaking. I'm not going to pull my punches. It took me decades of public speaking to realize some of these techniques were even possible, let alone get good at them.

1. Stop Using Slides

Slides are the training wheels of public speaking. They stop bad accidents that might scar you for life. Yet they are often the biggest barrier to improving your public speaking. Speaking without slides forces you to start learning the real skill of talking to, and not at, your audience.

Once in OSLO (NDC) I gave a talk on open source community building, and four people came. One was the sound engineer, one was an organizer checking on me, and two were there by accident. I gave the talk. The slides were, I think, excellent. My best slides ever, perhaps. And my last.

I've been giving public talks for a long time. Mostly they were the usual affair: twenty slides, five points each, weak delivery from behind a lectern. Good content, mediocre delivery, and a somewhat bored audience. And then sometimes, I'd give a brilliant, life-changing talk.

In June 2005 I came to talk at a small press conference-slash-debate organized by the European Patent Office and the FFII, fighting on opposite sides of the software patent war. This was the Christmas Football Match, a small moment of peace between the hostile camps. The EPO brought their usual suspects: bogus Microsoft-funded startups, industry lobbyists, and regulators. The FFII brought small businessmen, and me.

I asked to speak last. The two panels presented in turns. Each time the FFII's speakers were trying to show their slides, one of the organizers, from the EPO, kept adjusting the projector, playing with the audio and so on. It was not subtle.

When it was my turn I switched off the projector and said, "I'm not going to use slides."

My company had been fighting a software patent troll in the mobile app sector. I'd been talking with other mobile app firms to fight the troll. They all capitulated and paid. We refused, and the troll started attacking our clients. We had to shut down our platform. So I was motivated, and had good stories to tell.

I told off the fake startups for lying to the audience. I told off the EPO guy for trying to sabotage the other speakers. I told the audience, "patents don't create jobs. Hard work by quiet people creates jobs. Patents are a tax, a tool for lawyers to extort and blackmail. Anyone who claims they need patents to write innovative software is a crook or a liar. We will fight this directive, and we will destroy it."

I spoke for fifteen minutes, unrelenting and angry. "You think you can come into our world of software, and change the laws so you can claim and tax our hard work?" We did crush the software patent directive (thanks to the hard work of others), and the FFII later asked me to become its president.

It was a lucky accident that day. Nothing to lose and enough anger to overcome my fear of talking without slides. For a decade that experience has been my benchmark and my goal. A lot has to come together to give a successful talk. No slides seems to be an essential starting point.

Let me break it down further:

• When you use slides, the audience watches the screen, not you. This reduces the strength of your communication. For instance your non-verbal body language is almost totally silent.

• Slides turn the audience from "participants" to "consumers." Half their brain switches off. They can always catch up later. They open their laptops. They think about their work.

• Slides interrupt your focus, as speaker. You switch from screen to audience and back. This switching is visible and makes the audience feel less important to you.

• A presentation, no matter how good, is fixed. There is no way to improvise, no way to adapt your talk to the audience in real-time. And yet that is how the best talks happen.

• Slides are an artificial device. We don't use them in real life, ever. No boy-met-girl story ever contained a 12 page presentation with bullet points. A successful sales pitch happens around food and drinks, not a meeting table.

I can go on. Enough to say that polishing your training wheels does not make you a better cyclist.

Let me quickly answer the common pro-slide arguments:

• "I need them to remember my talk" - it's a question of practice, like cooking without recipes.

• "They help the audience follow my talk" - if your talk doesn't make sense without slides, maybe you are fixing the wrong problem.

• "They make it easier to remember my talk" - visual experiences are entertainment. Learning needs words and voice, work and argument.

• "Organizers and audience demand them" - there are lots of conferences. Being a better speaker gives you more, not less choice.

• "It gives people a chance to catch my talk, later" - this is what the videos are for. If your talk can truly be captured on slides, why even be present in person?

• "I need a way to show data, source code, diagrams" - put them on line, and talk about issues that don't need such illustrations, rather than technical ones. Your talk will be far more interesting and useful.

We're so used to the crutch of slides that we've built our rituals around it. We treat our slot as a chance to dump data on the audience. And the audience expects it, groomed by years of mind-numbing meetings. Stockholm syndrome, perhaps. The audience pretends to understand and remember, and the speakers pretend to believe them. "Great slides! Where can I download them?" We all feel it's bogus yet can't quite verbalize it.

2. Remove the Barriers

When you talk to an audience, you must build trust rapidly. If you fail at this, everything you say will be rejected, no matter how plausible or well explained. Some audiences are more hostile than others. The majority will always be at least skeptical. The fastest way to build trust is to remove all props and barriers and show vulnerability.

Now we come to the deeper psychological reason for not using slides. It is about projection of power, the fixing of trust, and the opening of your audience's mind to your message.

In my work I've often had to resolve conflicts. There are some simple ways to build trust in a difficult meeting. You start by asking for token gestures, "could you pass me the water, please?" Then you remove barriers. Instead of sitting across the table from a person, you sit right beside them. Not so close to make them uncomfortable, yet within stabbing distance, so to speak.

Someone once described my speaking style as "vulnerable." This is indeed what I show, standing in front of the audience without any props to support me. And yet a show of vulnerability is not weakness. It is a claim of power. The other party either rebels, or accepts.

As Mark Rendle said once, this is like patting a large dog on the head. The dog can bite your naked hand. Yet if it does not, then it accepts your dominance. It looks like a friendly gesture, to pat a dog, yet it is all about power.

Slides are a prop. The moment you have something on the screen, you're not patting the dog's head, you're throwing it a stick. It is not the same thing at all. Watch an expert give a weak talk and you see this effect. The more they put on their slides, the less people believe them.

What other props do we tend to use? Obviously, the lectern. Don't stand behind anything. Walk down to the front of the stage and come close to your audience. If you can, get off the stage and get even closer.

How do you dress? Do your clothes make a statement? If so, they're a prop and working against you. I've learned to dress in plain, neutral colors. The less my clothes talk, the more the audience will accept my words.

How do you speak? Are you using jargon, making clever in-jokes, and referring to important companies and people? All of these are props, and they make you look weak. Learn to speak plainly, and if you tell jokes, make them about yourself.

Your own ego must disappear. The more you act or claim to be important, the less people trust you. The same goes for your technology or product. If you give a sales pitch, you are asking for rejection. Always talk from the audience's view. What problems do they encounter?

And here again, why slides are so toxic. How do you know the audience's experience unless you ask them? I like to discover this in the corridor, before my talk. Even a few conversations can give me the key to a talk. It is essentially unscriptable.

Don't use notes. I once had a last-minute panic, at a large event, and came on stage holding some small pieces of paper. The room was huge and filled, and instead of connecting to the audience, I connected with my hand. It was not a good talk.

3. Challenge Your Audience

Everyone is an expert in their own experience. Your best value as speaker is to ask the right questions, not provide the answers. As a layperson you can challenge an expert crowd. The best material you have is the data of your own experience. The best moments are entirely improvised, not rehearsed.

We come to the deeper question of how we think individually, and in a group. There is zero benefit in an exposition of existing finished material. "Here, I will now read the Wikipedia page on apple juice." Or worse, a corrupted version of that. "A fifty minute reading of my version of the Wikipedia page on apple juice." I call this Robot Reader Syndrome.

You must know the problems you're talking about in depth. This means collecting data from your daily experience, and using it to develop new crazy theories and models. It is just the scientific process: take real problems, develop wild answers, and get others to catch your errors. To present a finished product, with no space for negotiation or falsification, sterilizes the collective thinking process.

Let me give you an example. I've often seen conflict in open source communities. That's my data, and no-one can argue with it. Now I'll develop wild theories about where this conflict comes from, and how to solve it. That's my work. Then I present these wild theories to audiences who I hope have experienced the same problems. I wait, breathlessly, for them to tell me where I'm wrong. Please tell me where my models are broken, so I can improve them.

There are many ways to work on your material. Observe your own organizations and projects. Write, write, write. Talk with others, at meetups and in the corridors at conferences. Always, step away from selling an answer, and focus on identifying the real problems and challenges. While a Robot Reader is bad, a Robot Salesperson is worse.

If you are a good communicator, you can do this without even knowing the subject. It's how good consultants work. They enter a troubled organization. They ask staff, "what are the biggest problems?" They then ask others, "what are the best answers?" They write this down in a report, and charge a fat fee.

When you know your problems, you can learn to improvise around them, with time and practice. Don't give the talk people expect. Give the talk they need. Surprise your audience. I've done this a few times: propose one topic, then switch to a different one at the last minute. People mostly like that.

You may be tempted, as a speaker, to prepare a talk and then give the same talk several times. I've seen the same brilliant, interesting speakers give the same talk over months, even years. Part of this is the reliance on slides, again, which creates that sunk costs fallacy. "I've spent so long perfecting these slides that I might as well use them again." Part is the inability to improvise. Part is that culture of ex-cathedra monologues, the speeches and lectures.

Don't be that person who thinks or claims they're smarter than the room. No-one ever is. The only way to truly shine, in a roomful of people, is to be part of that room.

4. Learn the Technical Skills

Public speaking demands a set of technical skills. Some of us are naturally talented speakers. The rest of us can learn these skills with guidance and practice. This is a solved problem. One good solution: find your local Toastmasters club, and work through the program.

We are mostly better at walking than running. Yet almost anyone can complete a marathon, with practice and determination. We are mostly much better at talking to individuals than a group, yet anyone can learn to talk to a large audience.

There are a number of aspects to learn:

• How to speak clearly and slowly, without, uhm, filler noises. The most effective and brutal way is to record yourself speaking a text. Play it back, and note the problems. Too fast, too quiet, dropping sounds, hesitating, and so on. If you can't hear your own errors, get someone to help. As a general rule, speak more slowly, be more precise about the sounds, and lower your timbre.

• How to lose or shift your accent. Some accents are barriers, some are attractively vulnerable. It depends on the culture. In general, the more neutral you can sound with respect to your audience, the better. This can mean adopting the accent of your audience, or a neutral international accent.

• How to structure a talk. There are classic structures you can learn and use, e.g. introduction, three points, and conclusion. Or, take a classic story plot like overcoming a monster, a common theme in the software industry. My preference is for less structure and more dialog, which we'll come to later.

• How to time yourself. This is an essential basic skill. You must at the least offer the audience time for questions. It is a rude and inconsiderate speaker who takes all their time reading their slides, and then starts to break the schedule by continuing to talk. The conference should cut speakers off if needed, yet for me, self-timing is as important as not mumbling. Start by learning to talk for exactly five minutes. Then work up over years until you can structure a 50-minute talk on the fly.

• How to use your body language. This means stepping away from the lectern, and standing in front of your audience. It can be terrifying. Nonetheless, so is driving a car the first time. Good body language for talking is relaxed, open, and vulnerable. For example, arms open and using your hands, yet never raising your hands above your shoulders, nor making aggressive gestures.

• How to connect visually with your audience. Again, removing barriers is a first step. You cannot connect when you are reading your laptop screen. Then, speaking to specific people in the crowd. Speak to people close to you but not at the front. At the end of every phrase, switch to someone else, so you scan the room. If the room permits it, come down to the audience and literally talk to them. All the time, stay relaxed and smiling.

5. Learn from Failure

You will always learn more from your failures than your successes. This means always pushing yourself out of your safe zone, and then using your stumbles to become a better speaker.

It can be intensely painful, humiliating, and discouraging to give a bad talk. When you aim for perfection and you set your standards high, you will fail (by your own measure) more often. The good part is that your worst talks will still be better than average.

For me, for example, a talk is a failure if I don't get more questions than I can answer in the time. My goal as speaker is to motivate people to do further research. So a cold room (or an empty one) is data that I did something wrong.

To avoid traumatic disaster, start with short talks in forgiving environments. Build up your confidence and skills over time. After every talk, do this:

• Get one or two points for improvement from someone in the audience whom you trust. "It was great," is not a help. "You spoke too rapidly," is helpful.

• Get someone to record you, and watch yourself on video. It can be unpleasant to watch yourself speak yet you need to do this to see yourself improve. It can often take months for conferences to publish videos, so don't depend on that.

Experiment with different formats so you build up your range. If you don't find events that offer you what you want, organize your own. For example, one of my favorite formats is a multi-day workshop, which I organize myself.

Watch other talks and identify what doesn't work for you. Be honest about your irritation and then ask whether you cause the same feeling when you talk.

When you do have a bad talk, use this as material. It makes a great opening, and I've done this. "Anyone here from Oslo? Yes? Well, I hate Oslo. Nothing personal, but last time I spoke there exactly four people came to my talk, and two of those had come to the wrong room."

Of course, it's never Oslo's fault. As speaker, you did something wrong, and you can fix that, and make it better next time. (My main lesson from Oslo was that the talk title makes all the difference.)

6. Maintain Protocol

Public speaking is an old art with a well-defined protocol. If a conference doesn't maintain protocol, organize it yourself.

I'm shocked by how few conferences know and maintain protocol. It's not complex, yet it does make a real difference. Here is how it tends to work:

• Speaker fiddles with laptop, while audience comes into room.
• After some extended silence, speaker asks, "everyone ready?" and starts talking.
• When speaker is done, audience applauds and walks out of room.

Here is how it should happen:

• The Master of Ceremonies (MC) stands on stage, watching the audience as they take their places.

• MC encourages audience to move closer, asks, "everyone ready?" and waits for an answer.

• MC welcomes the audience and thanks them for coming. He then introduces the speaker by name, and cites some of their accomplishments.

• MC asks the audience for a warm hand of applause, and welcomes the speaker up on stage.

• MC passes the mike, shakes hands with the speaker, and walks off.

• When the talk is over, MC asks the room to applaud, while speaker takes a bow (or looks bashful), and then MC takes back the mike.

This protocol isn't just decoration. It creates trust between speaker and audience, and it provides a fail safe in case the microphone isn't working (it's the MC who suffers, not the speaker). It is less stressful for the speaker. A good MC can save a flailing speaker from disaster, by intervening if the room is too cold to ask questions. And an MC can be a natural timekeeper, intervening when a speaker talks too long or bores the audience with endless slides.

If you organize a conference, please have an MC on every stage. If you are a speaker without such support, get another speaker to be your MC.

7. Build a Dialogue

Personally, both in the audience and on stage, I want a dialog. I want to be able to interrupt and challenge the speaker, and as speaker, I want the audience to do this. It is not about argument: it is about thinking together.

As speaker, you face Newton's First Law of Thermodynamics: people will sit quiet and not talk unless they feel provoked. So creating a dialog requires some planning and effort.

Start with the talk title. My best ever title was click-bait, "One Weird Trick for Making Perfect Software". I asked the organizer, is this acceptable, and she laughed, and told me it was great. And it was. A good title attracts and annoys, fifty-fifty. It fills the room with people who already want to argue with you.

Next, take control of the room. There are many ways to do this. It depends on the room, the audience, the culture, the time of your talk, and how much beer you had the night before. Here are some of my techniques:

• Watch the audience and make them wait, just a little too long, until there is total silence. This works in a larger room.

• Ask everyone to stand up and wait until literally the entire room is standing. Then ask them to sit again. I like to do this when I see people with open laptops.

• Ask everyone to stand up, and shake the hand of the person behind them. I've tried this twice in large rooms. One time the whole audience broke out in laughter. The second time, not a smile.

• Tell a self-deprecating joke about yourself. When you can get a room to laugh, you have taken control, as every comic knows.

• Start the talk by asking a simple yet difficult question, then give a reward to the person who answers correctly. I bring along copies of my books for this. It focuses attention and wakes everyone up.

• Thank the organizers again, and ask the audience to applaud. It's cheesy, yet it works.

When you've taken control of the room, you can start to talk seriously. You must anchor your talk in the experience of your audience. Again, I like to ask questions that show how common a problem is, or how rare a good answer is.

With large audiences (over 1,000) it can be helpful to bring someone on stage as your foil. If you follow my advice about having an MC, you have your foil. What the foil does is challenge you, in name of the audience. It is hard to chat with thousands of people, so the foil substitutes for the crowd.

I've also used a Twitter feed for questions, and that can work nicely. There is the danger of breaking the connection to your audience, as you read the tweets. The conference where I used that technique had arranged large screens at floor level, aiming up at the speaker, which was incredibly useful.

Every room is different, so you must spend time in the room and watch other talks in that same space if you can. In small rooms you don't need to pass microphones around. In larger rooms, you must. Or, you must repeat the question for the camera and rest of the room.

You can also change the layout of the room, in some cases. I've done this successfully in smaller events, where the seats started out in classic lines facing the stage, like a school class room. Ten minutes of moving stuff around, and we had a nice circle of sofas and chairs, almost like a campfire.

There are some terrible rooms: huge flat rectangles designed to hold a thousand people all sleeping at the same time. In such rooms, bully the organizers into switching off all video, then move left and right on the stage so you can speak to most of the crowd. You may also kick off by asking people to move into the center, if the room is not full.

I like to poll for questions about half way through the time, and then drive the talk by answering questions. When this works, it is great. When it fails, it's dramatic. Recently there were no questions and so I simply stopped the talk early: the audience (like me) was exhausted from the party the evening before. Having an MC on stage is a good backup: I learned my lesson there, again.

Answering questions well takes skill at improvising. You don't actually need to answer the question. Rather, it can be an excuse for exploring another interesting topic. The best questions are a direct challenge to the ideas you are talking about.

8. Keep It Simple

This is perhaps the hardest skill of all: to explore one idea in depth rather than touch the surface of a hundred different ideas. Yet it is the difference between garbage and gold. Every piece of your talk should be carrying your story forwards, not telling other random stories.

There are logical (yet false) reasons why speakers try to cover way too much. They feel they have to fill their speaking slot (rather than give the audience time to think and argue). They still use slides, so work backwards from the time: "in one hour, I can convey ten big ideas." They suffer from the "ten slides good, twenty slides better" type of false reasoning (it works better as "ten slides bad, twenty slides worse.")

I think public speaking is like writing, carpentry, cooking, or music. You add value by making your product simpler and easier to digest, not by adding more. Any fool can make complexity. Simplicity is the real challenge.

The core of your talk should be an idea or argument or model that is rare, controversial, and valuable enough to justify the moment. It should be worth the time the audience takes to travel, and the cost to you and the organizers to be there. Above all, you should be answering real questions the audience is facing now or will face soon.

You can then explain and describe your model from many angles. These are not different stories. They are the same story, told in more and more detail. Each explanation gives the audience another perspective, and if you are doing your job well, they start to see the whole picture.

Do not expect people to understand a complex new idea in a single sitting. That is not how it works. All you can do is break the ice of skepticism and plant a tiny seed. If conditions are right, the seed will grow and many months or years later, come to fruit.

If you are challenging established culture, or authority, or habits, you will face resistance. The more investment in out-of-date models, the more resistance to change. Imagine this as wind. The simpler your idea and the more focused your presentation of it, the smaller your wind profile. And so, the lower the resistance.

9. Keep the Discussion Going

Your work starts long before you give a talk, and ends long after. Subtle and deep truths can take years to hit home. And your own thinking will evolve and deepen over time. So you help yourself, and your audience, by keeping the discussion going over months and years.

I use various techniques for this. Obviously, there's this blog and my books, which act as shorthands. Instead of repeating myself, I can refer to an existing chain of argument.

This article, for instance, came from a corridor discussion (well, more of a beer and rock music discussion) with Dylan Beattie. Perhaps in the future it would make a subject for a talk, in its own right. When you can, connect the past with the future like this. It just takes the habit of writing short pieces regularly.

However, articles are opinion. Facts are more compelling. So a key part of the long term discussion is code and formal documents such as RFCs. These are not truth, yet they are easily falsifiable. If no-one uses a particular piece of code, or a given contract, you can assume they are inaccurate or irrelevant.

I also like to collect my videos, as it gives people a view that stretches over years. It is also about self-promotion, something you must do as a speaker. People need to want to come to my talks. Empty rooms are no fun for anyone.

Self-promotion sounds negative, yet it's essential. Partly you need to build and understand your own success, to be a better speaker. This is to deal with imposter syndrome, which I'll explain in the next section. Partly, you need access to interesting conferences, and that means gaining a reputation and a following.

My advice is to build your reputation by your work, no more or less. The world cares about results, not ideas or vision, or even history. Don't tell us who you worked for, who your best friends are. Tell us what you made and give us a link so we can check it out.

If you are, like me, in the software business, invest in open source. It is the number one way to build your knowledge, and reputation. Your Github profile is your CV. This is mostly true today and it is almost entirely true tomorrow. Show me a solid history of work and I trust what you say. Show me a blank page, and I'm going to start questioning your motives.

Once you drink the github koolaid, you can use it for everything. Put your talks there, and accept pull requests. Put all your writing there, and accept pull requests! Use it for your blog (and accept pull requests).

10. Manage Your Emotions

Lastly, and hardest for many of us, is dealing with the fear and anxiety of speaking to a crowd of strangers. Many of us don't feel competent, or confident enough. We are afraid of saying the wrong things, of looking foolish, and of being rejected. Yet there are ways to overcome all these emotions.

The fear of performing is called "imposter syndrome." It affects many speakers, as it affects many people who put themselves in front of an audience. The feeling is one of, "I don't deserve to be here," and it can be crippling. It is worse for anyone who suspects they were invited for reasons other than their skills and experience. It is often worse for women than for men.

All our emotions can be tamed, it is a matter of technique and practice. I've written about this on my blog and in my book The Psychopath Code. This also applies to imposter syndrome, which is essentially the fear of exposure as a fraud, and rejection.

The fear is based on a chain of subconscious assumptions. One, that we're not really good, just lucky. Two, that we will stumble, and reveal our incompetence. Three, that people will react by rejecting us in horror.

Let's tackle competence first. This is, I feel, a symptom of society placing too much burden on the individual to be special, and different. Accept that we're all ants, little pieces of a complex system that works astoundingly well. Sure, we can flicker with individual brilliance now and then. It means little. Our superpower comes from working together. Simply by existing, you add value.

Next, fear of failure and exposure. This is a symptom of culture asking us to be perfect. I've said, embrace your mistakes and learn from them. Fail with happiness and grace, recover, and continue. When there is nothing to hide, there is nothing to expose.

Last, fear of rejection. There's a small mantra you can repeat to yourself. It's our vulnerabilities that make us attractive to others, not our strengths. We reject people who are difficult to work with because they are anti-social, arrogant, and deceptive. Such people never feel imposter syndrome. The simple fact you're afraid to fail makes you precious to others.

Here are other techniques I recommend to fix imposter syndrome:

• Remove the barriers between you and the audience. It feels more scary at first and then you realize that the less you try to hide, the less fear you have of exposure.

• Treat failure as science. It is better to try ten things, and see eight fail, than to try just one and hope it works. For me, this means starting lots of projects, writing lots of articles, and speaking at lots of talks. Many are failures. The ones that survive are my successes.

• Before the audience comes into the room, go onto the stage, and walk around. Tell yourself, "this is my stage, this is my place, and the audience are my guests and friends."

• When you talk, never lie, bluff, or make claims you can't back up. When you don't know something, just say, "I don't know." There are people who are professional liars: you're not one of them.

• Stay busy. If you have a talk coming up, write an article about the topic. Keep researching it, and try out different story lines on people.

• Trust yourself. You are the child of a billion generations of survivors. Good luck has nothing to do with it. You have winning genes backed up by hard work.

Conclusions

If you're passionate about your work, then sooner or later you will want to share that passion with others. There are many reasons to want to become a good public speaker. Perhaps the main one is simply to get out into the world and meet interesting people. For me, it is a way to learn from others, and shake up my ideas in a way that's impossible working alone.

I've explained my best techniques for capturing an audience's attention, and getting them to consider your ideas seriously. It is not easy. The typical conference attendee remembers only 9% of what they hear, by the end of the week, and a month later that figure is down to 0.1%. OK, I'm making this up, yet you get my point.

Stop using slides, get closer to your audience, and challenge them. Engage them in dialog, and make them think. Keep your voice and body focused on the room. Use no props, gimmicks, or notes. Leave your fear off-stage, and if a talk bombs, accept that with grace.

It is a slow, long process, so be patient and always kind to the organizers and staff.

by pieterh

Standard practice (Pessimistic Merging, or PM) is to wait until CI is done, then do a code review, then test the patch on a branch, and then provide feedback to the author. The author can then fix the patch and the test/review cycle starts again. At this stage the maintainer can (and often does) make value judgments such as "I don't like how you do this" or "this doesn't fit with our project vision."

In the worst case, patches can wait for weeks, or months, to be accepted. Or they are never accepted. Or, they are rejected with various excuses and argumentation.

PM is how most projects work, and I believe most projects get it wrong. Let me start by listing the problems PM creates:

• It tells new contributors, "guilty until proven innocent," which is a negative message that creates negative emotions. Contributors who feel unwelcome will always look for alternatives. Driving away contributors is bad. Making slow, quiet enemies is worse.

• It gives maintainers power over new contributors, which many maintainers abuse. This abuse can be subconscious. Yet it is widespread. Maintainers inherently strive to remain important in their project. If they can keep out potential competitors by delaying and blocking their patches, they will.

• It opens the door to discrimination. One can argue, a project belongs to its maintainers, so they can choose who they want to work with. My response is: projects that are not aggressively inclusive will die, and deserve to die.

• It slows down the learning cycle. Innovation demands rapid experiment-failure-success cycles. Someone identifies a problem or inefficiency in a product. Someone proposes a fix. The fix is tested and works or fails. We have learned something new. The faster this cycle happens, the faster and more accurately the project can move.

• It gives outsiders the chance to troll the project. It is a simple as raising an objection to a new patch. "I don't like this code." Discussions over details can use up much more effort than writing code. It is far cheaper to attack a patch than to make one. These economics favor the trolls and punish the honest contributors.

• It puts the burden of work on individual contributors, which is ironic and sad for open source. We want to work together yet we're told to fix our work alone.

Now let's see how this works when we use Optimistic Merge. To start with, understand that not all patches nor all contributors are the same. We see at least four main cases in our open source projects:

1. Good contributors who know the rules and write excellent, perfect patches.
2. Good contributors who make mistakes, and who write useful yet broken patches.
3. Mediocre contributors who make patches that no-one notices or cares about.
4. Trollish contributors who ignore the rules, and who write toxic patches.

PM assumes all patches are toxic until proven good (4). Whereas in reality most patches tend to be useful, and worth improving (2).

Let's see how each scenario works, with PM and OM:

1. PM: depending on unspecified, arbitrary criteria, patch may be merged rapidly or slowly. At least sometimes, a good contributor will be left with bad feelings. OM: good contributors feel happy and appreciated, and continue to provide excellent patches until they are done using the project.
2. PM: contributor retreats, fixes patch, comes back somewhat humiliated. OM: second contributor joins in to help first fix their patch. We get a short, happy patch party. New contributor now has a coach and friend in the project.
3. PM: we get a flamewar and everyone wonders why the community is so hostile. OM: the mediocre contributor is largely ignored. If patch needs fixing, it'll happen rapidly. Contributor loses interest and eventually the patch is reverted.
4. PM: we get a flamewar which troll wins by sheer force of argument. Community explodes in fight-or-flee emotions. Bad patches get pushed through. OM: existing contributor immediately reverts the patch. There is no discussion. Troll may try again, and eventually may be banned. Toxic patches remain in git history forever.

In each case, OM has a better outcome than PM.

In the majority case (patches that need further work), Optimistic Merge creates the conditions for mentoring and coaching. And indeed this is what we see in ZeroMQ projects, and is one of the reasons they are such fun to work on.

References:

• ZeroMQ RFC 22, C4.1: the Collective Code Construction Contract.

My top tips were, for what it's worth:

• People before code: build the right community and it will build the right code.
• Progress before consensus: look for processes that work without upfront consensus (except on rules).
• Problems before solutions: use a problem-driven process.
• Contracts before internals: use contracts to test behavior, not inspection of internals.
• Rules before hope: write down your development process or use C4.1.
• Merit before power: treat everyone fairly and equitably.
• Market before product: aim for a market of competing and interoperating projects, not a single product.

by pieterh