Standard practice (Pessimistic Merging, or PM) is to wait until CI is done, then do a code review, then test the patch on a branch, and then provide feedback to the author. The author can then fix the patch and the test/review cycle starts again. At this stage the maintainer can (and often does) make value judgments such as "I don't like how you do this" or "this doesn't fit with our project vision."

In the worst case, patches can wait for weeks, or months, to be accepted. Or they are never accepted. Or, they are rejected with various excuses and argumentation.

PM is how most projects work, and I believe most projects get it wrong. Let me start by listing the problems PM creates:

• It tells new contributors, "guilty until proven innocent," which is a negative message that creates negative emotions. Contributors who feel unwelcome will always look for alternatives. Driving away contributors is bad. Making slow, quiet enemies is worse.

• It gives maintainers power over new contributors, which many maintainers abuse. This abuse can be subconscious. Yet it is widespread. Maintainers inherently strive to remain important in their project. If they can keep out potential competitors by delaying and blocking their patches, they will.

• It opens the door to discrimination. One can argue, a project belongs to its maintainers, so they can choose who they want to work with. My response is: projects that are not aggressively inclusive will die, and deserve to die.

• It slows down the learning cycle. Innovation demands rapid experiment-failure-success cycles. Someone identifies a problem or inefficiency in a product. Someone proposes a fix. The fix is tested and works or fails. We have learned something new. The faster this cycle happens, the faster and more accurately the project can move.

• It gives outsiders the chance to troll the project. It is a simple as raising an objection to a new patch. "I don't like this code." Discussions over details can use up much more effort than writing code. It is far cheaper to attack a patch than to make one. These economics favor the trolls and punish the honest contributors.

• It puts the burden of work on individual contributors, which is ironic and sad for open source. We want to work together yet we're told to fix our work alone.

Now let's see how this works when we use Optimistic Merge. To start with, understand that not all patches nor all contributors are the same. We see at least four main cases in our open source projects:

1. Good contributors who know the rules and write excellent, perfect patches.
2. Good contributors who make mistakes, and who write useful yet broken patches.
3. Mediocre contributors who make patches that no-one notices or cares about.
4. Trollish contributors who ignore the rules, and who write toxic patches.

PM assumes all patches are toxic until proven good (4). Whereas in reality most patches tend to be useful, and worth improving (2).

Let's see how each scenario works, with PM and OM:

1. PM: depending on unspecified, arbitrary criteria, patch may be merged rapidly or slowly. At least sometimes, a good contributor will be left with bad feelings. OM: good contributors feel happy and appreciated, and continue to provide excellent patches until they are done using the project.
2. PM: contributor retreats, fixes patch, comes back somewhat humiliated. OM: second contributor joins in to help first fix their patch. We get a short, happy patch party. New contributor now has a coach and friend in the project.
3. PM: we get a flamewar and everyone wonders why the community is so hostile. OM: the mediocre contributor is largely ignored. If patch needs fixing, it'll happen rapidly. Contributor loses interest and eventually the patch is reverted.
4. PM: we get a flamewar which troll wins by sheer force of argument. Community explodes in fight-or-flee emotions. Bad patches get pushed through. OM: existing contributor immediately reverts the patch. There is no discussion. Troll may try again, and eventually may be banned. Toxic patches remain in git history forever.

In each case, OM has a better outcome than PM.

In the majority case (patches that need further work), Optimistic Merge creates the conditions for mentoring and coaching. And indeed this is what we see in ZeroMQ projects, and is one of the reasons they are such fun to work on.

References:

• ZeroMQ RFC 22, C4.1: the Collective Code Construction Contract.

My top tips were, for what it's worth:

• People before code: build the right community and it will build the right code.
• Progress before consensus: look for processes that work without upfront consensus (except on rules).
• Problems before solutions: use a problem-driven process.
• Contracts before internals: use contracts to test behavior, not inspection of internals.
• Rules before hope: write down your development process or use C4.1.
• Merit before power: treat everyone fairly and equitably.
• Market before product: aim for a market of competing and interoperating projects, not a single product.

by pieterh

Comments are welcome

I'm going to propose one answer at the end of article. First, a typical definition: "Harassment is any form of unwanted and unwelcome behaviour which may range from mildly unpleasant remarks to physical violence." You should immediately see a problem: it is broad and subjective (violence apart). Next, let me suggest ten myths about harassment. These are claims or assumptions people often make when discussing harassment.

1. Harassment equals discrimination

Discrimination is prejudgment of individuals based on recognizable yet irrelevant criteria: ethnic origin, gender, religion, appearance, etc. It is often accompanied with harassment, used to implement discriminatory policies. Yet these are not the same. Much, even most harassment has no element of discrimination.

2. Harassment is a gender/race issue

This is the same myth, restated in terms of the feminist and "minority" struggles against discrimination. The danger with this myth is that it misses the bulk of harassment, misunderstands it, and ironically, makes it harder to fix. Worse, the myth splits us. We solve harassment together, not divided.

3. Harassment is rare so it's irrelevant

In fact slow, methodical harassment is widespread. It is often covert and indirect: violence against our time, space or belongings. Lies and distortion of events. Attacks on our reputation. Neglect of us and our environment. Intrusions into our privacy. We are so blunted by this that we shrug and dismiss harassment as inevitable.

4. Harassment is an inevitable part of life

While harassment is common, it is not inevitable. Many groups are free of it. Yet others are riven by it. I've seen both cases in my work with communities. It shows in people being broadly happy, or broadly miserable. Consider it a form of infectious disease that can hit anyone, and the goal to be 100% eradication. It takes science, knowledge, and patience to cure a disease.

5. Harassment is obvious when you see it

Since we've learned to unsee harassment, we assume it must be dramatically visible when it happens. In fact, most harassment is a chain of small, almost invisible acts spread out over time. The drama comes at times, yet it hides the far larger burden of costs. Looking for drama is counter-productive. We must instead look for damage.

6. Anyone can be the harasser

Like the best myths this is half-true. We're all capable of joining a mob, as in that video. Temporary power over others can be addictive: see how predatory that mob is. Yet most harassment is a long term, focused activity. Anyone with empathy is jolted when they realize they are hurting someone else. It can take a while for empathy to react. We can all make mistakes, yet most of us self-correct.

7. Harassment is a motiveless act

Harassment is driven by a hunger for power, unencumbered by empathy for others. It is the tool of a small slice of people who use it to capture victims, attack rivals, discredit critics, and divide opposition. Such people learn the techniques young, and practice them all their lives. They are almost invisible, except by the damage they do to others.

8. Adults will police each other

Sometimes this works, yet it is not reliable. People tend to accept the word of the most dominant, charismatic individual in a group or couple. Charming lies survive for years. Did you learn the tongue taste map at school? It is 100% bogus. If the harasser is assertive, they can turn a mob into their tool. Most of us are too polite, afraid, and timid to police others.

9. Outlawing harassment will stop it

There are those who follow rules and social mores. And there are those who disrespect others, and harass them when it suits. Bullying is banned in every civilized school, yet is widespread. Rules without fair and accurate enforcement is worse than no rules. Harassers are the first to exploit poor rules and weak authority.

10. We don't need rules, we have manners

Look at anti-harassment rules differently. They are not to tell the harasser what not to do. Rather, they are to tell the rest of us when to ask for help. Weak rules say, "Don't do A or B." Good rules say, "X and Y are not acceptable. If this happens to you, follow procedure Z."

Conclusion

I'm a reluctant authoritarian, assuming everyone has the freedom to walk away from corrupt authorities. I've seen groups riven by harassment (like the FFII I was president of for two years), and groups almost entirely free of it (like the ZeroMQ community). The difference is not accidental.

In my experience, the cure for harassment is a mix of a clear contract, impartial and neutral enforcement, education, and freedom. The most plausible punishment is exclusion (banishment), giving a perpetrator time and chance to self-correct, apologize, and prove their goodwill. Education means teaching people about the underlying models of harassment, to recognize distress in themselves and others, and to respond to it appropriately. Freedom means the ability to leave and form new groups, if authority is corrupt.

In the video, I guess the harasser is one of the louder students attacking the professor. He or she is using the situation to build up power over their peer group, and over the faculty. They're using the professor as an easy common enemy, to create emotions. They already have a slice of the crowd as followers. Tomorrow, a bunch more. Their goal is to use the entire campus as a power base. It's practice. They'll graduate to business or politics.

by pieterh

1. Newton's First Law of Motion

Everything we do has an economic motive. Things only move as a result of energy coming from somewhere else. People and organizations also follow this law. Economic motive is a force. The more accurate the economic motive, the more accurate the direction of movement. If a factory produces 500,000 rubber boots because the CEO has a boot fetish, that's inaccurate. If it's responding to rising sea levels and demand, that's accurate.

Lesson: give every team and individual access to accurate economic incentives.

2. Newton's Second Law of Motion

The bigger the team, the more force it needs. Sometimes it's a bike. Sometimes it's a car. The smaller the thing you want to move, the easier and faster it moves. If you try to change a 5,000 person company, you need massive force. To change a two person team takes a few beers and a single client.

Lesson: smaller teams move faster than larger teams, given the same force.

3. Newton's Third Law of Motion

When you push an organization, it pushes back. Or, put this another way, if you try to change a 5,000 person company, you will be fired. Unless you're the CEO. Then you'll be given a huge bonus, and then fired.

Lesson: build fresh organizations instead of changing existing ones.

4. The Equivalency Principle

Falling is indistinguishable from making progress. Until your face hits the pavement, you will believe you're in control. Since all movement is a reaction to an external force, you can only judge your success by aiming for the pavement, and failing.

Lesson: the bigger your leaps, the more damaging your failures.

5. Murphy's Law

If it can break, it will break. In fact the full law adds, "in the worst possible way, at the worst possible time." Failure isn't just all around us, it's inevitable. We cannot make perfect systems. Rather than trying to prevent failure, we need to learn to embrace failure, and make it part of our process.

Lesson: embrace failure as the best way to learn what works.

6. The Uncertainty Principle

The more you know about one topic, the stupider you become. Or, as my mom used to tell me, "never trust someone who has all the answers, especially yourself." Experts are dangerous, if they are not balanced by naive laymen. Diversity is more valuable than expertise.

Lesson: diversity is not a political slogan. It's the basis for collective intelligence.

7. Zipf's Law of Power Distributions

20% of any system always has 80% of the power. It applies to cities, languages, earthquakes, and economies. And organizations, and software systems. You'll spend most of your effort on a fraction of the software. Over-engineering code that isn't in the critical path is a waste of time.

Lesson: if shitty code solves the problem, it's not shitty code.

8. Moore's Law of Cost Gravity

Cool stuff gets 50% cheaper every 18-24 months. Whether we like it or not, our software lives on an exploding cloud of silicon. Pavel Baron said, "architecture is dead." Even a single program behaves differently than its authors believe. This is why we need profilers, to optimize code. A system with many pieces is unknowable.

Lesson: the size of our systems — the number of moving pieces — doubles every 18-24 months, and no-one fully understands any system.

9. Amdahl's Law

The more you need consensus, the less work you can do. If you spend one hour a day in meetings, that limits your effective team size to eight people. At best. In practice most of us spend a lot more time getting consensus before we can start working. Do you need approval to start on work? Do your team have standards for language and platforms? Do you have to wait for approval to put your patch into production? These are all wait states. The maximum effective size of most teams is probably less than five.

Lesson: read about the Actor model, and become a message-driven, zero shared state Actor.

10. Conway's Law

The software you make looks like your organization. If you are in a shitty organization, you will make shitty software. To grow a large scale decentralized system (and nothing else survives these days) you grow a large scale decentralized organization. If your organization causes you pain, the software it makes will cause pain to its users.

Lesson: if you're not happy in your job, build yourself a better job.

by pieterh

1. People Before Code

This is the Golden Rule, taught to me by Isabel Drost-Fromm. Build community, not software. Without community your code will solve the wrong problems. It will be abandoned, ignored, and will die. Collect people and give them space to work together. Give them good challenges. Stop writing code yourself.

2. Use a Share-Alike License

Share-alike is the seat belt of open source. You can boast about how you don't need it, until you have a bad accident. Then you will either find your face smeared on the wall, or have light bruising. Don't become a smear. Use share-alike. If GPL/LGPL is too political for you, use MPLv2.

3. Use an Zero-Consensus Process

Seeking upfront consensus is like waiting for the ideal life partner. It is kind of crazy. Github killed upfront consensus with their fork/pull-request flow, so you've no excuse in 2015. Accept patches like Wikipedia accepts edits. Merge first, fix later, and discuss out of band. Do all work on master. Don't make people wait. You'll get consensus, after the fact.

4. Problem, then Solution

Educate yourself and others to focus on problems not features. Every patch must be a minimal solution to a solid problem. Embrace experiments and wild ideas. Help people to not blow up the lab. Collect good solutions and throw away the bad ones. Embrace failure, at all levels. It is a necessary part of the learning process.

5. Contracts Before Internals

Be aggressive about documenting contracts (APIs and protocols) and testing them. Use CI testing on all public contracts. Code coverage is irrelevant. Code documentation is irrelevant. All that matters is what contracts the code implements, and how well it does that.

6. Promote From Within

Promote contributors to maintainers, and maintainers to owners. Do this smoothly, easily, and without fear. Keep final authority to ban bad actors. Encourage people to start their own projects, especially to build on, or compete, with existing projects. Remove power from people who are not earning it on a daily basis.

7. Write Down the Rules

As you develop your rules, write them down so people can learn them. Actually, don't even bother. Just use the C4.1 rules we already designed for ZeroMQ, and simplify them if you want to.

8. Enforce the Rules Fairly

Use your power to enforce rules, not bully others into your "vision" of the project's direction. Above all, obey the rules yourself. Nothing is worse than a clique of maintainers who act special while blocking patches because "they don't like them." OK, that's exaggeration. Many things are much worse. Still, the clique thing will harm a project.

9. Aim For the Cloud

Aim for a cloud of small, independent, self-organizing, competing projects. Be intolerant of large projects. By "large" I mean a project that has more than 2-3 core minds working on it. Don't use fancy dependencies like submodules. Let people pick and choose the projects they want to put together. It's economics 101.

10. Be Happy and Pleasant

Maybe you noticed that "be innovative" isn't anywhere on my list. It's probably point 11 or 12. Anyhow, cultivate a positive and pleasant mood in your community. There are no stupid questions. There are no stupid people. There are a few bad actors, who mostly stay away when the rules are clear. And everyone else is valuable and welcome like a guest who has traveled far to see us.

by pieterh

C4.1 claims its first goal is to "maximize the scale of the community around a project." Community means many things. Above all it means life. If no-one uses or contributes to your software, the code does not matter.

Here is my analysis of the growth of the ZeroMQ community since we moved to git in mid-2009:

The picture speaks volumes. You can see the activity grow, peak, and slow down for each major version. You can see the clear difference between "before" and "after" the fork in late 2011, when the original maintainers left to create Crossroads.io, and ZeroMQ moved to the C4.1 process.

What is most fun is that ZeroMQ takes zero management these days. It is self-steering, self-feeding, and self-organizing. We have applied C4.1 to old projects like libzmq, and brand new ones. The results are the same. If the project does something useful, it takes off, and contributors join, and everyone enjoys themselves.

C4.1 is an exothermic process. It produces almost perfect software. ZeroMQ's oddities all date from before 2012, and are slowly disappearing. ZeroMQ master is almost always stable. It just does not break stable APIs or protocols. The mailing list is polite and happy. We do not argue.

If you run an open source project, and your commit chart does not look like this, go read ZeroMQ RFC 22.

Here is how I produced this chart:

• Take all of the repositories in the ZeroMQ organization on github. There are many times more projects that aren't in the organization (about 20 for each one in the organization). I ignored these. There are "stabilization forks" of projects, like zeromq4-x. I ignored these as well.

• Clone each repository to my laptop, then listed all commits since the start, using this command: git log —date=short —format="format:%ad", for each project.

• Get the number of commits per day using this command: sort activity.log | uniq -c

• Run a Perl script that calculates the totals per week, and prints out a comma-separated values file.

• Load that into LibreOffice and generate a graphic.

by pieterh

The Economics of Contracts

In this section I'll explain why we need contracts and standards in software.

Definitions first: a "software contract" is an API, or a protocol, or some other formalized interaction that allows two otherwise unconnected components of a system to work together. I've written a lot about contracts, because they are so vital to making successful distributed software. Contracts let us build overall systems in parallel, in many places and at many times, and with many teams. However, contracts are a vital concept in all types of software.

All libraries and products implement contracts, explicit or implicit. Contracts may be documented (using formal prose like RFCs, ideally), or embedded in code (less than ideal). A contract may have multiple independent and competing implementations, and the more implementations, the better.

In ZeroMQ we have wire-protocol contracts like the ZMTP protocol and CurveZMQ security protocol, and the semantics of various socket patterns. We also have contracts for the API, specified as man pages.

When a contract is written and published by third parties, it becomes a standard.

The economics of standards are simple: they enable competition between suppliers, and this drives down the cost of products. Suppliers have an incentive to keep contracts undocumented, over-complex, hard to implement, or even proprietary. Customers have an incentive to turn contracts into standards, to simplify them, make them easier to implement, and bring the cost of knowledge to zero.

In markets where standards are properly regulated to remain free and open, the cost of goods does fall as it should. In markets where prices are not falling, the cause is generally lack of competition, enabled by closed, complex, or undocumented contracts. Free and open standards promote a competitive free market, whereas captive standards or captive contracts raise costs.

This is how it works in large-scale real world economies, and it is also how it works in software projects, which emulate economies (often they emulate an insane centrally planned economy, though that is another story).

In any software project, even an open source one, the incentives of competition and capture are present, even if Euros and Dollars aren't literally on the table. We are all Comcast, at times. When you're choosing a software stack, your primary questions should be, "does this stack implement public contracts?" and then, "who else implements these same contracts and are they standards?"

A contract needs several properties to work as a real public standard. Above all, it must be published, and it must be freely remixable (to prevent capture). The more we tend to formal processes and organizational ownership, the more it costs to make a contract, and the more vulnerable it is to capture.

So ideal public contracts are owned collectively, and published centrally only for convenience. Every project and product must be free to define its own contracts, either new ones, or remixes of existing works. These contracts should move, over time, towards stability and acceptance as standards by the market (if they are successful), or else should die and disappear.

The Problem With Software Versions

In this section I'll try to nullify the dominant theory of software versions.

The worst times in ZeroMQ's history were when we changed the library version to signal major and incompatible change in its contracts. Version 2.0 did not talk to version 3.0, which did not talk to (the then) 4.0. They had incompatible protocols and different APIs. These were in effect different products, each trying to define its own private (and poorly documented) contracts.

To fix this problem, we introduced the rule that new stable releases of the software must (a) talk to old stable releases and (b) they must support existing apps, without changes. We more or less succeeded with that, so ZeroMQ versions 3.2 and 4.0 work nicely with 2.2 and 2.1, for example. We slipped only once or twice, by shipping a stable release with new contracts that turned out to have flaws which needed fixing.

Initially, this rule made it harder to make stable releases, as the size and reach of the ZeroMQ contracts grew. Yet by moving to a more organic process we found that our master branch was stable most of the time. Banning breaking changes had the effect of reducing the impact of errors. Fewer large changes means fewer bugs in released code. Today, we make stable releases of core ZeroMQ projects like CZMQ by tagging master.

If you look at libzmq (the original "ZeroMQ"), you'll see a single header file that defines the API contract. This is the most significant to most users, since it is what they write code against. If you read that file, you'll see it is rather chaotic and unstructured. It ends in an "undocumented" section marked "use at your own risk".

There is no ZeroMQ "team". Change to the code base (in all ZeroMQ projects that use our C4.1 process) is driven by a distributed process that collects real world problems from users, prioritizes them, estimates their importance, and then rapidly tests solutions against them until an acceptable solution emerges. The emphasis is on low latency and accuracy.

Since change is driven by use, as a product gets wider use, change also hits a wider front. We're not talking deep, dramatic shifts (though some threads may sound like that). Rather, we're talking about tiny evolutionary changes happening in many areas.

Change to a system creates shifts in knowledge and costs, resulting in "vertical tipping points". This is when a series of answers to a whole set of problems can suddenly be recast as a new model, with new abstractions that hide much or all of the complexity that has grown up.

For example in ZeroMQ, to make a workable client-server pattern, you take DEALER and ROUTER sockets, configure them in various ways, add heartbeating and sessions and timeouts, and perhaps discovery. We start to see that this pattern happens over and over again. And suddenly we hit a tipping point where it makes sense to design "server" and "client" classes.

We accumulate change horizontally, and then we refactor vertically. This is a classic and effective pattern for building software systems. The outcome is, however, that the library is never really stable, and never finished as long as people are using it in new applications.

This seems nightmarish, and yet it is not. The trick is to understand two things. First, changes are naturally localized. Even if zmq.h looks like one long contract, it really covers several different areas. And these areas each have their natural cycle through experimentation to stability.

Here is the key understanding: a real product has multiple contracts, that change independently. Take CZMQ, which has a better-designed API than libzmq. This library has about 40 classes, each with its own contract.

When we claim "version X is stable", we are claiming either "all contracts in version X are stable". In the reality of popular products, the only way to guarantee this is to lock all contracts into synchronization, by force.

From our experience, we want a decentralized, lock-free development process. That has proven to be the fastest, most accurate, and most enjoyable way to work. This process is incompatible with coercive synchronization of all the contracts in a widely used product. Thus, it is incompatible with software versions.

Since every distributor insists on stable versions of code, there are of course answers to this puzzle. One is to reduce the size of products so they do implement only one contract, which can be coerced into stability. We get an explosion of libraries (we'd need 40 to cover CZMQ), and we start to depend on power and force for our work (something that's both distasteful and from experience, unproductive). "You cannot make this patch because we're making a stable release" is a failure, IMHO.

Before I explain another answer that does not depend on coercion or fragmentation, let's recap what we really want to achieve (apart from making packagers and Enterprise customers happy):

• We want a real-time pipeline connecting our software economy, so problems flow in one direction, and answers in the other. Annual releases just don't cut it. We want real-time, continuous delivery of small, incremental patches.

• We want guaranteed, testable backwards compatibility with stable contracts. I'd love to be testing libzmq master against the test cases from older stable releases.

• We want freedom to refactor vertically at any time and without upfront coordination. This means creating and delivering (so, testing) new contracts at any time.

• We want a coercion-free process where power is used only to regulate contracts and cheats. This is essential to get scale. Upfront coordination is the mutex of software development.

OK, we probably want a lot more than this, however it's a good list to work with.

Contract Life-Cycles

The first step to enlightenment is to see that contracts have a life-cycle. This applies to all contracts, both in software and in the real world. The contract life-cycle is not an invention, it's a feature of real world economics, and a useful one for software engineering.

Public contracts most definitely change over time. Changes are not always backwards compatible. I cannot plug a horse into the house electricity mains. My 3/4" flat-headed screw does not fit the 1/2" hole in the wall. If I plug a 110V US toaster into my 220V kitchen in Brussels, the toaster dies rapidly and dramatically.

Do I care what version the toaster is? No. Do I care what contracts it implements? Most definitely so. These are written in bold on the packaging, and those claims are wrong, I can sue the distributor and manufacturer for fraud.

To be of general use, a standards must be stable. Of course there can be variation. I can plug a 220V toaster into a 210V, 220V, 230V, or even a 240V circuit without risk. Standards can have variations, as long as these are compatible with stable versions. This goes one way: we do not usually update shipped products over time, as standards change.

How do we know when a contract is stable enough to qualify as a "standard" then? Well, for public contracts (the ones we care about and strive to make), stability is a negotiated peace between users (clients) and suppliers. Change can lower costs, yet can also create costs as the contract affects more points. So eventually the cost/benefit trade-offs drive both sides to agree that change now stops.

If you look at the CZMQ classes, you'll see some are marked "deprecated". If you look at the ZeroMQ RFCs, you'll see more elaborate tagging.

The life-cycle of a contract can be obfuscated, by not documenting it. Let's assume our contracts are written down, not embedded in the software somewhere.

A new contract usually has no implementations. Until someone signs a contract (demonstrates it in code), it's just speculation. I call these "raw" contracts. Since no-one is using a raw contract, and it's usually incomplete and immature, it changes a lot, without hurting anyone.

When you can demonstrate a contract in code, it gets some status. I call these "draft" contracts. A draft contract will change a lot, often in tandem with its implementation. This is fine, and hurts no-one.

When two or more parties implement a contract, and especially when there are third parties depending on the contract, however, things get serious. I call these "stable" contracts. Change to stable contracts needs upfront agreement. You cannot rely on this, except in young stable contracts with few users.

Eventually a stable contract gets replaced by new draft contracts. I call these "legacy" contracts. Changing legacy contracts is just an all-round bad idea. Finally, when new stable contracts emerge, the old ones become "retired".

Sometimes, new stable versions of a contract can talk to older legacy versions. Alternatively, software may implement both older and new versions of a contract.

There are three key points here:

• A realistic software system will implement many contracts, rather than just a single one. Whether or not these contracts are documented, a successful software system will, over time, add more and more contracts to the set it implements.
• Each of these contracts goes through a draft-stable-legacy-retired life-cycle, which can be less or more formally defined.
• While the life cycles of the different contracts may drive each other, they are naturally independent. To force all contracts in a system to the same state requires significant coercion and cost.

Hopefully you see why trying to force an entire product into one state does not scale well.

Defining Interoperability

Interoperability is the ability of products X and Y to work together. My WiFi router has a list of USB 3G modems it works with. This seems good, right? Actually, no, it's a rubbish approach. It's like saying, "this toaster is compatible with versions 3.1 and later of my home".

To define interoperability as a relationship between products is to assume that the set of products is limited and testable, and exists only within a given time and space. Yet real systems stretch over space and time. When I build a house, I equip it slowly, over years. The process of improving it never stops. The only way I can get the right toasters at the right price is to search widely and slowly (i.e. over space and time).

It's far better to say, "my house has a 16A 220V-50Hz electric circuit" and "this toaster draws 4A at 220V-50Hz". That is, to define an electricity contract, and then define interoperability as "respects this contract". The supply of toasters that implement that contract is huge. The supply of toasters that implement House V3.1 is small, if not tiny to the point of "only a Russian billionaire would consider this."

A list of "compatible devices" is a red flag that tells us, "this area is not standardized and thus you can expect to pay many times what you should."

Ditto for software. When you install a software product you are making a long term investment. The accounting of that decision depends on the contracts the software respects. It is interesting to know what's inside the software, for sure. Is it fast, does it crash often, can it handle many users? However it's far more interesting to know what standards it implements, and how well.

Software systems that do not respect and promote public contracts create a sense of unease and disquiet. It does not really matter whether they're open source or not. Technically and economically, the alternative to an architecture based on contracts is one based on coercion.

I believe that in removing the concept of software versions, and focusing instead on the contracts that a system implements, we can get a more accurate description of what we are getting, as consumers. Further, it's a description that is mechanically testable, and could even be held up as a legally binding statement by software producers.

The Software Bill of Materials (SBOM)

Let me introduce the Software Bill of Materials, or SBOM. The SBOM is a testable statement of the contracts (e.g. RFCs) that a software product implements. The SBOM is a claim of backwards and forwards interoperability with other products, based those product's own SBOMs, rather than product version numbers.

To implement an SBOM, a software project has a section in its README that says, for instance:

## Software Bill of Materials This software implements the following public contracts: * project/contract/name/state * ...

If an implementation supports several versions of a contract, it should list them all. Thus libzmq, for example, implements ZeroMQ RFCs 13, 15, and 23 (three versions of ZMTP).

This may seem optimistic, yet the goal should be that the SBOM defines *fully* the behavior of the software system as far as it affects other software. As in any product, there will be design trade-offs, e.g. performance vs. resource consumption. If these are relevant to the outside world, they should be defined as public contracts.

The SBOM depends on a very cheap, fast way to develop RFCs through their life-cycle. I developed and tested this starting with the Digital Standards Organization (Digistan) in 2005, and then with ZeroMQ RFCs up to the present day.

An RFC is simply a contract that's published on a public platform. Under the Digistan rules, such documents use the GPLv3 license, so can be freely forked and improved. This solves the problem of capture ("No, I won't improve my RFC to make it easier for you to implement").

We've additionally been using man pages as contracts for APIs, and this works up to a point. Both libzmq and CZMQ as well as other projects do this. It is however too easy to change the contract together with the software, and break existing applications.

Long Live Version Numbers!

In ZeroMQ we still use version numbers for two main reasons:

• We don't have fully testable or tested contracts, and so code on master still has faults that emerge over time. We reduce the risk by forking master to a separate repository and slowly fixing this, without taking new functionality from master. This emerges as major.minor.patch.

• Version numbers are powerful tools for marketing, and especially for replacing old rubbish with new, exciting, rubbish.

• If (if!) we remove support for retired contracts, the version number would make this clear. "Version X no longer supports the frongle interface."

We can still provide version numbers by tracking the SBOM. I'd simply number the SBOM changes from 1 upwards, unless we find we need more detail. Perhaps the repository commit number within the SBOM number.

Worked Example

Since I like to eat my own cooking, I'm going to start using the SBOM approach for CZMQ, and if it works there, spread it to other projects. CZMQ implements a variety of ZeroMQ RFCs, and its own API contracts.

From experience we know that a single CZMQ class can contain multiple contracts, each with its own state. E.g. zsys contains draft methods, stable methods, and retired methods. I'll to tag these API contracts as "draft", "stable", "legacy", or "retired". Further, when we change contracts (draft and stable, probably) we'll number those changes.

So the SBOM for CZMQ might contain entries like this:

* czmq/zsys/udp.1/stable * czmq/zsys/presets.1/stable * czmq/zsys/interrupt.1/retired * czmq/zsys/interrupt.2/stable * zeromq/RFC/4/stable

Where each entry in the SBOM matches a test case living either in CZMQ, or (ideally elsewhere). Incidentally, moving the tests out of the project solves the common question with contract driven development: "what if someone breaks a contract and then modifies the test case to hide that break?"

Conclusions

As I've explained, a living software product will usually deliver a mix of draft, stable, legacy, and retired contracts. Rather than try to assign a "major.minor.patch" version number to the overall mix, to indicate stability and maturity, it is more sensible to list the contracts explicitly in the living code (on git master).

Implementations of draft contracts carry no promise of interoperability. It is fair to assume that the market for draft contracts is small, and the cost of change is low. Implementations of stable contracts are guaranteed to work with all other implementations of the same stable contract. A stable contract should be testable, and the claim of a vendor to implement it could be legally binding.

By listing all contracts that a software project implements as a "Software Bill of Materials", or SBOM, we make it possible to test every single master commit against these contracts. While today we tend to run CI tests within a single project, it would be better to have external test suites, one per contract. Since people will demand version numbers of some sort, we can version the SBOM, and commits within that. So version 12.122 of a product will be SBOM version 12, commit #122.

As a worked example, I will experiment with CZMQ, one of the projects I'm mainly responsible for.

by pieterh

Why "Living Systems"?

Since the beginning, life has relied upon the transmission of messages. — RFC 3164 (syslog)

Wikipedia writes, "Living systems are open self-organizing living things that interact with their environment. These systems are maintained by flows of information, energy and matter." The term was originated by psychologist James Grier Miller to formalize the concept of life.

I want to use the term to define a new metaphor for software systems and organizations, the two types of system I'm most interested in. The two are more than just similar. Software is the product of an group of people, and as Conway observed, the structure of a software system mimics the structure of the organization that produces it. I've written that "the physics of software is the physics of people," and by that I meant psychology.

Most software products today are highly planned, and they fail as living systems. They are essentially dead on delivery, sold by force and bluff. For a software to be a "living system", it must be used by the organization that builds it, and it then lives or dies along with that organization. An "organization" can be much larger than one company or one team. It can consist of thousands of teams, businesses, customers and suppliers, connected in ineffable yet vitally real networks.

Nowhere is this more clear than the Internet, a Living System of software, of people, of businesses and other organizations. The organization that built the Internet is nothing less than human society itself. There are many Living Systems, all around us. It is a strangely simple truth: that the better we get at writing large-scale software systems, they more they resemble the real world around us.

In contrast to Living Systems, we have Planned Systems. It is far easier to plan a system than to grow it. However, plans are inevitably built on wrong assumptions and poor judgments. Planned Systems look attractive and efficient from some perspectives, yet they inevitably fail catastrophically. Real life provides many examples, such as collective farming, planned cities, Microsoft Windows 8, and so on.

In the software business, this Living vs. Planned dichotomy is best expressed by free software vs. closed source. Free software (and its corporate cousin open source) usually grows out of real use, where as closed source is usually planned. This is mainly why I don't work on closed source: it dies rapidly and predictably. I prefer my work to survive as long as possible.

I'll make a few broad claims, starting with: the most successful large-scale software systems are Living Systems. That is, in a competitive market, a Living System will wipe out any competing Planned Systems. It will recognize and solve real problems faster, cheaper, and more accurately. If your business depends on a Planned System, you are vulnerable to attack by a Living System.

The second claim I'll make is that this also applies to organizations. If your company is a Planned System, it is already dead. Whereas if your company operates as a Living System, it can dominate its market. Interestingly, when two Living Systems meet, they don't usually fight. Rather, they specialize into different areas, and they then merge to form a single Living System. Competition and conflict usually work for the benefit of the Living System, even if individual components fail.

Let me take this question of conflict and competition further. Of course individuals do compete, and quite harshly sometimes. This is our biological imperative. However we also have a biological imperative to collaborate, a far more profitable strategy, most of the time. A Living System embraces competition between individuals, and survives the failure of individual components. It actually depends on that process of competition and failure. A Planned System is essentially trying to act as a single individual, and cannot tolerate internal competition, nor failure of individual components.

What do Living Systems Look Like?

A Living System consists of a large number of loosely-coupled components. It is essentially spread out in space (thus, "distributed"), and in time (thus, "asynchronous"). That means things happen in unpredictable places, and at unexpected times. To a central planner, this looks like dangerous chaos.

In a Planned System, by contrast, the times and places of events are meticulously scripted. The focus is on "command and control," where decisions are made centrally and communicated to the structure. Planned Systems are essentially hierarchical, for this is the optimum way to communicate decisions rapidly down from the top.

We build a Planned System, whereas a Living System grows itself. My goal is to learn, and to teach, how to grow Living Systems artificially. I'm studying the genes and patterns of care and nurture for a self-growing system. In fact I'm talking about creating artificial life, and artificial intelligence, though in a shape that traditional AI researchers might not recognize. I don't believe individuals components — including you and me — can be "intelligent" at all, except in a narrow and superficial sense: intelligence is a property of systems.

Living Systems are typified by their lack of central planning or decision making. Look at a software project and ask, "who is the designer?" If there is a clear designer, individual or organization (and there almost always is), that is a Planned System. A Living System has no designer, no road-maps, no clear future plans except "survive and grow."

A Living System looks more like Adam Smith's free market than Stalin's Five Year Plan. Economics, politics, and psychology are as important — perhaps more important — in growing successful Living Systems as technology. A free market depends on several key things: clear laws, standards, contracts, and fair regulation. A Living System likewise depends on these.

So whereas a Living System has no central planner, it may have central regulators. Let me explain the difference. In a planned city, a committee decides where to build schools, homes, factories, offices, railway stations, shops, sports facilities, and so on. In a Living City, all these are decided by independent agents (school boards, businesses, home owners, etc.) and the city regulates the provision of electricity and clean water, the disposal of garbage, and so on. Further, the city runs police and courts, as regulators, to dissuade criminals and cheats.

A regulator makes laws that define a fair market, and then enforces those laws. Units of measurement, currencies, contracts, and such. In software systems, these laws are, for instance, the source code license, and contribution policy. A fair market lets anyone create a new venture, and compete with other ventures. To allow true competition (meaning, free choice by customers), clients can demand clear contracts, which in software terms are documented APIs and protocols.

The DNA of a Living Systems is essentially a set of regulated contracts. Thus the Internet is grown out of a set of RFCs (protocols called Requests for Comments), regulated by the Internet Engineering Task Force (IETF). Living cities are grown out of criminal and civil laws, standards for water and power and waste, transport, and so on.

If all strategies were honest, there would be no need for regulators. However any Living System will be vulnerable to cheating strategies. A certain segment of people, for instance, are systematic or opportunistic cheats. Given a market, they will always seek a way to convert value to their own benefit, even at a higher cost to others. They will lie, steal, deceive, coerce, and so on.

Without resistance against such cheats, the market will suffer, and the system will eventually die. Top-down authority is one defense against cheats. However it has a significant vulnerability: cheats can, and often do, capture the authority itself. In Living Systems, cheats can try to capture the regulators, and this happens in real life all the time.

When cheats capture the regulators of real-life Living Systems, the usual response is to move away, if we can. In open source software systems, we can fork and continue under a better regulator. This is why forking is an essential freedom, rather than a failure. Since forking can also be a strategy for capture, a fork-safe license (GPL or similar) is best for Living Systems in software.

Living Systems grow, constantly and organically. This is their most visible trait: the lack of the usual massive construction efforts. Rather, you will see a smooth flow of small changes. It may seem boring or unambitious. However, it is a better algorithm for survival. A Living System must do two things. First, it must solve some profitable set of problems. Second, it must adapt and change over time, as its environment changes.

Shifting a Planned System to cope with a changing environment is very hard, often impossible. Resources define power. Thus, Planned Systems actively and aggressively resist change, deny it, and when it becomes inevitable, they die catastrophically. However, a Living System feeds off change. It makes no distinction between exploring the landscape of problems of "now" and of "tomorrow." It grows through continuous learning. To actually destroy a Living System you have to do widespread damage to it, which is hard when a successful Living System has spread wide.

To a Living System, small amounts of damage are indistinguishable from normal activity. In fact Living Systems thrive on challenge, so long as it is not overwhelming. Challenge is what allows components to compete, and develop better answers. What does not kill a Living System makes it stronger.

So, as Living Systems learn and move quickly and opportunistically into new areas, they will tend to thrive and grow dominant, wiping out any competing Planned Systems. They can react rapidly, shifting resources around to areas where they are more valuable. And since they do not need any upfront coordination to act, they can scale to any size. Zero upfront coordination means infinite scale.

Components of a Living System

Let's now look at the individual components of a Living System. Remember that a Living System resembles a free market, where components compete to provide some identifiable and measurable service. The components of a Living System have some traits that set them apart from the components of a planned system.

Every component of a Living System has a clear set of owners and investors, and ownership is usually highly localized (in contrast to a Planned System, where all components have the same owners). Components organize into chains of suppliers and clients, and they have identities, names, and addresses, so that clients can find them. One classic way to cheat is for one group to provide a poor quality component that claims to be a high-quality one. Thus the regulator may have to enforce identity, and protect investment in an identity.

Components are, as far as possible, location independent. That creates a larger, and more efficient free market. It means that we strive for location independence as a feature of our Living System. This is contrary to a Planned System, where location is highly significant, and where there is little or no competition between components.

Similarly, components may come and go in time, quite arbitrarily. There are no guarantees that a component we depend on today will still exist, or be available tomorrow. This may sound fragile, yet it is highly robust. Rather than depending on specific components, we depend on contracts. If our need is real, there will be many alternatives. If one disappears, another will take its place. If you miss one taxi, you will catch another.

Components are highly independent, decoupled from one another. That is, they exist and change at their own rate, in their own direction. A change in one component is essentially invisible to another component except through its public interfaces. This freedom is essential to a free market, driven by specialization and trade. Thus one component may focus on speed, while another on security.

Since there is no centralized decision about what components exist, nor who makes them, they will be highly heterogeneous, and this diversity is essential to the intelligence of the overall system. A set of diverse components in a Living System, connected in a free market, will solve large problems faster, and more accurately, than a monolithic Planned System.

Components are abstracted, meaning they may represent entire systems in themselves. For instance a web address can represent a single, small piece of software (one web server), or it may represent a massive infrastructure (an Internet business). It is up to each group of owners to decide whether they build Living Systems or Planned Systems, in turn. A Living System will happily embrace Planned Systems as components. The opposite isn't true.

Components avoid upfront consensus, also known as "shared mutable state". Every component has knowledge, and they may share knowledge, yet they do so asynchronously. So while the Living System represents a large, coherent pool of knowledge, there is no guaranteed consistency between components. This may seem paradoxical. Surely every person in a meeting, for instance, agrees on the agenda for the meeting?

In fact meetings, with their agendas and minutes, are the epitome of the shared mutable state that a Planned System depends on. Planned Systems cannot function without systematic upfront agreement. In concurrent software design, we use "locks" to achieve the same result. It is provably true that a software system that uses locks to share state between components will not scale. You can try to make distributed software as a Planned System: it starts easily yet scales poorly, if at all. Whereas a Living System takes a little more thought at the start, and then scales without limit.

Finally, components are lazy and opportunistic. They only work when there are tasks waiting, and they only change and grow when there are new, profitable opportunities. This means components can remain lightweight and minimalistic. Further, they can solve the "problem landscape" much more accurately, without excess baggage. In a Planned System by contrast, components are built upfront, on the assumption of future problems, or at best, knowledge of past problems.

An example: in a planned conference, the organizers choose speakers on certain topics, based on their experience of the previous year. Now, one month before the conference, a significant event drives public demand for speakers on a totally different topic. How long will it take the conference to react? A participant-driven conference can react in real-time, whereas a planned conference will take a full year to respond.

Protocols of a Living System

The components of a Living System are connected in relationships. Each relationship consists of a flow of information, knowledge, or requests, in both directions. The best way to model such relationship seems to be as discrete events, or "messages," that carry a formalized set of interactions we call "protocols."

In natural Living Systems, we also see messages and protocols. Cells, for instance, communicate with chemical messages. We humans appear to communicate with a set of protocols that underly our human languages. For instance, male-dominated hierarchies are a consistent feature of human society, suggesting that the command-and-control protocols they depend on are built-in to our minds, not learned. I'd hypothesize that the male mind, driven by the ancestral need to plan hunting parties, is responsible for Planned Systems.

Protocols have a number of common patterns. We see broadcast protocols where one component signals to many listeners. A broadcast protocol is typically one-way. The signaler may, in rare occasions, get feedback from a few listeners.

We see one-to-one protocols where two components exchange knowledge, tasks, requests, and so on. One-to-one protocols can be more or less chatty, and ideally are fully asynchronous. Chattier protocols take longer to conclude, thus raising overall "latency." For example if I'm cooking a pizza and I have to confirm every ingredient, it will take longer. "Do you like mushrooms?" "How about garlic?" "Ok, what kind of cheese do you prefer?"

The ideal relationships aim for lowest realistic latency, since the latency of the overall system is the sum of the latency of its entire supply chain. That is, if I'm making a meal, and I have to spend one minute solving the "pizza" issue, that adds one minute to the overall preparation time. In an asynchronous low-latency dialog, I'd ask all the questions at once, and deal with the answers as I got them back, one by one.

To make effective asynchronous systems we need queues, and smart queuing strategies. Ideally, we have queues at any point where messages may arrive, and we push messages as close to their consumers as possible, to reduce latency. We need strategies to deal with full queues (space is not infinite): it might be to throw away older messages, or to pause the sender (this works for one-to-one dialogs, not for one-to-many). We may need multiple incoming queues, one per flow, and the ability to wait for a message on any of these queues.

The protocols of a Living System are highly ritualized. They implement formal contracts. If I ask, "Do you like garlic?" then I expect a yes/no answer. A discussion about the weather is a breach of contract. When we're growing our own Living Systems, we have to write the protocols down, so they can be learned and verified. The simpler and clearer, the better. Complex, arcane protocols are expensive to learn and implement, which distorts the free market.

Some Living Systems use earned trust, together with identity, in place of verifiable contracts. This can be a valuable short-hand, especially when exchanging knowledge, though it is also vulnerable to cheats (frauds). An alternative is to ensure that every contract is verifiable, backed by meta-contracts on performance. This is often better for trading work. Any taxi driver is fine, so long as drive to the right address and don't over-charge. However we want our news from trusted sources.

Once we have testable contracts we can deal with violations. One strategy is to fail, and let someone else deal with it. Another is to discard that peer and try another. However, after a contract violation, you generally don't want to continue blindly, as that can cause wider damage.

Case Study: the ZeroMQ Community

The ZeroMQ community is a Living System of people that builds a Living System of software (the software collection under the same name). Though I originally designed the ZeroMQ community with most of the properties of a Living System, it only really came true in early 2012, when the community rejected its central planners.

The community consists of a large number of loosely-coupled projects that share a common goal, which is to provide the queuing and messaging needs for other software systems. I've argued, and believe, that only a Living System can use ZeroMQ optimally.

The ZeroMQ projects are connected into a supply chain by formalized relationships, over APIs and wire protocols. We spend a large amount of time documenting these APIs and protocols, and ensuring they are testable. Indeed, we do not usually document the internals of components, just their external APIs.

There is no central planning nor coordination. Instead, each project evolves organically as its users invest in patches and improvements. By making this process simple, the ZeroMQ collaboration contract ensures that the ZeroMQ organization expands to include all its competent users.

Anyone can start a new ZeroMQ project, or fork an existing one, for competition or experimentation. As a community we encourage this, and so we have multiple competitors at most levels. This works well in practice. The basic license is LGPL v3 or MPL v2, ensuring that forks are always safe (patches can flow in both directions).

The regulator in the ZeroMQ community is a self-elected group, headed by iMatix, the firm that developed the original software. There is not much regulation needed, except to stop abuse of the name "ZeroMQ". Clear documentation of protocols is sufficient to allow clients to verify their suppliers.

ZeroMQ is highly scalable. The cost of adding a new project is close to zero, apart from the discovery cost. Projects communicate asynchronously, using GitHub issues and pull requests. There is little or no upfront coordination. We review code after the fact, and fix poor code through further patches rather than discussion.

ZeroMQ's full transition to a Living System was hard because we had no prior successes to imitate. The bulk of free software projects still depend on significant planning. To go against standard practice was seen as highly eccentric, if not actually insane. The loss of key contributors — who had provided the authority that central planning depends on — was seen as potentially catastrophic.

However the ZeroMQ community rapidly expanded into the space left by the central planners, and flourished. We disproved the theory that central planning was essential to quality. In fact we found that without central planning, the software improved significantly in quality and in accuracy. Whereas the ZeroMQ development branch had been highly unstable, experimental, and discordant with users' needs, it became mostly stable, trusted, and a close fit for users' needs.

Today we can hold the ZeroMQ community as a worked example of how to do Living Systems "right." It is all the more valuable as data since there have been numerous attempts to replace it, both by the fleeing central planners, and by other teams. Noticeably, every Planned System that claimed to be "better than ZeroMQ" has failed, whereas every Living System that began by competing with ZeroMQ ended up becoming a valuable part of it.

Transforming into a Living System

Can we turn a Planned System into a Living System? If we assume we have the technical right (consensus from enough participants, or legal right through software licensing), what are the practical requirements?

The most difficult part will be to get the size of components right. This will often mean breaking up existing components, and creating new ones. That can be catastrophic if done in too many places at once. Thus, in a larger migration, you would start in one area, refactor that, and then grow the resulting culture out.

Components are usually sized around the people, so a good size is "the work that a few people can do." The scale of a Living System comes from adding more components, and allowing them to use and replace each other in whatever fashion makes sense locally, not increasing individual component size. A component is too small when it cannot provide a full service by itself, and it is too large when it does not focus on one thing.

Finally, you need the contracts. For a software system, we have had good results simply by taking the the ZeroMQ C4.1 process contract, and using that together with a code style guide and the software license. For several reasons, I strongly recommend a share-alike license, such as LGPL (my thesis is that if you use a leaky license like Apache or BSD, you in fact won't get a successful Living System at all).

Launching such a Living System in the past was difficult, as self-organizing software ecologies were poorly documented and little understood. We lacked empirical evidence that processes like C4.1 could work, let alone work so well. As far as I know, this was the first documented contract for a Living System in software.

Economics of Living Systems

How do we make money from free software? It is a question I'm often asked. The answer comes in various forms depending on whether I'm talking to individuals, to small firms, or to large firms.

A key understanding of Living Systems is that they are essentially about economics. No component exists for random reasons. However, to offer a choice between selfishness and altruism is a false dichotomy. Living Systems are driven by selfishness and altruism at the same time. It is a basic theory of economics: by selfish specialization and trade, we create common wealth. It is the human species' superpower: specialization and trade at a massive scale, between individuals, families, generations, villages, cities, and entire regions.

A Living System is owned by all participants, so it can be harder to measure its value, whereas a Planned System, owned by a few at the top, can have very visible value, to its owners and outside observers. However the overall value and economic power of a Living System will always overwhelm any competing Planned System. A Living System can be incredibly profitable, its profits are just widely distributed among all its participants.

This is the first answer: a Living System can kill competing Planned Systems, and thus liberate large amounts of captive value, which can be absorbed by the Living System. We see this in real life, where free market economies out-perform planned economies, leading to movement of skilled labor (value) from the latter to the former.

The second answer is that we can build new markets on top of successful Living Systems, that are impossible on Planned Systems. The Internet is a clear example of this: it has enabled a massive new economy that was impossible on older networks. Those new markets can be very profitable.

A Planned System can only survive by taking value away from its components. In many ways, it resembles a cult, and depends on cult techniques like brain washing, where a few prosper at the expense of many. Planned Systems are inherently unethical, as well as unsustainable. There is an inherent morality in a fair and free market, despite the large number of of Planned Systems that claim to represent "the market."

Conclusions

In this essay I've looked at artificial Living Systems, which imitate and can be modeled on real living systems. Living Systems are spread out in space and time. They consist of large numbers of independently owned components that work together, competing and collaborating, in a free market for services, labor, resources, and knowledge. These components evolve independently, under pressure from their market. They live and die according to their success in finding accurate answers to real problems faced by their clients.

The components in a Living System communicate asynchronously by passing messages around, in various patterns. These flows are highly ritualized, in the form of protocols. The more accurate the protocol, the easier it is for clients to choose suppliers freely, and the more efficient the market.

A Living System has no central controlling owner, though it may elect authorities to regulate (define, and enforce) contracts. It has no single points of failure. Rather than treating failure as exceptional and to be avoided, it uses failure as a basic learning technique. Inaccurate components are allowed to fail and are discarded rapidly, and replaced by more accurate components.

Living Systems grow by learning, into supply chains that connect components to the external environment. We can measure the efficiency of a Living System by looking at overall latency as a problem enters the system, and a response emerges. Such latencies can vary from years in Planned Systems to hours in highly adaptive Living Systems.

Living Systems thus organize opportunistically, accurately judging the relative cost of a given problem, and the value of solving it. Unlike Planned Systems, they are driven by live data rather than assumptions, beliefs, or old data. This lets them operate more accurately, faster, and cheaper than Planned Systems.

To build a large scale Living System in software, build a Living System of people. The two will co-evolve and done correctly, will dominate any given market. Whereas competing Planned Systems will fail as whole units, competing Living Systems will tend to specialize into different areas, and then merge into a single unified Living System.

For further reading on similar topics, buy the book "Culture & Empire: Digital Revolution" from Pieter Hintjens on Amazon.com, in ebook or paper form.

by pieterh

Why Capture the Flag?

As Ars Technica says, "It's easy to give something away when you're in last place with zero marketshare, precisely where Android started. When you're in first place though, it's a little harder to be so open and welcoming."

Android is, to be fair, largely Google's investment. You could argue that they are entirely justified to turn it from an open system into a closed one, and you'd be right. However, it is like arguing that a central bank is entirely justified in issuing too much currency and creating devaluation. Sure, there is a justification. However there is also a cost, paid by other people. The question is not, is this act justified, but is the price paid by wider society acceptable, and if not, how do we prevent it?

Android is, like any "open source" system sold to the market on that basis, common property. When someone privatizes it, they are increasing their profits, like a money-printing central bank, at the expense of everyone else. By forking Android applications like search, calendar, music, and making their own better versions, Google is competing with other firms using Android on their devices.

The question of capture, how it happens, and how to prevent it, is especially important if you are not Google, i.e. if you are the user of, or a contributor to, an open source project. Android contains many patches from other firms, like LG, Samsung, and so on. As Google turns the operating system into its own private garden, those patches start to be used against the very people who made them.

I believe Google is making a huge mistake in moving the goalposts like this, simply because it will encourage competition against Android. However, that's not my point. I'm just interested in applying any lessons I can learn to my own work, and my own projects.

Two things stand out:

• Out of pure self-interest, I will not contribute to an open source project that does not guarantee me, as contributor, that my patches and changes will never be turned into private code, and used against me.

• Out of a sense of ethics, I will never create an open source project that does not provide these guarantees to anyone contributing to it.

The Use Case

Let me be very explicit about the use case. It is the Android case: one firm starting an open source project as loss leader, to break an existing market, and asking for help from others to do so. It is a classic strategy and can be very successful. However this is most definitely not the same as a student's research project, a "let's open source our legacy payroll system" dump, or a "five of us got together in a garage and decided to make a new framework" case.

These overlap, and I think the lessons here do apply more widely (and I certainly apply them systematically) yet again, my use-case is the "open source as market breaker" one.

The important thing about an open source market breaker is that it depends on a community to pitch in. Any market follows a power curve where a few players dominate the market, and a majority of players are frustrated. It's by promising this frustrated crowd a way out, that you can convince them to invest in something new and open and potentially game-changing.

Most open source is a failure (seriously, go read some random GitHub projects and see how many are relevant), and even most successes are modest successes that barely matter. As long as there's no serious shift in power, the project can remain a potential market breaker for a long time. It can look very stable and happy. Well, it's easy to be friendly when there's no money on the table.

If and when the project succeeds, the game changes, and the clever guys who launched the market breaker seek to pluck the fruit, and keep it for themselves. And only now do things get interesting.

A Level Playing Field is Not "Restrictive" FFS

There are several ways to capture an open source project, including trademarks and patents. I'm going to look only at copyrights, because this is the most common case. The key agreements that govern the copyright status of an open source project are (a) the license and (b) the contribution policy.

It's a common misconception that "open source" means the code cannot be captured. That is simply wrong. Broadly, there are three types of agreement for copyright:

1. A "locked down" license that does not allow remixing, in other words, classic copyright plus some restrictive license.
2. A "free to take" license that allows one-way remixing, such as Apache/BSD/MIT.
3. A "share-alike" license that enforces two-way remixing, such as GPL, LGPL, and cc-by-sa.

Imagine a DJ who releases a popular beat under the "free to take" model. A major record label takes his beat and makes a remix, and releases it. It becomes a massive hit. Now that new version is locked down. The DJ cannot remix that new work, and may find himself unable even to play the remix. Sure, he can take his old version and improve it, yet it's the commercial version that will get the money.

I trust you see what I am getting at here. Even the best individual talent cannot compete equally with a large firm with marketing and money. The only way I know to guarantee a level playing field in a war of control over culture is a bilateral guarantee of remixing. Bilateral means it goes two ways.

When people call that guarantee "restrictive", I sigh. It's like calling the lock in my car "restrictive" because it stops others from making my car theirs. To call protection from thieves "restrictive" is… well, a failure to think things through, at least. Making rules apply both ways is not restrictive, OK!?

How Does the Capture Work?

Let's clearly restate the goal again, with this exercise. It is to prevent the capture of an open source project by someone with lots of money and power, who is determined to harvest the fruits of the project for their own benefit, at the cost of the community who helped make, or who made the project. I don't care how "justified" such a capture might be, it's what I'm explaining how to prevent.

The license and contribution policy are two halves of one puzzle.

Who owns the copyrights? Are they "centralized" by the project founders, or are they shared by all contributors? It's a vital question. If they are centralized then it is a trivial exercise to buy the copyrights, fork the project, change the license unilaterally, and move off in a closed direction. However, if the copyrights are shared, i.e. many people own the work, together, you need all of their agreement (not a majority, but 100% consensus) to change the license. And that is logistically impossible.

As an aside, if you knew how many people had offered me money for a commercial license for ZeroMQ, you would be astonished (it is a lot). The notion is simple: I sell them a non-LGPL license, they pay me good money, and they make their own versions of ZeroMQ. If I'd not made this impossible, on purpose, a long time ago, I'd be very wealthy. As it is, I have to settle with poor but happy in the knowledge that ZeroMQ will survive me.

Let's examine again the problem with offering commercial licenses to a collaborative work on the side. Imagine a club that hosts DJs, who mix their beats. But the club keeps the copyrights, and sells them to a record label, which makes its own remixed album that the original DJs cannot play for free. So yes, I consider dual GPL/commercial licensing to be a corrupt practice.

No-one will pay for a commercial license for a "free to take" project, since they can just take the code and use it. To some extent I think that is already corrupt, since it breaks the level playing field. A large firm can obviously benefit more from such a license than small teams. Again, imagine your independent DJs against a record label with their marketing and media connections and concert venues.

Now we come to step two of the capture: hire the devs.

"But the code is still free!", people say. Sure. Back to the record label vs. the DJs. Let's say the label hires just one DJ, the key man, and uses him to push the new commercial mix album. Where is the public going to go?

You don't need to hire all the contributors to a community in order to 0wn it. In any random project there will be at most 2-3 top contributors and a large mass of minor ones. Hire the top two, and you can take the project anywhere you like. If the results are remixable, that journey will be entirely fair to those who contributed before. And if not remixable, all other contributors will find their own investments used against them.

Preventing Capture

There is only one model I know that prevents capture of an open source software project, and that is:

1. A GPL-family license (or MPLv2, which works the same).
2. Distributed copyrights.

This is how I construct the open source projects I start, and it's the requirement for any community I join. Your right to make money does not include the right to use my work in a competing product, unless that's reciprocal.

by pieterh

The Castle

The Castle is large, and built over time by hard work. It represents power, influence, and authority. There is one Castle, it has gates, a moat, and probably a nice little garden tended by old men who mutter about dragons and their taste for lemon trees.

In the old days, the Castle was the only real way to build a large structure, mainly because anything built on a smaller scale would tend to burn, rot, or collapse, usually taking the unfortunate occupants, and their pigs and goats, with them.

A Castle organization is built as an endless series of special cases, accumulated over time, sometimes demolished and rebuilt, but always treated as a whole. There's no consistent structure, no regularity, just a plan that is always somewhat out of date. To learn the layout of a Castle can take you years, and it's pretty pointless unless you're one of the people in charge, or a servant, since you can't go anywhere without permission or orders anyhow.

Castle organizations have many levels, and each level has as much complexity as it can hold. Each level has its specialists, in a neat hierarchy from the Duke down to the gardeners. If you look at the blueprint of a Castle, what you see in fact is a hierarchy of Important People, since each space in the Castle is owned by some people.

So, the layout of a large organization, an organigram, is a map of people and their power relationships.

The City

The City is also large and built over time by hard work. It also represents power, influence, and authority, and there is one City. However, it is a collection of many smaller or larger buildings spread out across a landscape.

The City is limited only by natural boundaries, and how dense the ground is (the denser, the higher you can build, which is why sky scrapers are built on rock, and why the skyline of a modern city is usually a reverse map of the geology it sits on).

The City has only a few levels, and you can ignore most of them. Principally, it's a collection of buildings, each with a front door, walls, doors, roofs, and varying numbers of loos. Once you've understood the concept of "building" you have pretty much got it. You have your apartment, your family house, detached, semi, converted loft, warehouse, garage, office. It's not an endless variation. Further, as we make more and more Cities, we use the same patterns over and over again for the buildings.

A City can grow much, much larger than a Castle. Thousands or millions of times larger. You can learn a City without learning its detail, because it's highly regular. I don't mean the street layout, but the fact that there are streets connecting every building, for example. In fact once you've learned the concept of City one time, you can apply it to most Cities in the planet.

In a City, you barely see the people. There are still a few token Castles dotted around but the vast majority of activity happens in a free market. People move around at will. They self-organize and the City represents an endless landscape of independent but connected works, rather than VIPs. There's no-one in charge, which is freaky to anyone coming down from the Castle.

The Difference

A Castle organization focuses on control, unity, coherence. A City organization focuses on a free market of problems and solutions. Our human minds are bad at dealing with the arbitrary complexity of a Castle and its many levels. Cities on the other hand are easy, even fun, to explore.

A Castle consolidates the work of one group of people, perhaps a dozen to a hundred. A single City building consolidates the work of a handful of people. But because the City can hold unlimited numbers of projects, the overall size of the community can be much larger.

The City model will scale hugely because they are easier to learn, so attract more people. If it takes you five minutes to understand the layout of a project, you're more likely to stay and read the README. If it takes you a week, you won't hang around unless someone is paying you.

To build an effective City, it must be easy for anyone to start a project, for people to compete, and to self-organize around problems that crop up over time. As benevolent dictator, you need to set just enough rules and standards that the plumbing works.

by pieterh

All the Wrong Reasons

Today I was trying to get my little Ubuntu notebook to detect a HDMI cable. When you speak at a conference you really want to be sure your display will appear on the projector. Noemie said, you have to reboot it. I said, "no way, it's Linux, it detects this kind of stuff." Reboot it, she said, trust me, it'll work. (I think), she added parenthetically.

So I rebooted the laptop and yes, it worked right away. See, she said, beaming with joy, that's how I get my laptop to work on the TV too.

Anyone who says women aren't entirely competent to learn technology or solve technical problems is ignorant. But at the same time, anyone who says that women are dropping out of the software business because of sexism is also ignorant. Yes, there's sexism, but that hasn't stopped women from striving for hundreds of years for ever more equality and power in the workplace, politics, on the road, in the family. If my daughter (who is half Congolese) ever comes to tell me she didn't get some gig because of racism, my answer won't be gentle.

I've worked in the software industry for 30 years, and I've seen the disappearance of women from software teams. It's quite visible, and quite astonishing, because the companies I used to work in were sexist, discriminatory, macho, and often caricatures of the unfair workplace. All the managers were white men. But when it came to talent, there was never much fuss. You sought the best young programmers, you trained them, you paid them all (men and women) as little as you could. The only discrimination was on age, mainly because of salary costs.

Yet today, there is little or no sexism in the teams I work most with. Most of the people I work with could be any age, gender, background, it's invisible and it is irrelevant. There is no visible gender on your GitHub profile unless you post a photo, or use your full name. True, globally distributed projects like ZeroMQ are perhaps still rare, but in our open source projects there are still very few women contributors. We've worked very hard in ZeroMQ to remove all plausible barriers to contributors. While overall diversity is growing rapidly (with contributors from across the globe, and literally, no discrimination of any kind I can detect), gender diversity seems even lower in our open source projects than any workplace I've ever seen.

It's the same at open source conferences. There are women speakers and attendees but they stand out by their rarity. And again, I repeat, it's not because of discrimination. You never met more tolerant, accepting, meritocratic people than the bulk of open source developers. We literally don't care about the things that make people racist, sexist, or otherwise prejudiced.

So what the heck is going on here? No lack of talent, no lack of economic motivation (programming does pay well), decreasing, not increasing barriers, and still, female participation is falling.

People seem to blame a number of factors (these are taken from a thread on Reddit):

• "Women don't tend to think in a way that lends itself to programming"
• "Programming is one of those professions that has a man stereotype to it"
• "Employers don't target women in coding jobs"
• "Being the only woman in a workplace or classroom is INCREDIBLY uncomfortable"
• "Discrimination is pretty much it. Being the sole girl in a programming class of boys is not very uplifting, and their comments are just ick."
• "Women are generally steered pretty hard AWAY from those things"

I like many of these answers and they feel intuitively right, but they don't seem to fit the trend of more female coders 20-30 years ago than today. Sexual discrimination and gender stereotyping was much stronger in the past. There were much greater barriers to entry. Boys made the same lewd comments to girls in their classes.

So we should, I think, classify these explanations as accurate, but not sufficient to explain what's going on. Indeed, all of these except the "women don't have the technical brain to code" are as plausible consequences of fewer female coders as they are causes.

In fact the biggest trend in our professional choices over the last 20-30 years has been a dramatic increase in personal freedom: we can learn anything, we can work from anywhere, we can be anonymous. There's infinitely more freedom for young girls to learn to code than ever before. How does more freedom fit correlate with fewer female coders?

A more plausible answer came to me when I watched my daughter playing with other kids today. It's not an easy answer and it will annoy some people. But when all the obvious answers have been eliminated, you have to start looking at less obvious ones.

Let's first ask the basic question, "why does it matter?"

Why Diversity Matters

Diversity is much wider than gender, of course. As a social species we're great at specializing, dividing up work and then collaborating on a grand scale to solve huge problems. But this strength is also a weakness. When too many people have gone to the same schools, eat the same food, speak the same language, read the same papers, then you get something like France, if you're lucky, and World War One, if you're unlucky: an elite that makes insane decisions because they have no diversity.

It's a basic tenet of collective intelligence: diverse crowds think better than homogeneous crowds. The wider your net, and the more different the backgrounds and thought patterns of your crowd, the smarter decisions they make. When it comes to diversity, there's age, background, education, and sitting like a basketball on a table of oranges, is gender. Whatever other differences you can see between individuals, gender is the main one. I think this is something most of us can agree on, though I've seen loud disagreement about the causes, and the consequences of gender differences.

Let's open this can of worms. Some people argue that gender roles are learned, others argue that they're innate. As the father of three kids I'm amused when people tell me we taught our daughter to be female, or my sons to be male. (For what it's worth, our youngest boy has long hair and people treat him like a little girl. Last weekend he chose to wear a pink hat. Let's see how this works out.) It's also confusing how the same people can argue that gender roles are learned, but homosexuality is not. Anyhow, I am getting off track here.

The point is, wherever they stem from, gender differences are real, big, and I think they're incredibly important. To imagine that men and women should somehow resemble each other seems like wishing all people had the same skin color, features, language, culture. It seems an extremist agenda, to want to wipe out individuality in this way, when as I've explained, it's our very differences that make us valuable.

It's no surprise to me that a key sign of a repressive society is gender separation. Women and men need each other not just to make babies, or be friends, or fight, but to think accurately. If Mars is action, then Venus is knowledge, and one without the other is crippled.

I'm by far not the first person to argue that gender diversity is necessary for a healthy software industry. It's not about being competitive since this industry outgrew national borders some time ago. It's simply about ensuring that as a society we're working on the right problems and building our digital economy with fewer wasteful mistakes.

Why Women are Dropping Out

OK, here's the meat of my argument.

Many years ago I observed that men and women tend to have different communication patterns. It's quite striking once you notice it. These are gross generalizations but I think necessary ones if we want to really understand and then fix the problem. Men tend to fit into a stand-up-and-talk-loudly dialog (publish-subscribe), while women tend to talk in a more peer-to-peer manner (credit-based flow control). In the caricature, one guy is telling a group a story, and convincing them to go off on some adventure. Meanwhile two women are catching up on what's been going on.

It's a systematic, repeated distinction. Men and women seem to form distinct types of social network. Facebook vs. Twitter. Sharing photos and stories vs. sharing jokes.

I realized that I've stopped using Facebook and rely on Twitter more and more. Why? Because it's easier. Sharing stories and keeping up with the endless stream of photos and comments that my female friends on Facebook post is so much work. I'd much rather just make another patch to the code.

And here we have it. From men's point of view, women are in fact rather anti-social. And I'm sure it's the same the other way around. As male software developers get more freedom to create the social networks they really enjoy, they build them to be increasingly hostile to women. It's not conscious discrimination but it becomes a powerful barrier just as the flow of stuff on Facebook or Flickr becomes a barrier to me.

I'm as guilty of this as anyone when I arrogantly try to build communities, only to realize that every pattern that works so successfully for male programmers also becomes a barrier to female programmers. It's taken me a long time to figure it out.

Put this another way: in the old offices, female programmers were competing on more equal terms. HR decided who got hired. As programmers, we came to work, sat down, attended meetings, talked to clients, wrote code, wrote docs. Any socializing was separate. It happened, but was not how we worked. But today if you want to join a community like ZeroMQ, the socializing is critical.

Put this yet another way: the female programmers I've worked with in the past weren't just great programmers, they were noticeably better than the men. They however didn't see the business the same way. They focused less on status and visible success, and more on other values. They were more geeky than geeks, which is weird but how it seems to me. And as the tech business opened up and came online, it became harder and harder for them to feel attached to the competitive sport that software has become.

Perhaps I'm reaching here, but bear with me. The one comment on that Reddit thread that made sense to me was, "even though it's behind a computer screen, it's still a community, and being the only female in a predominantly male community is uncomfortable".

Suggesting that even anonymously, it's as hard for a woman to feel comfortable talking to a group of men as it is for a man to feel comfortable talking to a group of women. I've only a vague idea what this "discomfort" could be, but perhaps simply the way men talk to each other, which can be quite brutal, direct, even rude and impersonal.

Consequently, and this is a vast generalization, self-organizing groups will tend to move towards one gender or the other. If the starting point is mostly male, women will drop out. If the starting point is mostly female, men will drop out. It doesn't take social stigmatization, discrimination, or gender-biased talent. Simply comfort and discomfort and ever expanding freedom of individuals to move away from groups they don't feel they belong, towards those where they do.

Closing the Gender Gap?

If we can accept that men and women think differently, and that with freedom, they'll tend to congregate in discrete groups, and that we're seeing the results of freedom, not sexism or discrimination, then this changes the discussion.

There's a good argument to make for putting freedom of choice above other outcomes. In the Soviet Union, programming was as much a woman's profession as it was a man's. Similarly, most workplaces were very mixed. But perhaps that was because people had no choice. We accept gender-biased groups in many other professions such as taxi driving, hair dressing, nursing, teaching, certain sciences, marketing, human resources, music, and so on.

Alternatively, why not create comfortable zones deliberately? By this, I mean literally, some degree of gender separation by default, as we have in many sports. There are some "women only" coding circles. This doesn't seem offensive or patronizing. As a real concrete step, how about programming workshops for girls only? Sure, men can come and teach or help, but they'd be kept in a strict minority and would not come to learn.

In any case, if the reason women don't code as much as they use to is that working in male communities is uncomfortable, there's no point in trying to bring individual women into male workgroups like adding blue dye to water. Until you hit a certain mass, which could be as much as 30-40%, the women will move away to more comfortable groups as soon as they can.

Sexism, racism, and discrimination of all forms is still loud, present, and something to quash. Intolerance of difference is a bad thing. Ignoring difference doesn't work, because it isolates large chunks of your potential participants. Trying to eradicate difference leads (in the least horrid cases) to weirdness like South Korea or Sweden, both model societies that seem to have high suicide rates and alcohol abuse.

Diversity means difference, and that is valuable. But it's quite plausible that the gender gap doesn't represent a social failure, but rather an expression of growing individual freedom and choice. If so, we should work with this, not try to fight it.

I don't have an answer to why the gender gap exists, how to fix it, or even if we have to try to fix it, only guesses based on too little data. But it's good to throw spaghetti at the wall to see what sticks. If we can even start to identify the right problem, that would be a significant improvement.

by pieterh

Tip #1: Choosing your language

The choice of language is more important than anything else. When people download a new project, their first question isn't going to be, "why should I spend five minutes to try this stuff?", or "what problems does this solve?" but "what language did s/he write this in, and why?" Choose a language that shows how smart you are, something few people have ever heard of. You might even write your own language for the project. If you don't have time for that, you can get pretty close in any real, modern language. Abstraction is leverage, and don't you forget it!

Tip #2: Choosing your project name

The name is going to have to be memorable. Don't choose a bland name that simply says what your project is. Choose something that will make people giggle, when they get the joke. If you can find a really spaced-out name you could even get the matching dot-com! Trust us, the stranger the name, and the harder it is to type, the more people will appreciate you.

Tip #3: Choosing your license

Many people believe that any license at all is a horrid imposition on the freedom to copy code, and they leave their open source projects without any LICENSE file. You gotta admire such boldness. We know that shorter licenses mean more freedom so obviously no license at all is the very most free of all. rm -f LICENSE my friends!

Tip #4: Defining a roadmap

Make a roadmap and stick to it, no matter how far you fall behind. People will respect you for this. Most projects just adapt to new circumstances and knowledge with no self-respect at all. Some projects don't even have roadmaps, which is clearly insane. You should make weekly promises to your users about upcoming features, on your blog, and when they make issues, mark them all as "accepted" right away. The more stuff in the pipeline, the better, and a promise is worth its weight in silver unicorns!

Tip #5: Making a website

You need a beautiful website, because people like that. Users are very sensitive creatures so put up pictures of happy people and animated graphics in the highest possible resolution. Don't forget diversity! We'd say 30% of your whole budget should go into the website. Put everything there: blogs, events, issue tracking, brochures, etc. You can add a big "Buy Now" button that takes people to your sales team.

Tip #6: Code quality

The thing about open source is the code is open, so anyone can download it for free, without paying anything. This is awesome. Even just making your project open source, it automatically becomes awesome! But the code quality is what really matters. We humans didn't invent OCD for nothing - it makes better code! When people submit patches, make sure to give them lots of feedback on every little detail and don't merge code that isn't perfect.

Tip #7: Working with others

Other people are the enemy, don't forget this. They don't share your vision, ethics, or interests in music. If you let them into your project, they'll take it over and steal your ideas. This is the only thing about open source that isn't awesome, is other people, and the way they always steal ideas. We like to use Bitbucket and Mercurial, two beautifully designed barriers to entry. Above all, don't deal with that devil GitHub and its so-called "pull requests" that make it easy for literally anyone to get involved.

Tip #8: Writing the docs

Most awesome open source developers don't bother with docs. After all, if the code is perfect, who needs docs? If you're going to write docs, make them as painful to read as possible. Use tortuous language and change your definitions arbitrarily. This is important as it keeps your readers awake, and attentive. If you really are going to write a README, usually a two-liner is enough. Line one should say, "For details, see the code". And line two should say, "Patches welcome!"

Tip #9: Attack the competitors

A good blog posting should always attack a competitor. It's important to take a feature that you do well, and that no-one really uses or cares about. This means no-one else will have implemented it. You can then attack them mercilessly, and they will have no good answer except to ignore you, which is the sure sign of a loser. If in doubt, use Wikipedia to explain why your project is so awesome. Always get the last word on any Internet thread that discusses your project.

Tip #10: Helping your users

Last but not least, when people come to your list asking for help, watch out! You can easily get drawn into endless discussions with people who just want to use your work for free. Realize that users are parasites. Our advice is to reply to every question with a link to the README (and now you see why we advised you to write one!).

Conclusions

Use these tips carefully, and your already awesome open source project will become more awesomer. It's simple math, and we mean that literally in the metaphorical sense. Next week, an in-depth study on editors: "Vim or Emacs, why you should care."

by pieterh

But gambling with hundreds of millions of others' money isn't for everyone. For the rest of us who want to build large-scale software, there's Option Two, which is open source, and more specifically, free software. You may be asking how the choice of software license is relevant to the scale of the software you build. It's the right question.

The brilliant and visionary Eben Moglen once said, roughly, that a free software license is the contract on which a community builds. When I heard this, about ten years ago, the idea came to me, can we deliberately grow free software communities?

Ten years later, the answer is "yes", and there is almost a science to it. I say "almost" because we don't have much evidence of people doing this deliberately with a documented, reproducible process. It is what I'm trying to do with Social Architecture. My pilot experiment was ZeroMQ, after Wikidot, after Digistan and the FFII. There were… a lot of pilot projects. Anyhow, it seems clear that any explanation of how to build software at scale (which, experience tells us, is where the Money (with a capital M) is) must discuss how to build free software communities.

The Game

Software is my business: I code to eat, and to feed my kids and to make sure my wife doesn't leave me for someone nicer and better looking. Of course I love coding and that sensation of sculpting perfect functionality out of the raw mass of possibility. But the world doesn't care how passionately we feel. We wrest our living from a market that is largely uninterested, and often hostile when it does notice. This is the Game, and the full strategy of how to win it is for a full book.

What I will say here is that you have to identify and destroy your competitors, that you have to develop active strategies, learn rapidly, move quickly, and in general appreciate the Game as a constant competition in which every participant is an ally, an enemy, or the ground over which you are fighting.

How seriously you take the Game is up to you, but keep in mind that my advice for the rest of this article is aimed at helping you win, if and when you play it.

The Contract

When I write my best-selling book, "How to Make Big $$$ From Open Source", the first sentence will be "Use the GPL."

Here is a true story, it happened to the eldest brother-in-law of the cousin of a friend of mine's colleague at work. His name was, and still is, Patrick.

Patrick was a computer scientist, with a PhD in advanced network topologies. He spent two years and his savings building a new product, and choose the BSD license because he believed that would get him more adoption. He worked in his attic, at great personal cost, and proudly published his work. People applauded, for it was truly fantastic, and his mailing lists were soon abuzz with activity and patches and happy chatter. Many companies told him how they were saving millions using his work. Some of them even paid him for consultancy and training. He got invited to speak at conferences. He started a small business, hired a friend to work with him, and dreamed of making it big.

Then one day, someone pointed him to a new project, GPL licensed, which had forked his work and was improving on it. He was irritated and upset, and asked how people — fellow open sourcers, no less! — would so shamelessly steal his code. There were long arguments on the list about whether it was even legal to relicense their BSD code as GPL code. Turned out, it was. He tried to ignore the new project but then he soon realized that new patches coming from that project couldn't even be merged back into his work!

Worse, the GPL project got popular and some of his core contributors made first small, and then larger patches to it. Again, he couldn't use those changes, and he felt abandoned. Patrick went into a depression, his girlfriend left him for an international currency dealer called, weirdly, Patrice, and he stopped all work on the project. He felt betrayed, and utterly miserable. Finally he took a job as an architect for a cloud company, and by the age of forty, he had stopped programming even for fun.

Poor Patrick, I almost felt sorry for him. Then I asked him, "why didn't you choose the GPL?" "Because it's a restrictive viral license", he replied. I told him, you may have a PhD, and you may be the eldest brother-in-law of the cousin of a friend of my colleague, but you are an idiot and Monique was smart to leave you. You published your work saying, "please steal my code as long as you keep this 'please steal my code' statement in the resulting work", and when people did exactly that, you got upset. Worse, you were a hypocrite because when they did it in secret, you were happy, but when they did it openly, you got upset.

Seeing your hard work captured by a smarter team and then used against you is enormously painful, so why even make that possible? Every proprietary project that uses BSD code is capturing it. A public GPL fork is perhaps more humiliating, but it's totally, utterly self-inflicted.

BSD says "eat me". It was designed specifically by a university (Berkeley) with no profit motive, to leak work and effort. It is a way to push subsidized technology at below its cost price, a dumping of under-priced code in the hope that it will break the market for others. BSD is an excellent tool in the Game, but only if you're a large well-funded institution that can afford to use Option One.

For the rest of us, small businesses who aim our investments like precious bullets, leaking work and effort is unacceptable. We cannot afford to subsidize our market. We cannot afford battles with those we should naturally be allies with. We cannot afford to make fundamental business errors because in the end, that means we have to fire people.

It comes down to behavioral economics and game theory. The license we choose modifies the economics of those who use our work. In the Game there are allies, enemies, and lunch. BSD makes most people see us as lunch. Closed source makes most people see us as enemies (do you like paying people for software?) GPL, however, makes most people, with the exception of the Patricks of the world, our allies. Any fork of 0MQ is license compatible with 0MQ, to the point where we encourage forks as a valuable tool for experimentation.

The Process

If you've accepted my thesis up to now, great! You may actually get rich from your free software products. Now, I'll explain the rough process by which we actually build an open source community.

You recall the simplicity-oriented design process, where I claimed that we build successfully accurate software by successfully exploring the problem landscape, rather than by sheer intellectual effort. Keep this in mind. Now, see your community as a group exploring that landscape and sharing the results of their work.

Your goal as leader of a community is to motivate people to get out there and explore; to ensure they can do so safely and without disturbing others; to reward them when they make successful discoveries; and to ensure they share their knowledge with everyone else.

It is an iterative process. You make a small product, at your own cost, but in public view. You then build a small community around that product. If you have a small but real hit, the community then helps design and build the next version, and grows larger. And then that community builds the next version, and so on. It's evident that the more control you try to assert over the material results, the less people will want to participate.

Crazy, Beautiful, and Easy

You need a goal that's crazy and simple enough to get people out of bed in the morning. Your community has to attract the very best people and that demands something special. With 0MQ, we said we were going to make "the Fastest. Messaging. Ever.", which qualifies as a good motivator. If we'd said, we're going to make "a smart transport layer that'll connect your moving pieces cheaply and flexibly across your enterprise", we'd have failed.

Then your work must be beautiful, immediately useful and attractive. Your contributors are users who want to explore just a little beyond where they are now. Make it simple, elegant, brutally clean. SOD it, and I mean that. The experience when people run or use your work should be an emotional one. They should feel something, and if you accurately solved even just one big problem that until then they didn't quite realize they faced, you'll have a small part of their soul.

And then, easy to understand, use, and join. Too many projects have barriers to access: put yourself in the other person's mind and see all the reasons they come to your site, thinking "Uhm, interesting project, but…" and then leave. You want them to stay, try it, just once. Use github and put the issue tracker right there.

If you do these things well, your community will be smart but more importantly, will be intellectually and geographically diverse. This is really important. A group of like-minded experts cannot explore the problem landscape well. They tend to make big mistakes. Diversity beats education any time.

Stranger, meet Stranger

How much up-front agreement do two people need to work together on something? In most organizations, a lot. But you can bring this cost down to zero, and then people can collaborate without having ever met, done a phone conference, meeting, or business trip to discuss Roles and Responsibilities.

You need well-written rules that are designed by cynical people like me to force strangers into mutually beneficial collaboration instead of conflict. The GPL is a good start. Github and its fork/merge strategy is a good follow-up. And then you want something like our C4 rulebook to control how work actually happens.

C4 (which I now use for every new open source project) has detailed (and tested) answers to a lot of common mistakes people make. For example, the sin of working off-line in a corner with others "because it's faster". Transparency is essential to get trust, which is essential to get scale. By forcing every single change through a single transparent process, you build real trust in the results.

Another cardinal sin that many open source developers make is to place themselves above others. "I founded this project thus my intellect is superior to that of others". It's not just immodest and rude, and usually inaccurate, it's also poor business. The rules must apply equally to everyone, without distinction. You are part of the community. Your job, as founder of a project, is not to impose your vision of the product over others, but to make sure the rules are good, honest, and enforced.

Infinite Property

One of the saddest myths of the knowledge business is that ideas are a sensible form of property. It's medieval nonsense that should have been junked along with slavery, but sadly it's still making too many powerful people too much money.

Ideas are cheap. What does work sensibly as property is the hard work we do in building a market. "You eat what you kill" is the right model for encouraging people to work hard. Whether it's moral authority over a project, money from consulting, or the sale of a trademark to some large, rich firm: if you make it, you own it. But what you really own is "footfall", participants in your project, which ultimately defines your power in the Game.

To do this requires infinite free space. Thankfully, github solved this problem for us, for which I will die a grateful person (there are many reasons to be grateful in life, but this is one of them).

You cannot scale a single project with many owners like you can scale a collection of many small projects, each with fewer owners. When we embrace forks, a person can become an "owner" with a single click. Now they just have to convince others to join, by demonstrating their unique value.

So in 0MQ we aimed to make it easy to write bindings on top of the core library, and we stopped trying to make those bindings ourselves. This created space for others to make those, become their owners, get that credit.

Making Money from Open Source

Now, finally, how to make money from your successful open source project. Start by realizing that it will take years to get to the point where you can make real money. Patience, an economical lifestyle, and no debts are as important as anything else.

To make money from your open source, you must ensure your competitors stay or become marginal. That is fairly straight-forward: find their weaknesses, attack those, protect or hide your own weaknesses, repeat until it's over. If you don't think this is an ethical way to do business, just wait a few years. You will either learn, or become an employee . By using the GPL we should already have converted most compatible projects into allies, so it's fair to say the GPL is pacifist as well as ultra-capitalist.

Then, you must brand your product. The branding is key to explaining to people why they will send you real money (instead of emails asking you to do their homework). Choose a bold, confident logo and name, and website, and focus your brand on your core values, which are "crazy, beautiful, easy".

Now, what can you sell? Forget about "commercial licenses". Part of your deal with your growing community of contributors was that you would never fork their work to compete against them. This means no copyright assignments, and every patch owned by its author.

How about commercial add-ons? Perhaps, but in my experience you will always benefit more from building more open source.

So, this leaves two or three options. One is to sell your services helping others use the technology. The nice thing is you can scale this rapidly. After all, you have a community of ready-trained experts to call on, right? Who better to hire than the guy who just spent a month learning the source code.

The second option is to sell your business, for which you have to plan well in advance. What you in fact sell then isn't the copyrights, since you don't own these fully (or even in majority). It's the trademark, reputation, and the goodwill of the community. The new owner may make a mess of things. In that case you, or anyone else, can fork the whole thing right back to where it was.

Depending on the type of product you're making, there will be other services to sell. Support agreements ("pay us X per year and we'll fix any issues you hit within X hours or days") are an easy sell to larger customers. Advertisements can be profitable if you run very large websites. You may find lecture tours and workshops good income.

Finally, and here's the real money maker: t-shirts with pictures of cats saying funny things about your software. "I UZES 0MQ TO GETS MAI CHEEZEBURGERS" is our most popular one these days.

by pieterh

First, what do I mean by "change latency" and why is this important? Without diving too deep into the theories of collective intelligence and problem solving, C4 asserts that the optimal design for a software development team is two halves, one that sets challenges, and one that solves them. While pair programming puts two drivers behind the wheel, it gives no guarantee that they drive in the right direction. The challenge/solution pairing gives one person, or group, the task of defining the right directions, and another the task of driving as fast as possible in that direction.

C4 drives a constant stream of such challenges, written as issues that capture problems. As long as the people writing the issues have some empathy with real users, or are real users, the product moves accurately in the right direction.

So each answer to a well-expressed problem comes in as a change to the software product. About 10-20% of these changes will be wrong, badly done, or solving problems that are themselves "wrong". The usual approach with software development is to accumulate a large number of changes, and then weed out the errors one by one, until the code is stable enough for production use.

Even the very best programmers will make significant numbers of mistakes. They don't always appear as bugs; they can be too-complex designs, solutions to problems no-one cares about, features that don't work as expected, functionality that people want which is "impossible" due to other design decisions, and so on. Even a really excellent library like ZeroMQ has had many of these mistakes in it. Often we end up choosing a piece of software just because the good outweighs the bad. None, it seems, can be perfect.

However, with C4 we apply changes directly to the master version and we look for a fast yes/no answer to the question, "was that a good change?" We don't use branches because they slow down the time it takes to get this answer. We don't over-discuss pull requests for the same reason. In fact we're doing whatever it takes to reduce the round-trip time from "I have identified a problem and propose solution X" to "Yes, solution X makes sense and it seems to work fine" or "solution X is insane and I've make a patch that reverts it".

With ZeroMQ we're now getting change latencies of a few hours.

What does this mean for stable releases? I'm not yet certain but what seems to be emerging is that the git master is perfect 80-90% of the time, and one patch away from perfection the remaining 10-20% of the time. The remaining issues in ZeroMQ are all in old code, none in new code. This is highly significant.

Let's think this through again. Historically, the only reason for making stable releases was that we had lots of change that took months or years to fully validate. I'm fully sure now, with evidence, that making large changes is both unnecessary, and sub-optimal. Instead, with C4 we aim for a steady stream of small iterative changes, where each can be validated or rejected very rapidly. Thus, we have removed the need for stabilization.

C4 is also a formal contract for ongoing interoperability. That is, we do not break old APIs or protocols. In the old days we happily broke these, and bumped the version number. That didn't help anyone, in fact. People stayed away from ZeroMQ/3.0 and ZeroMQ/4.0 because interoperability is more important than new features.

We will make a stable release candidate of ZeroMQ/3.1 soon. But I'm curious to see how far that stable candidate diverges from master. We'll continue to make formal releases for obvious reasons: people expect this, they do not trust github masters, they need well-labeled packages for internal deployment. But really, over time, I think the whole concept of "stable releases" will disappear, and be replaced by "latest master that passed all tests".

by pieterh

Why Discuss This?

Presenting ZeroMQ at Mix-IT in Lyon a couple of weeks ago, I was asked several times for the "roadmap". My answer was, roadmaps are bad for several reasons. First, they make promises we can rarely keep, which causes problems for our users. Second, they claim territory and make it harder for others to participate. Lastly, they preempt the thinking process of the community. The audience didn't really like my answer. So un-French. Software engineers don't like the notion that powerful, effective solutions can come into existence without an intelligent designer actively thinking things through. And yet no-one in that room would question evolution. A strange irony, and one I want to explore further.

The Theory of Individual Creativity

In the dominant theory, brilliant individuals reflect on large problem sets and then carefully and precisely create a solution. Sometimes they will have "eureka" moments where they "get" brilliantly simple answers to whole large problem sets. The inventor, and the process of invention are rare, precious, and can command a monopoly. History is full of such individuals.

Looking closer, however, the facts don't match. History doesn't show lone inventors. It shows lucky people who steal or claim ownership of ideas that are being worked on by many. It shows brilliant people striking lucky once, and then spending decades on fruitless and pointless quests. The best known large-scale inventors like Thomas Edison were in fact just very good at systematic broad research done by large teams. It's like claiming that Steve Jobs invented every device made by Apple. It is a nice myth, good for marketing, but utterly incorrect.

Recent document history shows this very well. The Internet is surely one of the most innovative and fast-moving areas of technology. It has no inventor. Instead it has a large economy of people who have carefully and progressively solved a long series of immediate problems, documented their answers, and made those available to all. The innovative nature of the Internet comes not from a collection of Einsteins. It comes from RFCs anyone can use and improve. It comes from open source software anyone can use and improve. It comes from sharing, scale of community, and the continuous accretion of good solutions and disposal of bad ones.

The Theory of Heuristic Innovation

Here is the Theory of Heuristic Innovation (v0.2):

1. There is an infinite problem/solution terrain.
2. This terrain changes over time according to external conditions.
3. We can only accurately perceive problems we are close to.
4. We can rank the cost/benefit economics of problems using a market for solutions.
5. There is one temporarily optimal solution to any solvable problem.
6. We can approach this optimal solution heuristically, and mechanically.
7. Our intelligence can make this process faster and more accurate but does not replace it.

It's an approximation. Feel free to send me patches.

What does This Mean?

First, and most broadly, it means that we do not invent solutions, but we discover them. Given a well-defined problem, there is exactly one optimal solution, and our work as software engineers is to climb the solution terrain until we find it. There may be local peaks. We may have to back-track. We make mistakes. But if we have a good process for climbing that terrain and we apply that process mechanically, we will eventually find the optimum.

Second, this means that individual talent is helpful only in the speed with which we move. Intelligence moves us faster but does not make any fundamental difference in the outcome. The process is much more significant than talent. Which is why so many talented engineers produce trashware, and why teams working in the same space systematically seem to discover the same solutions.

Thirdly, it means that old solutions to old problems can become obsolete as the terrain shifts. Technological and environmental changes are the main factors here.

Finally, it means that our primary goal in producing software should be to define the best possible process, and then learn to apply it mechanically and systematically.

As a correlation, we can bootstrap this by applying any given process to the process itself.

Elements of a Mechanical Process

We need several elements:

• A starting point in the problem/solution terrain.
• A way to collect and measure problems in the near space around us.
• A way to experiment with solutions cheaply and rapidly.
• A way to measure these experiments against our problems.
• A way to discard the failures and propagate the successes.
• A way to isolate layers so we can review and re-solve old problems.

Let's apply the mechanical process to software design and see what we get. This will sound repetitive to readers of my blog.

• Starting point: identify a single interesting problem and make a minimal, plausible prototype. Time taken: 3-5 days maximum.

• Collecting and measuring problems: publish your prototype on github.com and ask people to play with it. Use it yourself and collect problems with it. Write these down as issues.

• Experimenting with solutions: pick out the most critical ones and figure out "good enough" solutions to these. Make patches and merge them.

• Measuring solutions against problems: try the results. You should always have executable, usable product that can be shipped if needed (even as a prototype).

• Discarding failures and propagating successes: if you don't like the solutions, revert the patches. If you like them, close the issue. Use the LGPL to ensure your successes get the widest trade with other projects.

• Isolating layers: use ZeroMQ to isolate layers so that you can throw away old pieces and start again without bringing your whole product into question.

Conclusions

Talent lets you move faster, and process gets you to the right place. The ideal process seems to be one based on highly incremental small steps, each step solving one identifiable problem with a "good enough" solution. Over time, the collection of small accurate solutions to real problems, and removal of mistakes, will give you a highly valuable product that people will love.

by pieterh

Background

Let me start with my credentials. My first open source project was Libero from 1991. I wrote Xitami, a popular open source web server, and killed that in 2001. I wrote most of OpenAMQ, the first AMQP implementation. I founded and steered the ZeroMQ community and have maintained its stable releases for years. If there is one thing I know really well, it's how to build excellent software.

Git is a revolution, especially when combined with github. In the last year or two, the github/git combination has become a key tool for organizing teams, and building processes like C4 and PC3 that are (as far as I know) the first reusable contracts of their kind.

Here is a section of PC3 that will shock some people:

• The project SHALL have one branch ("master") that always holds the latest in-progress version and SHOULD always build.
• The project SHALL NOT use topic branches for any reason. Personal forks MAY use topic branches.
• To make a stable release someone SHALL fork the repository by copying it and thus become maintainer of this repository.

To be clear, it's public branches in shared repositories that I'm talking about. Using branches for private work, e.g. to work on different issues, appears to work just fine.

The PC3 text is not accidental. This section came from trial-and-error, mainly in the ZeroMQ community. Originally, when Martin Sustrik and I (the pragmatic core developers) started using forks instead of branches for ZeroMQ's stable versions, many people reacted with shock and horror. Today, people have less emotional response. Tomorrow, I think it'll be clear that branches were, in fact, an entirely wrong approach inherited from the days of Subversion and monolithic repositories.

More profoundly, the branches vs. forks argument is really a wider design vs. evolve argument about how to make software optimally (both PC3 and C4 fully embrace the "evolve" approach). I may address that wider argument in a future article.

To make my argument here, I'll look at a number of criteria, and compare branches and forks in each one.

Complexity

The simpler, the better.

There is no inherent reason branches are more complex than forks. However, git-flow uses five types of branch, whereas PC3 uses two types of fork (development, and stable) and one branch (master). Circumstantial evidence is that branches lead to more complexity than forks. For naive users, it is definitely easier to learn to work with many repositories and no branches.

Learning Curve

The smoother the learning curve, the better.

Evidence definitely shows that learning to use git branches is complex. For some people this is OK. For most developers, every cycle spent learning git is a cycle lost on more productive things. I've been told several times, by different people, that I do not like branches because I "never properly learned git". That is fair but it is a criticism of the tool, not the human.

Cost of Failure

The lower the cost of failure, the better.

Branches demand more perfection from developers since mistakes potentially affect others. This raises the cost of failure. Forks make failure extremely cheap since nothing that happens in a fork can affect others not using that fork.

Upfront Coordination

The less need for upfront coordination, the better.

You can do a hostile fork. You cannot do a hostile branch. Branches depend on upfront coordination, which is expensive and fragile. One person can veto the desires of a whole group. In the ZeroMQ community for example we were unable to agree on a git branching model for a year. We solved that by using forking instead. The problem went away.

Scalability

The more you can scale a project, the better.

The strong assumption in all branch strategies is that the repository is the project. But there is a limit to how many people you can get in agreement to work together in one repository. As I explained, the cost of upfront coordination can become fatal. A more realistic project scales by allowing anyone to start their own repositories, and ensuring these can work together. A project like ZeroMQ has dozens of repositories. Forking looks more scalable than branching.

Surprise and Expectations

The less surprising, the better.

People expect branches and find forks to be uncommon and thus confusing. This is the one aspect where branches win. However, it's also a reason for sticking to FORTRAN and COBOL. We do not refuse innovation just because it's surprising.

Economics of Participation

The more tangible the rewards, the better.

A fully free process like PC3/C4 lets people organize around problems. Most organizations are not ready for such a radical management approach. But even a top-down approach needs people to feel rewarded for their work. Branches don't act like "product" but like "discrete variations of product". People have less interest in contributing to a discrete variation. Whereas everyone wants their name on a successful product. So the economics of branches are worse than the economics of forks.

Robustness in Conflict

The more a model can survive conflict, the better.

Like it or not, people fight over ego, status, belief. If your organizational model depends on agreement, you won't survive the first real fight. Branches do not survive real arguments and fights. Whereas forks can be hostile, and still benefit all parties. And this is indeed how free software works. Score one for forks, zero for braches.

Guarantees of Isolation

The stronger the isolation between production code and experiment, the better.

People make mistakes. I've seen experimental code pushed to mainline production by error. I've seen people make bad panic changes under stress. But the real fault is in allowing two entirely separate generations of product to exist in the same protected space. If you can push to random-branch-x you can push to master. Branches do not guarantee isolation of production critical code. Forks do.

Visibility

The more visible our work, the better.

Forks have watchers, issues, a README, a wiki. Branches have none of these. People try forks, build them, break them, patch them. Forks sit there until someone remembers to work on them. Forks have downloads and tarballs. Branches do not. When we look for self-organization, the more visible and declarative the problems, the faster and more accurately we can work.

Conclusions

Git branches are, in my experience and in shared repositories, harmful. It is better to work with a branch-free process that uses forks for stabilization. This comes from some years of trial and error on a wide range of projects. We have systematically found forks to be cheaper and safer and easier than branches. Branch-free processes like C4 and PC3 are real, and they work, in anger, both on closed source and open source projects. The only downside of a branch-free process seems to be that it shocks people with previous git experience. This is a passing effect, in our experience.

by pieterh

Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119[1].

Goals

PC3 is meant to provide an optimal collaboration model for commercial software projects. Broadly, PC3 helps an organization build consistently good software, cheaply, and rapidly. It has these specific goals:

• To maximize the scale of the community around a project, by reducing the friction for new Contributors and creating a scaled participation model with strong positive feedbacks;

• To relieve dependencies on key individuals by separating different skill sets so that there is a larger pool of competence in any required domain;

• To allow the project to develop faster and more accurately, by increasing the diversity of the decision making process;

• To support the natural life-cycle of project versions from experimental through to stable, by allowing safe experimentation, rapid failure, and isolation of stable code;

• To reduce the internal complexity of project repositories, thus making it easier for Contributors to participate and reducing the scope for error;

• To reduce the need for meetings, face-to-face presence, and timezone synchronization, by capturing knowledge more accurately.

• To optimize the efficiency of worker resources, by using on-time self-assignment instead of up-front task allocation.

Design

Preliminaries

• The project SHALL use the git distributed revision control system.
• The project SHALL be hosted on github.com or equivalent, herein called the "Platform".
• The project SHALL use the Platform issue tracker.
• The project SHOULD have clearly documented guidelines for code style.
• A "Contributor" is a person who wishes to provide a patch, being a set of commits that solve some clearly identified problem.
• A "Maintainer" is a person who merge patches to the project. Maintainers are not developers; their job is to enforce process.
• A "Reviewer" is a person who reviews patches and who has deep familiarity with the code base.
• Contributors SHALL NOT have commit access to the repository unless they are also Maintainers.
• Maintainers SHALL have commit access to the repository.
• Reviewers SHALL NOT have commit access to the repository unless they are also Maintainers.
• Everyone, without distinction or discrimination, SHALL have an equal right to become a Contributor under the terms of this contract.

Patch Requirements

• Maintainers, Contributors and Reviewers MUST have a Platform account and SHOULD use their real names or a well-known alias.
• A patch SHOULD be a minimal and accurate answer to exactly one identified and agreed problem.
• A patch MUST adhere to the code style guidelines of the project if these are defined.
• A patch MUST adhere to the "Evolution of Public Contracts" guidelines defined below.
• A patch MUST compile cleanly on at least the most important target Platforms.
• A "Correct Patch" is one that satisfies the above requirements.

Development Process

• Change on the project SHALL be governed by the pattern of accurately identifying problems and applying minimal, accurate solutions to these problems.
• To initiate changes, a user SHALL log an issue on the project Platform issue tracker.
• The user SHOULD write the issue by describing the problem they face or observe.
• The user SHOULD seek consensus on the accuracy of their observation, and the value of solving the problem.
• Thus, the release history of the project SHALL be a list of meaningful issues logged and solved.
• To work on an issue, a Contributor SHALL fork the project repository and then work on their forked repository.
• To submit a patch, a Contributor SHALL create a Platform pull request back to the project.
• A Contributor SHALL NOT commit changes directly to the project.
• To discuss a patch, people MAY comment on the Platform pull request, on the commit, or elsewhere.
• To accept or reject a patch, a Maintainer SHALL use the Platform interface to merge the patch.
• Maintainers SHALL NOT accept their own patches.
• Maintainers SHALL NOT make value judgments on correct patches, this is handled by the optional Code Review Process.
• Maintainers SHOULD ask for improvements to incorrect patches and SHOULD reject incorrect patches if the Contributor does not respond constructively.
• Maintainers MAY commit changes to non-source documentation directly to the project.

Code Review Process

• The project MAY use a code review process, particularly if it is a shipping project with non-trivial complexity.
• In code reviews are enabled for the project, Maintainers SHALL NOT merge a patch until a Reviewer has examined and approved the patch.
• If code reviews are not enabled for the project, Maintainers SHALL merge correct patches rapidly.

Creating Stable Releases

• The project SHALL have one branch ("master") that always holds the latest in-progress version and SHOULD always build.
• The project SHALL NOT use topic branches for any reason. Personal forks MAY use topic branches.
• To make a stable release someone SHALL fork the repository by copying it and thus become maintainer of this repository.
• Forking a project for stabilization MAY be done unilaterally and without agreement of project maintainers.
• Maintainers of the stabilization project SHALL maintain it through pull requests which MAY cherry-pick patches from the forked project.
• A patch to a repository declared "stable" SHALL be accompanied by a reproducible test case.
• A stabilization repository SHOULD progress through these phases: "unstable", "candidate", "stable", and then "legacy". That is, the default behavior of stabilization repositories is to die.

Evolution of Public Contracts

• All Public Contracts (APIs or protocols) SHOULD be documented.
• All Public Contracts SHALL use Semantic Versioning[2].
• All Public Contracts SHOULD have space for extensibility and experimentation.
• A patch that modifies a Public Contract SHOULD not break existing applications unless there is prior consensus on the value of doing this.
• A patch that introduces new features to a Public Contract SHOULD do so using new names.
• Old names SHOULD be deprecated in a systematic fashion by marking new names as "experimental" until they are stable, then marking the old names as "deprecated".
• When sufficient time has passed, old deprecated names SHOULD be marked "legacy" and eventually removed.
• Old names SHALL NOT be reused by new features.
• When old names are removed, their implementations MUST provoke an exception (assertion) if used by applications.

Issue Format

• One issue SHOULD address one single identifiable problem or a small set of tightly related problems.
• The issue title SHOULD state the observed problem in minimal fashion.
• The issue body SHOULD capture all relevant data in a minimal and accurate fashion.
• The issue body MAY propose solutions.
• Users SHALL NOT log feature requests, ideas, suggestions, or any solutions to problems that are not explicitly documented and provable.

Task and Role Assignment

• All tasks and roles SHALL be self-assigned, based on individual judgement of the value of taking on a certain task or role.

References

by pieterh

The Lazy Perfectionist

Never design anything that's not a precise minimal answer to a problem we can identify and have to solve. — Pieter Hintjens

The Lazy Perfectionist spends his idle time observing others and identifying problems that are worth solving. He looks for agreement on those problems, always asking, "what is the real problem". Then he moves, precisely and minimally, to build, or get others to build, a usable answer to one problem. He uses, or gets others to use those solutions. And he repeats this until there are no problems left to solve, or time or money runs out.

The Benevolent Tyrant

The control of a large force is the same principle as the control of a few men: it is merely a question of dividing up their numbers. — Sun Tzu

The Benevolent Tyrant divides large problems into smaller ones and throws them at groups to focus on. He brokers contracts between these groups, in the form of APIs and unprotocols. The Benevolent Tyrant constructs a supply chain that starts with problems, and results in usable solutions. He is ruthless about how the supply chain works, but does not tell people on what to work, nor how to do their work.

The Earth and Sky

The ideal team consists of two sides: one writing code, and one providing feedback. — Pieter Hintjens

The Earth and Sky work together as a whole, in close proximity, but they communicate formally through an issue tracking. Sky seeks out problems, from others and from their own use of the product, and feeds these to Earth. Earth rapidly answers with testable solutions. Earth and Sky can work through dozens of issues in a day. Sky talks to other users, and Earth talks to other developers. Earth and Sky may be two people, or two small groups.

The Happy Failure

To succeed you must learn to fail rapidly, cheaply, and often. — Pieter Hintjens

The Happy Failure embraces failure as the only real way to learn. He focuses on reducing the cost of failure, and documenting failures so that everyone can learn from them. He does not over-specify functionality, stays away from upfront documentation, does not use test-driven development. All of these make it harder, and more costly to fail.

The Open Door

The accuracy of knowledge comes from diversity. — Pieter Hintjens

The Open Door accepts contributions from almost anyone. He does not argue quality or direction, instead allowing others to argue that and so get more engaged. He calculates that even a troll will bring more diverse opinion to the group. He lets the group form its opinion about what goes into stable code, and he enforces this opinion with help of a Benevolent Tyrant.

The Laughing Clown

Perfection precludes participation. — Pieter Hintjens

The Laughing Clown, often acting as the Happy Failure, makes no claim to high competence. Instead his antics and bumbling attempts provoke others into rescuing him from his own tragedy. Somehow however, he always identifies the right problems to solve. People are so busy proving him wrong they don't realize they're doing valuable work.

The Mindful General

Make no plans. Set goals, develop strategies and tactics. — Pieter Hintjens

The Mindful General operates in unknown territory, solving problems that are hidden until they are nearby. Thus he makes no plans, but seeks opportunities, then exploits them rapidly and accurately. He develops tactics and strategies in the field, and teaches these to his men so they can move independently, and together.

The Social Engineer

If you know the enemy and know yourself, you need not fear the result of a hundred battles. — Sun Tzu

The Social Engineer reads the hearts and minds of those he works with and for. He asks, of everyone, "what makes this person angry, insecure, argumentative, calm, happy?" He studies their moods and disposition. With this knowledge he can encourage those who are useful, and discourage those who are not. The Social Engineer never acts on his own emotions.

The Constant Gardener

Do not repeat the tactics which gained you one victory, but let your methods be regulated by the infinite variety of circumstances. — Sun Tzu

The Constant Gardener grows a process from a small seed, step by step as more people come into the project. He makes every change for a precise reason, with agreement from everyone. He never imposes a process from above but will let others come to consensus, then he will enforce that consensus. In this way everyone owns the process together and by owning it, they are attached to it.

The Rolling Stone

After crossing a river, you should get far away from it. — Sun Tzu

The Rolling Stone accepts his own mortality and transience. He has no attachment to his past work. He accepts that all that we make is destined for the trash can, it is just a matter of time. With precise, minimal investments, he can move rapidly away from the past and stay focused on the present and near future. Above all he has no ego and no pride to be hurt by the actions of others.

The Pirate Gang

Code, like all knowledge, works best as collective — not private — property. — Pieter Hintjens

The Pirate Gang organize freely around problems. They accept authority insofar as it provides goals and resources. The Pirate Gang own and share all they make: every work is fully remixable by others in the Pirate Gang. They move rapidly as new problems emerge, and are quick to abandon old solutions if they stop being relevant. No persons or groups can monopolize any part of the supply-chain.

The Flash Mob

Water shapes its course according to the nature of the ground over which it flows. — Sun Tzu

The Flash Mob comes together in space and time as needed, then disperses as soon as they can. Physical closeness is essential for high-bandwidth communications. But over time it creates technical ghettos, where Earth gets separated from Sky. The Flash Mob tends to collect a lot of frequent flier miles.

The Canary Watcher

Pain is not, generally, a Good Sign. — Pieter Hintjens

The Canary Watcher measures the quality of an organization by the their own pain level, and the observed pain levels of those he works with. He brings new participants into existing organizations so they can express the raw pain of the innocent. He may use alcohol to get others to verbalize their pain points. He asks others, and himself, "are you happy in this process, and if not, why not?" When an organization causes pain in himself or others, he treats that as a problem to be fixed. People should feel joy in their work.

The Hangman

Never interrupt others when they are making mistakes. — Pieter Hintjens

The Hangman knows that we learn only by making mistakes, and he gives others copious rope with which to learn. He only pulls the rope gently, when it's time. A little tug to remind the other of their precarious position. Allowing others to learn by failure gives the good reason to stay, and the bad excuse to leave. The Hangman is endlessly patient, because there is no shortcut to the learning process.

The Historian

The public record may be tedious, but it's the only way to prevent collusion. — Pieter Hintjens

The Historian forces discussion into the public view, to prevent collusion to own areas of work. The Pirate Gang depends on full and equal communications that do not depend on momentary presence. No-one really reads the archives, but the simply possibility stops most abuses. The Historian encourages the right tool for the job: email for transient discussions, IRC for chatter, wikis for knowledge, issue tracking for recording opportunities.

The Provocateur

When a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully. — Samuel Johnson

The Provocateur creates deadlines, enemies, and the occasional impossibility. Teams work best when they don't have time for the crap. Deadlines bring people together and focus the collective mind. An external enemy can move a passive team into action. The Provocateur never takes the deadline too seriously. The product is always ready to ship. But he gently reminds the team of the stakes: fail, and we all look for other jobs.

The Mystic

When people argue or complain, just write them a Sun Tzu quotation — Mikko Koppanen

The Mystic never argues directly. He knows that to argue with an emotional person only creates more emotion. Instead he side-steps the discussion. It's hard to be angry at a Chinese general, especially when he has been dead for 2,400 years. The Mystic will act as the Hangman when people insist on the right to get it wrong.

This article is an extract from Chapter 6 of the ZeroMQ book, on community building. If you like this, buy the book :-)

by pieterh

Prelude

Making friends quickly may not seem relevant for a blog on software engineering, but to quote my favorite author (me, in case you doubted it), software is all about people. As well as maybe getting you neat and unexpected upgrades, social engineering in smaller or larger doses is key to good management, to good interpersonal relationships, and to a happy and successful life in general. Just as hell is other people, so is heaven. The key to happiness is not money, security, or even good health. It is quite simply to create and enjoy good relationships with the people you cross paths with during your life.

There are only three kinds of relationship. There those you cannot make work, no matter how hard you try. There are relationships you don't need to work on, they will always be good. And there relationships you can profitably invest in. Superficially, social engineering works on practically anyone, in almost all situations. But long term relationships are based on the economics of mutual need, which only emerge over time. It would be cool to have social networking software that understood these patterns and could help do the triage (from the Latin "to carve into three pieces").

What's remarkable about social engineering is how broadly it applies. The identical patterns work both on casual acquaintances, in the space of minutes, and on long term relationships over the span of years. They work on individuals, and on groups of practically any size. They work on women and men, of all ages.

It strikes me as ironic that I'm using social engineering to sit alone in a larger space, as far from my fellow passengers as possible without climbing into the pilot's lap. Well, what is life without its little ironies?

The Basics

Good vs. Evil

Let's start by pulling "engineering" apart from "hacking" (or "cracking" if you prefer). The two semantics are often confused, and people often use "social engineering" as a euphemism for "lying to and conning people". For the sake of consistent terminology, engineering means to create something, whereas hacking is to unlock something, from a puzzle to a bank account. You can engineer a lock, and you can hack a lock. You can engineer your way into a night club and you can hack your way in. Engineering and hacking are both are valid skill sets, but they aren't the same any more than pepper spray is a foodstuff.

At the heart all social interactions is the question of ethics. Many people have a hard time to define ethics concretely, but that's because they never thought it through. Ethics is just about the relative power of both parties. In short, whether that power is balanced, and the interaction is thus fair to both parties. Would you switch sides in a deal? If not, it's unethical. If yes, it's ethical. You cannot discuss the ethics of eating a slice of cheese since there is no plausible "you" to switch sides with. You can discuss the ethics of knocking down a beautiful old building to set up a concrete office block, because both represent people, their dreams and needs and investments.

Social engineering must be ethical by definition. Social hacking is by definition unethical because it depends on an unequal exchange. You would never agree to be the one tricked into giving up your password.

Setting the Scene

To kick off a social engineering session you define a clear goal and you set your mental state to equal proportions of angelic happiness, gentle charm, firm determination, and infinite patience. It doesn't matter whether you're buying a loaf of bread, ordering espresso in an Italian mountaintop bar, dating the woman of your dreams, or tracking down lost luggage. If you are incoherent, stressed, unpleasant, uncertain, or impatient, you will be rejected or treated badly.

These are difficult techniques that take most of us years or decades to learn. Few people are natural social engineers and those that are can be somewhat disconcerting individuals. I can give some broad hints but the key is practice and observation. Social engineering starts with creating, in real time and with strangers, a tangible shared experience that the other person truly gets a kick from being in. Let me take those aspects one by one:

• Clarity: Define your own goals clearly, and explain them without decoration as a problem that the other person might, perhaps, be able to solve. Don't demand, but suggest. This gives the other person a simple, consistent and interesting conceptual model. In a world full of confusion, stress, and complexity, we lust after simple tangible challenges. It lets us relax, revert to a childlike joy, and drop our guard. You can learn to give people this kind of experience. Sometimes, indeed often, things really are very simple. Though sadly, they are probably impossible.

• Happiness: Watch for your own negative emotions (anger, jealousy, fear, insecurity, hate, self-pity, impatience) and learn to turn them off. It is an interesting trick I learned so I could manage a crazy girlfriend, a long time ago. This leaves the remaining emotion (joy) to run around like a kid in the garden. If you can't be where you want to be, enjoy where you are. When you are purely happy, you express that happiness in your face, and people see it. When we talk to someone who shows no emotion except a sincere smile, we lower our defenses and relax. These are ancient and powerful instincts.

• Charm: The salesman's patter isn't just a caricature. Soothing words, delivered with a broad and true smile, are the gentle back-rub of social interaction. It doesn't really matter what you say except that you must enjoy what you are talking about, never express any negative emotions, and speak directly to the person. Look into their eyes, smile sincerely, and say that it rained in China last Tuesday, and you will still get a positive response.

• Determination: Be firm in restating your problem and exploring all routes to get them. A "no" just means that particular avenue might be closed. There are always many options, so explore them until you really can't think of any more. It is shocking how many people just give up at the first rejection. Partly, because they haven't learned to deal with their insecurity. But also because they just don't realize that we create our own options.

• Patience: Being in a hurry is both rude and counter-productive. It might take all day to get things right. And you are so happy right here, now, why on earth would you rush it? Patient people make us feel patient too. That means we're less in a hurry to close the transaction, with a final "no". Perhaps there is one more thing we could try. Of course, you need to be realistic and measure your potential success with the time you're spending. You're not going to wait an hour to buy a loaf of bread. Unless that's the only bread in town.

These techniques, when you get them right, can be powerful. My father, a natural and gifted social engineer, would walk into a shop where I went often, to conduct some minor transaction, one time, and the sales staff would remember him and ask after him for years.

Getting Closer

It is important to focus on the other person, listen to them and watch them. People express themselves in body posture, facial expressions, voice tone and volume. You don't need to read this consciously, and in fact that's is too slow and clumsy. You can't walk by making each step deliberate, you look like you're on some kind drug. Once as a teenager I tried walking like a ninja through a shopping center, taking each step deliberately, silently. People came up to me in hushed tones, pressed folded five-quid notes into my hands and asked me for baggies.

You can become skilled enough to read people intuitively, rapidly and accurately. Just practice every day for years. It's just like speaking to a crowd, another exercise in social engineering. You need to watch the crowd, get closer, feel their response to you, and amplify that by adjusting your own expression, to create that win-win shared experience. Incidentally, why PowerPoint is so tragic.

You also have to remove potential barriers to entry. This is, any excuse for the other person to reject you using their natural preconceptions and prejudices. People love to say "no", it gives them a feeling of power. That's why you never make a direct request. There are two main reasons people will refuse to listen to you at all, and both are fixable with some effort:

• Language: You absolutely need to speak the native language of the other person. The more, the better. You can get away with an awful accent but if you force them out of their comfort zone, they will not like you for it. A few words is a start but really not enough. If you are conducting business in a foreign country you must get language courses and get proficient as soon as sanely possible. Listen to the language everywhere you are, learn its melody and rhythm, imitate it until you get it right.

• Appearance: If you try to "express yourself" through your appearance, you are really just pandering to your emotions, insecurity and fear of being alone. Wearing a style means putting on some group identity. This is fine if you are doing social engineering in a well-known context where your peers are all in that group. I'll wear a suit when doing business in Korea because all my clients there wear suits. I'll wear jeans in Dallas for the same reason. But when we talk to someone wearing the "wrong" clothes we instantly and instinctively put them in the "them, not us" category. Which breaks that whole "shared experience" thing. Do not wear piercings, jewelry, aftershave, heavy glasses, weird mustache, funny t-shirts, open sandals, tattoos, or anything else that makes you stand out. Do, constructively, be prepared to shift styles easily in any direction it's needed.

Service with a Smile

Many of the favors we ask from society come from people who are paid to "serve" others in one way or another. From security staff to ticketing agents, waiters and salesmen, we put an awful lot of our life in the hands of people who don't know us, and have no reason to help us except that's their job, and maybe we tip them better.

Service staff mostly have lousy jobs. They always deal with problems, people rarely appreciate it when things work well, everyone's stressed and in a hurry, and everyone is always demanding stuff. The default emotional state for most service staff is a mix of boredom, loneliness, jealousy, and hate. That's what you feel when you walk up to a counter and the other person puts on their corporate smile just a little late to hide the "here comes another idiot" look that flashes across their face.

You can almost always turn that negative state of mind into a win-win transaction by smiling happily, confessing that you did something idiotic and really could use some help, and suggesting that it's impossible to fix your messed up situation. One in ten will give you the evil grin of someone who's paying rather too much attention and say, "yes, it's impossible, have a nice day". The other nine will shake their heads to wake up, and start to think of ways to help this kind, somewhat lost stranger who obviously trusts them with their very life.

If you are unsure how to do this, start simply by saying "hello" and "thank you" to every service person you come across. I always greet and thank the security staff at airports, and often get into short friendly discussions with them. Practice when it doesn't matter, so that when it does matter, it comes naturally. You have to repeat an action a thousand times before it becomes automatic.

As a frequent traveler I take the bored malice of security personnel as a rather juicy exercise. Some years ago I was enjoying a late whiskey with a guy called Jean-Paul, the manager of a small hotel in Baguida Plage, Togo. He mentioned that he had a dying Dutchman in one of his rooms, who hadn't paid his bills. Perhaps I could do something? We went over to the room where I saw a shivering, sweating gray body on the bed. It turned out, after much mumbling, that he was called Ruud, and after some long distance phone-based diagnostics with my sister, who's a doctor in Scotland, that he probably had celebral malaria and a failing liver.

The Dutch have such silly names, it's hard to have pity for them. Honestly, what kind of mother would call her son "Ruud"? It's like tattooing "kick me" on his forehead. But it seemed wrong to abandon him. For one thing, Jean-Paul was a friend and dying Dutchmen do not a happy hotel make.

It took some weeks, but eventually we got him patched up, able to stagger about, and with a ticket home thanks to the Dutch consul in Ghana. Slight problem, Ruud was an illegal immigrant, with no papers except a letter from a doctor saying he could travel, and a document from the consulate. Well, what the heck. I rebooked my ticket home so we could travel together, and we hit the airport.

If you ever traveled in sub-Saharan Africa you'll know that it can be quite fun to enter and leave the country. Once in Burkina Faso I was frisked in a private booth, "I know you're hiding something and by golly I'm going to find it" style by the most stunning policewoman I've ever seen. When she'd finished I offered her the contents of my wallet if she would do it again. She exclaimed, "Aha! Illegal currency exportation!", and took the money, but to my eternal regret, waved me out and on.

Ruud could barely walk, was pale as an old brothel sheet, and still shivering from bouts of malaria and the long-term professional drinking that plagues many expatriates in Africa. We went through half-a-dozen layers of interrogation, each time explaining our case to two or three officers, who called their colleagues and superiors to listen to us. I kept my money well hidden, left just a tattered and worn 50 CFA note in my wallet, which I showed pathetically to them, saying, "if this will help, take it". 50 CFA is about 25 cents. They laughed uproariously at that, and let us through with smiles. I used nothing more than comedic hand waving and smooth talking. West African French, like its English, has a lovely song that once you learn, is instantly familiar to the listener and puts him off his guard.

My Dutch companion shivered the whole way to Casablanca, where we repeated the entire exercise with Moroccan security. I think at some point I told them, my friend is going to die and I want to get him home so his wife and thirteen children can bury him properly. Whatever it was, it got us to Belgium, and again through Belgian immigration, where his brother picked him up at the airport.

The man called Ruud survived and rather predictably, and to the dismay of his family, flew right back to Togo.

Yes, social engineering can save your life. No kidding.

Gender Differences

Finally, there are strong gender and cultural differences that you have to learn. I can generalize grossly and say that women open up to smiles and confidence, while men open up to confidence but also, and strangely, to touch. I think it's a form of physical assertion, "look, I'm confident enough to break into your personal space and I'm not showing any hostility or fear, so I'm now your equal, or possibly your superior". I'm talking about hand shaking, hand slapping, arm touching, and most dominantly, shoulder slapping and hugging. Off the football field, those are the only safe male-male contact zones. I've not tried it but I'd predict that grabbing random men's crotches will cause Bad Things to happen unless you're extremely dominant.

There is an easy way to gain the trust of a group of men you don't know. They need to be younger than you (gets easier every day). You find the largest dominant male (the Alpha), go right up to him, smiling and happy, and free of any emotions (especially fear) and start chatting to him. Shake his hand with a "diplomat's handshake" (right hand on his, left hand wrapped around his right forearm), and make him at ease with random chat about his background and occupation. Then break his comfort zone with a shoulder slap and make some comment that sounds idiotic but harmless and fun. Smile broadly and then focus on the rest of the group. That's about it. You are now Guest Alpha and every time you meet them, as long as you don't mess up, you'll be treated with respect, even adoration.

I've done this a hundred times, in every random country and continent I've been to. The fastest time was with a group of a dozen or so drunken and noisy Polish skinhead youths who confronted me outside a bar on a snowy street at 2am one morning in Torun. I spoke to them for about thirty seconds, back-slapped the alpha, then turned and walked back into the bar and most of the group followed me like puppies. We drank together until five in the morning.

Workshop: Upgrades

The incident that prompted me to write this article was getting not one, but two airline upgrades to first class in an hour, without paying anything or using any miles or club points or whatever they call the little virtual "not money, promise!" things. It's fairly unusual. The only previous time I flew in first was in 1995 or so, to New York for a weekend, with a girlfriend at the time who worked for an airline. That trip cost me ten years of my life in a dead-end relationship, and taught me one thing about life: first class is really expensive.

I've been traveling a lot in the last months, business class, and as I woke up at 4am to catch an early flight to London, and then to Dallas, I thought to myself, "I'm going to travel first class today". I like to set my goals high because one thing's for sure: you won't ever do better than you aim for. Doing worse is fine, you just adjust your settings and fire again. I got to the airport early and checking-in, asked if I could use my miles to upgrade. "You need to book that 24 hours in advance. But you can ask at in London, maybe they can do something there," said the British Airways check-in agent. Sure, that's fine. Gold frequent flyer status, they will do that for me.

But we didn't get to London. Instead, we sat on the runway for a while, filling up with extra fuel, while the captain explained that Heathrow had woken up to fresh snow, and had only one runway open. It's a fine English tradition that Heathrow celebrates every few years. An inch of snow falls, and everything comes to a total halt. Things don't just slow down, the system goes from cheery functionality to Texas road kill. In Scotland it takes a foot of snow to make anyone notice, and even then it has to be drifting hard.

With dozens of incoming international flights, we weren't going to get a slot soon, he explained. I had visions of airplanes circling London, captains checking the fuel and thinking, "another two hours and then we drop like a rock". The cabin crumbled. We waited. I chatted to the pretty blond Dutch woman beside me and discovered she was BA staff and lived across the canal from me.

Somewhat later, our captain announced cheerfully that we had gotten a takeoff slot and please could we fasten our seat belts. He also mentioned we'd taken on a lot of extra fuel but no-one really paid attention to that. Blondie told me she was off to San Francisco. I thought about the previous Friday evening, when I'd spent two hours driving ten kilometers, as snow hit Brussels, and predicted that things would not be quite that simple.

The planeful of excited bunnies sat down, and buckled up. An hour later we were circling over Heathrow with the captain telling us, "there's no way to land and the whole of the UK is snowed under. We've asked Heathrow but they can't get us a slot. It's really chaos down there. The closest airport is Brussels. So we're going back, sorry about this, folks". A hundred travel plans broke and cried, and Blondie went to ask the crew what was happening.

She came back and told me that Brussels airport had told the captain to leave anyhow, knowing there were no landing slots in Heathrow. We could have just disembarked without leaving and all found new flights. So, we all lost two hours. The captain didn't tell the passengers this. What was far more scary was that we'd had a near miss with real disaster. It turned out the captain — whose judgment I was trusting less and less by this time — had asked Aberdeen airport (after several others) for landing. Aberdeen was, thank the gods of travel, also snowed under. They refused and that's why we eventually came back to Brussels.

Now I love Scotland, it's my maternal homeland, filled with wrenchingly beautiful lochs and mountains and rivers and landscapes. The Scottish highlands make my heart sing the ancient song of "what the fuck were my ancestors thinking when they came here, and will I ever get warm and dry again?" I love so many things about Scotland: its desolate villages, its ruins and its tragic history of clearances and war and loss, forever immortalized in drunkards ballads like Flower o' Scotland. I love the Scottish currency, which everyone in Scotland believes is the real thing, but which won't buy a loaf of bread in a foreign country.

One time a Scottish army almost conquered London and then turned back because they couldn't find a decent haggis and neeps anywhere south of Brunswick. It took until the Blair government (all Scots) before we took over England properly, and did to it what the English did to Scotland all those centuries.

I love the Scottish weather, which protects you from the skin-damaging sun with a constant protective cloud cover and generous, nay endless rains. The Gaelic word for "weather" is actually "rain". We Scots only discovered sunny weather in 1832 when some travelers returned with stories from the far south. Scottish rain is like Italian traffic: it comes at you from all directions, and though you may at first think it is trying to kill you, all that directed violence is just a form of affection. We love you, says the rain, and here it comes right up your kilt to prove it. How can a true Scot not love the rain back? It is the main reason the English stay away.

Scotland is where I grew up, became a real man, and discovered that the first thing a real man does is leave, by train or plane or boat or hitching a hike in a goods lorry. Scots have a long and proud tradition of picking up and leaving their homeland as soon as it's physically possible. The few times I've been back to Scotland were firm reminders that I'd made the right decision. It is a beautiful, precious homeland that is rightly abandoned by its natives, to make space for tourists. Let the Belgians come to Scotland to shoot grouse and die from exposure in the Cairngorms I say. I'll take beer, and cheese, and quaintly archaic Dutch/French language wars, anytime.

Aberdeen, the oil rig maintenance capital of Scotland, is one of the most boring cities ever. Its main claim to fame is that its squat red-gray granite buildings exude a gentle alpha radiation that kills anyone not mobile and smart enough to leave the place. Also, it's the where the game of golf was, I'm told, invented. It makes sense, since golf is the third most boring British sport (and they invented a lot of those) after cricket, and the infamous Scottish sport of "hill walking". Yes, the Scots literally go outside, in the rain (which will occasionally turn to snow, as we see), put on a cheerful smile and walk up, and then down hills. Sometimes several hills in a row. It's not even as if there's a great pub at the top. Just a random pile of rocks they call a cairn. Get this… it's not enough that they walk up near-vertical gradients for pure pleasure, but they will actually carry rocks up with them as they go. For fun. Go figure.

So as you can imagine, when Blondie confided closely that we'd escaped a snowy landing in Aberdeen didn't just tickle me, I whooped so loudly that several other business class passengers peered over at me, and then Blondie, visibly wondering what kind of deal we'd just sealed. Of all the fates I could plausibly expect when getting into a plane to London — hours on the tarmac, burning falling death into the Channel, downgrade to economy, having an triple-sized American sit beside me, no power outlet in Business (hey Korean Air, I'm talking to you! And you too, KLM. I mean, wtf?) — getting shunted to a snow-covered Aberdeen was possibly the most horrific. That "American" part was a joke. I love Americans! You guys have heating!

Having escaped near-Aberdeen, nothing could depress me and I let my euphoria last for hours. Good thing too, because it was one of those "can someone remind me why I got up this morning?" days.

Belgium is a fine place to live. I really like it. The food is excellent and diverse. It's not too expensive. The cities are small and sociable. Brussels, especially, is kind to immigrants and looks after its poor and weak. They don't dub the movies like the French and Germans do, they use subtitles like civilized people. They even subtitle the beer glasses, so we can tell the difference between a Westmalle Trippel and an Orval. The beer of course, is a book in itself. It's the only country that could — and I swear by all that's holy that this is the honest truth — plan to convert its main stock exchange building, the beautiful Bourse, into a beer museum. How awesome is that? Belgium rocks.

But there's one thing about Belgium. When things go even slightly wrong, they go totally wrong. We have one hung election, and the country goes without a government for a year and a half. One driver brakes suddenly, and there's a traffic jam that lasts for hours. It snows one inch on a Friday afternoon, and there's enough traffic jams to wrap the country three times.

And one plane returns unexpectedly, with only two hours' warning, and there is literally not a single baggage handler available to deplane the bags. So we sat there, in the arrival hall, watching our luggage dramatically and suddenly not appearing on belt 8. For hours. And hours. The smarter folk moved briskly upstairs to the BA ticketing desk, leaving one of their party downstairs to wait for their errant freight. I found a woman at the Avia desk, responsible for ground handling. I asked, can I leave my as yet — and by the way it's been like an hour and seriously, what's up — pending luggage here while I go try to get a new ticket? "Sure", she said, "just show your ID and bag ticket to security to get back in. And the bags will arrive at any moment".

Upstairs there were maybe thirty people already waiting, so I joined the queue. The last person in the queue was Blondie, so we smiled our "hey, we meet again!"s and got back to our chit chat. Every hour or so I dashed back downstairs, negotiated past security, and went back to belt 8 where the captain was bravely going down with his ship. Passengers sat and stood, getting more and more impatient. It was the utter lack of any news that was the worst thing. They flocked around him, telling him his company sucked.

I'd dash back upstairs to continue chatting with Blondie and waiting. The agents were interminably slow. And terribly organized. Surely they could do passengers in groups. All those going to Dallas. All those to San Francisco. Business class first, at least. But it was one by one, first come, first serve, if you can count taking an average of fifteen minutes to reroute one person "service".

Finally the bags arrived, in random batches, and people came upstairs looking haggard and shocked at the queue. Which got longer, and longer, until it stretched right across the departures hall. I had to go film it, it was so surrealistic. We'd been waiting for four hours and there were at least a hundred people behind us. One dignified old man in a suit who'd sat in front of me in business class shook his head in dismay and abandoned his bags in front of the ticket desk. He went outside to shoot himself or perhaps set himself on fire. Blondie finally told me she was going home. I think she'd just stayed to keep me company and my place in the queue because after all, it was her airline and she could rebook herself trivially.

Some other passengers going to Dallas joined with me, and then it was my turn. The agent told us, "I can get you to Amsterdam, then Atlanta, then Dallas. A full economy flight, no business class, arriving at Dallas at midnight". The other Dallas-bound people asked, "we paid for business class, how do we get our refund?" "You have to write and ask for it, I can't help you with that," said the agent. I told the agent, thanks but no, I'm not traveling economy. I have a fifteen hour flight to Korea in two days and unlike that nice old man who just went outside to jump under a taxi, have no wish to die quite yet. Please check for tomorrow. She told me she could get me to Chicago on United, thence to Dallas on American. Business class? I asked. "Yes," she confirmed. Departure time? Eleven A.M. from Brussels, she said. OK, that means I can wake at a normal hour, which is excellent, thank you, I told her, please book it.

Outside, there were the usual airport taxis. These are a form of extortion practiced in every civilized country, whereby a taxi firm will pay a percentage to the airport in return for a monopoly, and large warnings of the form, "Warning: for your own safety, only take official taxis!" It's funny because half the point of a taxi ride anywhere is that you never know whether the driver will rip you off or not, and if so, how bad it can get. An "official taxi" is such a weird concept. Why don't they just push a subway then?

I was once in Burkina Faso and heard of a taxi driver being hunted down, arrested and then shot (with bullets, to be clear) for overcharging a passenger. He'd driven one of the president's friends — a visiting president — from his hotel to the conference center. He charged about $200, which is an immodest sum but not excessive for a VIP. The problem was the two buildings were next door to each other. In memoriam of that taxi driver, I never take "official taxis".

More to the point, and possibly because I'm half Scottish, they're over-priced. There is a trick, which works in most modern airports. All taxis leaving the airport are crooks. Official crooks. So instead you go to the departure area and wait for ordinary taxis arriving from town. Flag one down and ask, how much to get into town? You can negotiate a good deal because they have to drive back empty otherwise. I waited for a while, caught a normal taxi and had a nice drive back home for a third of the "official" price. I got home, grateful to spend another day with my surprised and joyful kids, and repaired the damage to my meetings, and hotel and car reservations.

The next morning at a far more decent hour, but still having missed breakfast, I checked-in with United and then went to the BA desk to ask the agent about collecting my missing miles. She was one of those who had been handling the problems yesterday. She looked tired and stressed, face thin and taut. Needed a cigarette and a coffee and a big warm hug. Instead I gave her a smile and said, in Dutch (most of the staff at Zaventem are trilingual Flemish), thanks for yesterday by the way, you all did an unbelievable job. She beamed for a second. "Thanks, it was kind of difficult." Well, I said, the snow messed up everything, but you all worked so hard to fix things. Sometimes it's best to know what parts to leave out. Yes, you were unbelievably slow, and you worked so hard and yet managed to make your passengers wait most of a day, without news or food or seats, like penniless refugees hoping to catch the last boat out of a burning city.

She beamed again as we discussed my miles and she suggested I email the frequent flyer program. I've already used BA's online support and it's excellent, so that was a great answer. I smiled and said bye-bye. Next to her (it's one long One World counter) was the AA agent.

AA lady, also Flemish, checked me in for Chicago-Dallas but couldn't issue the boarding pass. Looking at the screen I noticed I was in row 18, which is economy. Business is full, I asked? No business, just first, she replied. There are some seats available but your ticket is Y class, economy. Back to BA lady, explaining, I really don't do economy. "But policy won't let me issue a first class ticket," she apologized.

I nodded, happily, not disagreeing. Policy is a bitch and she was undoubtedly right. I don't do economy, I repeated. I have to leave tomorrow to Korea. Long flights. Delays which already cost me a lot. You might want to put me in first class, that would make me very happy. I'd be very sad to travel in economy because I paid for business, and what they call first class on domestic US flights is just business, but what's worse, I like BA and then I'd be forced to find another airline. And as I travel once a month, in business, that would be really sad. I shrugged my apology but continued to smile at her happily. We would fix this tragic situation together, obviously.

"Oh, but if we issued you this ticket yesterday, there was no room in first," she thought out loud. I nodded and indicated the AA agent. She told me there was space in first, I said. BA lady ticked at her keyboard. "Yes, right, there is." Let me see. A little later, she shook her head, but said, "OK, I'm canceling your existing ticket, and issuing you a new one in class A" (first class). I thanked her sincerely and waited like a happy Buddha statue waiting for rain.

The computer system didn't let her though. I watched with sympathy and patience as she fought the software that had been specifically programmed to not allow ticket agents put business class passengers into "let's agree to not call it business" class. After a while she swore at it and said, "hang on, Mr Hintjens" and disappeared into the back office. She came back with a paper voucher slip and started writing on it.

Finally she had it all, and gave it to me. "This is for United, this second copy is for AA in Chicago". I thanked her and wished her well, and crossed the hall to the United desk. There I explained my tragic story to two ladies of the Flemish persuasion, as if it was a comic story, including sound effects and hand movements as the plane nearly landed in Aberdeen but finally came back to Brussels. Oh, and this voucher is for you, I finished. They were somewhat confused since I already had my boarding pass. "Did you get an award ticket? It says here on the old ticket, value zero". Nope, I explained, BA bought those tickets. Involuntary rerouting. Snow, chaos. Aberdeen, the whole mess. All in Dutch. Languages are like pre-paid plastic you pull out at the right moments.

United lady #1 ticked on her keyboard and nodded, "OK, it's done. Thanks a lot for bringing us that voucher!" I shrugged happily, no problem. I'm looking forward to trying United, it's been BA all the time so far, and their handling was pretty tragic yesterday. Then, I asked, "so since I am so nice, how about an upgrade to first?" They both smiled and looked at each other and said "ha!", but not unkindly. I gave them my best happy bunny smile and repeated, "so, how about an upgrade to first?" United lady #1 asked United lady #2, "could we do it?" and United lady #2 thought and said, "in theory, yes, but the price differential is huge!"

I'm not a frequent flyer on United, but I'd learned a few minutes earlier that ticketing agents have power. I smiled happily, "well, the original ticket was priced zero, and you have the voucher now, so the upgrade isn't really costing anything, if you think about it". United lady #1 shook off that incoherent sentence and ticked some more. "I've put you on the upgrade list. They can upgrade you at the gate if they need to for operational reasons".

Wow, you're both amazing, I said, happy as a kid opening his Christmas presents. Upgrade list is already half the struggle. It means you're a VIP of some kind.

I went to the gate, via the crowded mess that was United's lounge. It was the "insert random airline here" lounge you get in every airport. Every last croissant and cheese plate had been eaten and every seat was filled. I didn't see the difference between that and the departure hall downstairs except there were more fat businessmen and laptops. Sigh. All this effort and I still have to search for a place to sit.

Across the corridor, the BA lounge was empty and visibly well-stocked with edibles. The man at the desk was the same one from the day before. Unfriendly, bored, uninterested. Hi, I'd like to get into the lounge please, I told him and showed him my boarding pass. "You're not flying British Airways", he told me. Yes, but I have a gold frequent flyer card and look at me, I'm a nice person. "Can't help you," he said, "my boss wouldn't like it". I looked around. Maybe his boss was hiding somewhere. But no, there was no boss. Just an empty, well-stocked lounge with lots of empty seats. BA does a great lounge, its one of the reasons I like the airline. Look, I said, I am a BA passenger and you guys canceled my flight and made me a day late and now you can't get me into the lounge?

But it wasn't going to work. He wasn't enjoying the discussion a bit and I realized I'd gotten off entirely on the wrong foot. He was the bouncer, francophone and I'd come right up and asked in English to get in. The answer was going to be "no" even if the place was totally empty and I had a hot woman in each arm and a pound of cocaine in my pocket. You should start with "hi, how is stuff?", move to some irrelevant chit-chat about the weather and yesterday's chaos, and then casually arrive at how the other lounge is so filled with people, and shrug, and just as he starts to realize he's the one all alone and bored, start to leave. That's when he calls, "sir, come and enjoy our lounge if you want to".

Some fights you can't win, some you will win anyhow. Save your energy for those you can profitably fight.

I went back to the United lounge and drank an espresso, standing up. I chatted (in French, because most manual workers in the airport are francophone) to the woman cleaning up and she told me there would be more snacks in thirty minutes. Too late for my grumbling belly. I went to the gate where the flight to Chicago was already boarding.

At the desk I showed my boarding pass to a neatly styled agent and asked, politely, if I had been upgraded. It seems a job for women and gay men, for some reason. Language and people skills, perhaps. Anyhow, he gave me a broad smile and said, "Ah, Mr Hintjens, I did the upgrade, here's your new boarding pass." I took it and thanked him, shook his hand and walked tenderly to seat 2A, waiting for someone to shout, "hey, smile, we punked you!"

First class isn't what it was in 1995, when we were fed more delicious morsels and champagne than the Playboy mansion holds on a holiday weekend. In the heyday of Sabena, the first class cabin attendants outnumbered the passengers and we sat in the nose of a 747 like kings. First class these days is like business class for very fat people. But it was still awesome and now I am totally going to ask for an upgrade every single time I fly.

Conclusion

I've presented you with a set of social engineering tools that let you turn everyday interactions with others into more meaningful and mutually profitable transactions.

The basic technique is to feel and then turn off your negative emotional responses such as fear, anger, self-pity. Then you find joy in the moment, and then you express that joy in your posture, smile, and language. You must speak with pleasure and say things that make the other person happy too.

If you want something, you need to know what it is, and find a way to express that. You never come out cold with a demand, but you can ask for anything if you have warmed up the other person with stories and humor. The worst they will say is "no", and even then, it will often be a challenge to try harder.

When you are certain of your success, without fear, it subtly infects others. You must be patient, and be able to get close to others by speaking their language, and dressing without prejudice. No matter what the culture nor the situation, you should be able to make anyone your friend in a few minutes, and you should sincerely enjoy the experience.

These techniques are not easy. Learn them by starting with the simplest possible steps, and building up your skills piece by piece. Just politely greeting every person you deal with is a good start.

Social engineering is a powerful toolkit that I use and practice daily when designing new software products with my teams, when negotiating contracts, when buying gas, when talking to people in a bar, indeed when doing anything at all with other people. For sure, some will consider it manipulative, but that is because they assume pessimistically that if you get something from someone, that other person must have lost something.

As I said, social engineering is essentially about creating win-win transactions. Don't confuse social engineering, a form of trade, with social hacking, which is a form of theft.

Let me know how social engineering works for you.

by pieterh

The problem with open communities is that you can't really turn away people. I explained in the Economics of Evil how annoying people can be healthy for communities. But I've also seen communities fracture and die because a few individuals insisted on turning every issue into arguments about style and personalities. It can be incredibly toxic, and confusing to the majority of people who come to the table to be helpful, and be helped in return.

It's not specifically an open source problem. In many closed teams, the level of hostility and paranoia is quite astonishing. I've watched people turn every email exchange they have with others into arguments. It can be educational, even fun, to watch from a distance. But what if you're the person at the other side? It can feel like being in a bad relationship, where no matter what you do, you will always lose.

So, from so much experience that I can almost call this science, here are a set of techniques you can use to moderate hostility, and eventually contain it, in a group. These techniques won't always work, and if they don't, and the problems continue, you need to evaluate your presence in the group.

First, respond to personal attacks and insults with humor. I'm regularly called names by people who sincerely think that by pinning a label onto someone, they prove a point. It is a schoolyard mentality, and the correct response is to laugh. If you can't find the humor in it, you say nothing. Don't laugh at the other person but if necessary, at yourself. It's hard to humiliate someone who shows no ego.

Second, understand the mentality of the person doing the damage. Most often, aggressive and emotional behavior comes from deep insecurity and feelings of inferiority. People who really do not believe in themselves often compensate with a superiority complex that turns aggressive at a button press.

You need to understand that the person launching diatribes at you is really sincere. They probably believe they are all that stands between order, and chaos. You will often hear arguments about "quality" and how good things were in the past. But this intense, emotional attachment to a myth of quality is a kind of mental disorder, an obsessive compulsive tick that you can't reason with, convince, or shift except with patience.

An insecure person, searching for order, will create a mental construction based on their own superiority, and that of those they can manipulate. At the same time, they distance themselves from any inputs that could disrupt this. If they write code, they don't use it. If they build processes, they make them as complex and arcane as possible, always citing "quality" as their rationale. This drives others away, which reinforces their model. They will use long, complex arguments to escape any real challenge. They do not respect others' experience (which will be different from theirs, and thus a challenge), while over-emphasizing their own (which reinforces their self-belief).

The more such individuals (or groups, because this can spread like a form of cult) entrench themselves in their self-belief, the more hostile they become to any opposing thoughts. They turn discussions into conflicts, and then use those conflicts to hammer in a "them and us" mentality.

For a weak, insecure person to shift their attention from their perceived ordered world to a new one takes time, and sometimes never happens. If it never happens, and the person continues to create friction, you have to choose some more permanent solution.

But before then, understanding that aggression comes from insecurity, you can assuage it by stepping back from conflict, and stressing how you will ensure security and continuity. When you are attacked, respond with platitudes and commiserations. Assure the other person that everything will be OK. Assure them that things will not change dramatically.

It can be a lot of effort, so the person in question has to be worth it. And, if this pattern has gone too far, you often cannot do anything. Then you have two choices. Either you leave the group, and abandon any investment in it. Or, you force out those people and recreate the group in a healthier form. Leaving a group is traumatic but simple: you stop reading the messages, and you do something else with your time. When you stop the fights you'll suddenly have lots of free time.

Ejecting troublemakers from a group is harder, and requires that a substantial majority of the group agree. The process is then simply to ignore the troublemakers. Since they have come to rely on conflict for self-identity, silence will rapidly bore them, and then repel them. This tactic is successful, if the whole group can maintain it.

You will realize that a lot of this discussion, and the tactics, mirror those you might need to use in abusive personal relationships. It's not surprising. Communities are made of people who often use exactly the same patterns in their personal lives as in their public lives. A solitary young male who is unable to create long term relationships with women will, trivially, express exactly the same angsts and emotions onto groups as he does onto women.

It might strike you that this article was triggered by recent email threads on the zeromq-dev list. Which is true, but the people and mindsets I'm describing are ones I've seen way too often in other places, principally the FFII from 2005-2008 and the AMQP workgroup.

by pieterh

Before we get to SOD, let's look at two other classic design processes. These don't work, yet are firmly applied by a majority of engineers and designers, especially in software where it's possible to construct byzantine complexity. They are slow-motion tragedies but can be fun to watch, from a safe distance.

Trash-Oriented Design

The most popular design process in large businesses seems to be "Trash Oriented Design", or TOD. TOD feeds off the belief that all we need to make money are great ideas. It's tenacious nonsense but a powerful crutch for people who lack imagination. The theory goes that ideas are rare, so the trick is to capture them. It's like non-musicians being awed by a guitar player, not realizing that great talent is so cheap it literally plays on the streets for coins.

The main output of TODs are expensive "ideations", concepts, design documents, and finally products that go straight into the trash can. It works as follows:

• The Creative People come up with long lists of "we could do X and Y". I've seen endlessly detailed lists of everything amazing a product could do. Once the creative work of idea generation has happened, it's just a matter of execution, of course.

• So the managers and their consultants pass their brilliant, world shattering ideas to "user experience" designers. These talented designers take the tens of ideas the managers came up with, and turn them into hundreds of amazing, world-changing designs.

• These bountiful and detailed designs get passed to engineers, who scratch their heads and wonder who the heck came up with such stupid nonsense. They start to argue back but the designs come from up high, and really, it's not up to engineers to argue with creative people and expensive consultants.

• So the engineers creep back to their cubicles, humiliated and threatened into building the gigantic but oh so elegant pile of crap. It is bone-breakingly hard work since the designs take no account of practical costs. Minor whims might take weeks of work to build. As the project gets delayed, the managers bully the engineers into giving up their evenings and weekends.

• Eventually, something resembling a working product makes it out of the door. It's creaky and fragile, complex and ugly. The designers curse the engineers for their incompetence and pay more consultants to put lipstick onto the pig, and slowly the product starts to look a little nicer.

• By this time, the managers have started to try to sell the product and they find, shockingly, that no-one wants it. Undaunted and courageously they build million-dollar web sites and ad campaigns to explain to the public why they absolutely need this product. They do deals with other businesses to force the product on the lazy, stupid and ungrateful market.

• After twelve months of intense marketing, the product still isn't making profits. Worse, it suffers dramatic failures and gets branded in the press as a disaster. The company quietly shelves it, fires the consultants, buys a competing product from a small start-up and rebrands that as its own Version 2. Hundreds of millions of dollars end up in the trash.

• Meanwhile, another visionary manager, somewhere in the organization, drinks a little too much tequila with some marketing people and has a Brilliant Idea.

TOD would be a caricature if it wasn't so common. Something like 19 out of 20 market-ready products built by large firms are failures. The remaining 1 in 20 probably only succeeds because the competitors are so bad.

The main lessons of TOD are quite straight-forward but hard to swallow. They are:

• Ideas are cheap. No exceptions. There are no brilliant ideas. Anyone who tries to start a discussion with "oooh, we can do this too!" should be beaten down with all the passion one reserves for traveling musicians. It is like sitting in a cafe at the foot of a mountain, drinking a hot chocolate and telling others, "hey, I have a great idea, we can climb that mountain! And build a chalet on top! With two saunas! And a garden! Hey, and we can make it solar powered! Dude, that's awesome! What color should we paint it? Green! No, blue! OK, go and make it, I'll stay here and make spreadsheets and graphics!"

• The starting point for a good design process is to collect problems that confront people. The second step is to evaluate these problems with the basic question, "how much is it worth to solve this problem?" Having done that, we can collect a set of problems that are worth solving.

• Good solutions to real problems will succeed as products. Their success will depend on how good and cheap the solution is, and how important the problem is. But their success will also depend on how much they demand in effort to use, in other words how simple they are.

Hence after slaying the dragon of utter irrelevance, we attack the demon of complexity.

Complexity-Oriented Design

Really good engineering teams and small firms can usually build good products. But the vast majority of products still end up being too complex and less successful than they might be. This is because specialist teams, even the best, often stubbornly apply a process I call "Complexity-Oriented Design", or COD, which works as follows:

• Management correctly identifies some interesting and difficult problem with economic value. In doing so they already leapfrog over any TOD team.

• The team with enthusiasm start to build prototypes and core layers. These work as designed and thus encouraged, the team go off into intense design and architecture discussions, coming up with elegant schemas that look beautiful and solid.

• Management comes back and challenges team with yet more difficult problems. We tend to equate value with cost, so the harder the problem, and more expensive to solve, the more the solution should be worth, in their minds.

• The team, being engineers and thus loving to build stuff, build stuff. They build and build and build and end up with massive, perfectly designed complexity.

• The products go to market, and the market scratches its head and asks, "seriously, is this the best you can do?" People do use the products, especially if they aren't spending their own money in climbing the learning curve.

• Management gets positive feedback from its larger customers, who share the same idea that high cost (in training and use) means high value. and so continues to push the process.

• Meanwhile somewhere across the world, a small team is solving the same problem using SOD, and a year later smashes the market to little pieces.

COD is characterized by a team obsessively solving the wrong problems to the point of ridiculousness. COD products tend to be large, ambitious, complex, and unpopular. Much open source software is the output of COD processes. It is insanely hard for engineers to stop extending a design to cover more potential problems. They argue, "what if someone wants to do X?" but never ask themselves, "what is the real value of solving X?"

A good example of COD in practice is Bluetooth, a complex, over-designed set of protocols that users hate. It continues to exist only because there are no alternatives. Bluetooth is perfectly secure, which is close to useless for a proximity protocol. At the same time it lacks a standard API for developers, meaning it's really costly to use Bluetooth in applications.

On the #zeromq IRC channel, Wintre wrote of how enraged he was many years ago when he "found that XMMS 2 had a working plugin system but could not actually play music."

COD is a form of large-scale "rabbit holing", in which designers and engineers cannot distance themselves from the technical details of their work. They add more and more features, utterly misreading the economics of their work.

The main lessons of COD are also simple but hard for experts to swallow. They are:

• Making stuff that you don't immediately have a need for is pointless. Doesn't matter how talented or brilliant you are, if you just sit down and make stuff, you are most likely wasting your time.

• Problems are not equal. Some are simple, and some are complex. Ironically, solving the simpler problems often has more value to more people than solving the really hard ones. So if you allow engineers to just work on random things, they'll most focus on the most interesting but least worthwhile things.

• Engineers and designers love to make stuff and decoration, and this inevitably leads to complexity. It is crucial to have a "stop mechanism", a way to set short, hard deadlines that force people to make smaller, simpler answers to just the most crucial problems.

Simplicity-Oriented Design

Simplicity-Oriented Design starts with a realization: we do not know what we have to make until after we start making it. Coming up with ideas, or large-scale designs isn't just wasteful, it's directly toxic to designing the truly accurate solutions. The really juicy problems are hidden like far valleys, and any activity except active scouting creates a fog that hides those distant valleys. You need to keep mobile, pack light, and move fast.

SOD works as follows:

• We collect a set of interesting problems (by looking at how people use technology or other products) and we line these up from simple to complex, looking for and identifying patterns of use.

• We take the simplest, most dramatic problem and we solve this with a minimal plausible solution, or "patch". Each patch solves exactly a genuine and agreed problem in a brutally minimal fashion.

• We apply one measure of quality to patches, namely "can this be done any simpler while still solving the stated problem?" We can measure complexity in terms of concepts and models that the user has to learn or guess in order to use the patch. The fewer, the better. A perfect patch solves a problem with zero learning required by the user.

• Our product development consists of a patch that solves the problem "we need a proof of concept" and then evolves in an unbroken line to a mature series of products, through hundreds or thousands of patches piled on top of each other.

• We do not do anything that is not a patch. We enforce this rule with formal processes that demand that every activity or task is tied to a genuine and agreed problem, explicitly enunciated and documented.

• We build our projects into a supply chain where each project can provide problems to its "suppliers" and receive patches in return. The supply chain creates the "stop mechanism" since when people are impatiently waiting for an answer, we necessarily cut our work short.

• Individuals are free to work on any projects, and provide patches at any place they feel it's worthwhile. No individuals "own" any project, except to enforce the formal processes. A single project can have many variations, each a collection of different, competing patches.

• Projects export formal and documented interfaces so that upstream (client) projects are unaware of change happening in supplier projects. Thus multiple supplier projects can compete for client projects, in effect creating a free and competitive market.

• We tie our supply chain to real users and external clients and we drive the whole process by rapid cycles so that a problem received from outside users can be analyzed, evaluated, and solved with a patch in a few hours.

• At every moment from the very first patch, our product is shippable. This is essential, because a large proportion of patches will be wrong (10-30%) and only by giving the product to users can we know which patches have become problems and themselves need solving.

SOD is a form of "hill climbing algorithm", a reliable way of finding optimal solutions to the most significant problems in an unknown landscape. You don't need to be a genius to use SOD successfully, you just need to be able to see the difference between the fog of activity and the progress towards new real problems.

A really good designer with a good team can use SOD to build world-class products, rapidly and accurately. To get the most out of SOD, the designer has to use the product continuously, from day 1, and develop his or her ability to smell out problems such as inconsistency, surprising behavior, and other forms of friction. We naturally overlook many annoyances but a good designer picks these up, and thinks about how to patch them. Design is about removing friction in the use of a product.

Conclusions

There are many aspects to getting product-building teams and organizations to think wisely. You need diversity, freedom, challenge, resources, and so on. I discuss these in detail in my forthcoming book, Culture and Empire. However, even if you have all the right ingredients, the default processes that skilled engineers and designers develop will result in complex, hard-to-use products.

The classic errors are to focus on ideas, not problems; to focus on the wrong problems; to misjudge the value of solving problems; not using ones' own work; and in many other ways to misjudge the real market.

Simplicity Oriented Design is a reliable, repeatable way of developing world-class products that delight users with their simplicity and elegance. This process organizes people into flexible supply chains that are able to navigate a problem landscape rapidly and cheaply. They do this by building, testing, and keeping or discarding minimal plausible solutions, called "patches". Living products consist of long series of patches, applied one atop the other.

SOD works best for software design, and some open source projects already work in this way. However many or most still fall into classic traps such as over-engineering and "what if" design. Wikipedia is a good example of SOD applied to the non-software domain.

I use SOD daily to build, and help others build, world-class commercial products. If you find SOD interesting and useful, read Culture and Empire when it comes out, particularly chapter two, where I explain the science of Social Architecture.

This article is an extract from Chapter 6 of the ZeroMQ book, on community building. If you like this, buy the book :-)

by pieterh