C4.1 claims its first goal is to "maximize the scale of the community around a project." Community means many things. Above all it means life. If no-one uses or contributes to your software, the code does not matter.

Here is my analysis of the growth of the ZeroMQ community since we moved to git in mid-2009:

The picture speaks volumes. You can see the activity grow, peak, and slow down for each major version. You can see the clear difference between "before" and "after" the fork in late 2011, when the original maintainers left to create Crossroads.io, and ZeroMQ moved to the C4.1 process.

What is most fun is that ZeroMQ takes zero management these days. It is self-steering, self-feeding, and self-organizing. We have applied C4.1 to old projects like libzmq, and brand new ones. The results are the same. If the project does something useful, it takes off, and contributors join, and everyone enjoys themselves.

C4.1 is an exothermic process. It produces almost perfect software. ZeroMQ's oddities all date from before 2012, and are slowly disappearing. ZeroMQ master is almost always stable. It just does not break stable APIs or protocols. The mailing list is polite and happy. We do not argue.

If you run an open source project, and your commit chart does not look like this, go read ZeroMQ RFC 22.

Here is how I produced this chart:

• Take all of the repositories in the ZeroMQ organization on github. There are many times more projects that aren't in the organization (about 20 for each one in the organization). I ignored these. There are "stabilization forks" of projects, like zeromq4-x. I ignored these as well.

• Clone each repository to my laptop, then listed all commits since the start, using this command: git log —date=short —format="format:%ad", for each project.

• Get the number of commits per day using this command: sort activity.log | uniq -c

• Run a Perl script that calculates the totals per week, and prints out a comma-separated values file.

• Load that into LibreOffice and generate a graphic.

by pieterh

What's the Problem?

The other day a friend asked me what the limit was on zproto's command numbering. My spidey sense immediately flashed, "Danger!" When someone asks "is it normal that the cat makes that strange noise?" the prudent person asks, "what precisely are you doing to the poor animal?" So it also goes for protocol tools like zproto.

Let me start by saying, if your protocol has even dozens, let alone hundreds, of commands, then you are Doing it Wrong. What "wrong" means is that you'll end up spending more time and money than you need to. Not in a good way either. While buying a nice, really good pair of shoes can be cathartic, there is nothing healing about telling your boss, "sorry, we can't make that essential change because it would break everything."

We sometimes mix the concepts of "API" and "protocol". I've done this myself, designing protocols that looked like APIs, and designing APIs that act like protocols. However it's like putting ginger in a curry. The art is to add spice without breaking the taste of the dish.

There are good APIs and there are poor APIs. A good API at least organizes itself into silos, which we can call "packages" or "classes". The poorly-designed API is typically one large bundle of loosely related methods. However, both map to nasty, fragile protocols.

Let me show with an example. AMQP (my favorite example of smart people making stupid mistakes) had server-side objects like "exchange" and "binding". So the API-protocol fudge had methods like "exchange.declare" and "binding.delete". On the plus side, the protocol was neatly organized into silos, and was easy to use as an API.

On the minus side, it was fragile as heck. The prime directive of every protocol should be "Interoperate!", and AMQP could not even interoperate with itself. AMQP/0.9.1 did not talk to AMQP/0.8. Let's not mention the alien invader that is currently wearing AMQP's face in public.

APIs have some endearing properties:

• They change often, because they are mapped closely to business language that is constantly evolving.
• When you use the wrong version, they tend to crash.

True, some platforms have managed to keep compatibility over decades. Windows and Linux, for instance. There can be a lot of pain involved.

More usually, though, when we turn an API with hundreds of methods into a protocol, we create a monster. If the monster dies rapidly, fine. If the monster lives and grows, that means real trouble ahead.

In a distributed system, it is essential to be able to evolve pieces independently. This also means evolving the business semantics independently across different parts of the system. Once you cast these semantics into protocols, that becomes approximately impossible.

With AMQP we could evolve the protocol for a few years because (a) it was not a real distributed system, all clients talking to central brokers, and (b) we could make brokers like OpenAMQ and RabbitMQ talk several versions of the protocol. Even then, the cost of change was so insanely high that we simply froze the protocol, and then it died (perhaps from frostbite).

Hell is Large Contracts

Beware of large integrated contracts. Do you really want your electricity, Internet, coffee, and cat food from a single supplier, on one contract? There's a reason monopolists and cartels love "bundling" and consumers hate them. It raises the switching costs ("I have a better catfood supplier") and so lets the supplier raise costs overall.

What we want are small, focussed contracts with the right level of abstraction. Heck, we don't even need very good contracts at the start. If switching costs are low to zero, we can improve individual contracts cheaply over time.

Back to AMQP as example. The protocol raised server objects like "exchange", "binding", "queue", and "consumer" to first class entities. Adding, removing, or changing these had the effect of changing the whole contract. Making a single tweak to any of these broke interoperabilty for everything.

This is what you get when you map an API (even one that's neatly designed) to a network protocol. A large, fragile contract that punishes experimentation and evolution, and creates pseudo-distributed systems you cannot improve gradually over time.

The Resource Abstraction

The answer I'm going to present — it's not the only one, yet it's a good one — is old and surprisingly boring. That is, instead of presenting your business objects as a set of objects with arbitrary methods and properties, use abstract resources.

With abstract resources we need only four methods: create, fetch, update, and delete. A resource is then a document (using a self-describing format like XML or JSON). Resources can be nested. Instead of executing methods on objects, we modify properties on a resource.

The best known design for this is REST, based on HTTP. It uses POST, GET, PUT, and DELETE for method names, and a host of headers like "If-Modified-Since" to condition how these work. RESTful APIs (so-called) are in fact decent protocols. Take a look at the OpenStack Compute API. You can see that there are dozens of distinct contracts. Requests and responses are both extensible by design.

Mapping REST to ZeroMQ

We can design a generic RESTful access protocol and map it to ZeroMQ. We did the foundation work some years ago for RestMS. I don't think many people picked that up at the time. However today it looks quite useful for ZeroMQ applications struggling with how to design their protocols.

Our generic use case is a server that holds a set of resources belonging to, and managed by, a set of remote clients. We wish to build APIs, based on network protocols that allow clients and servers to be distributed across both local and wide area networks. We hold that such protocols are essential for the clean separation of components of a distributed architecture.

Protocols define agreed and testable contracts. Our resource access protocol has two levels of contract:

• Agreement on the resources that the server provides; the names and properties, and semantics of such resources; and their relationship to one another. This consists of any number of independent "resource schemas".

• Agreement on how to represent resources on the wire, how to navigate the resource schema, create new resources, retrieve resources, return errors, and so on. This is the "access protocol", which XRAP provides.

The main goal of XRAP is to make it extremely cheap to develop, test, improve, and discard resource schemas. That is, to allow specialists in this domain to develop good schemas with minimal marginal cost. XRAP achieves this economy is by splitting off, and independently solving the access protocol.

• In technical terms, it allows the provision of 100% reusable solutions (libraries and internal APIs in arbitrary languages) for the access and transport layers.

• In specification terms, it allows the formal specification of schemas as small independent RFCs.

Here is the XRAP encoding over ZeroMQ:

xrap_msg = *( POST | POST-OK | GET | GET-OK | GET-EMPTY | PUT | PUT-OK | DELETE | DELETE-OK | ERROR ) ; Create a new, dynamically named resource in some parent. POST = signature %d1 parent content_type content_body signature = %xAA %xA5 ; two octets parent = string ; Schema/type/name content_type = string ; Content type content_body = longstr ; New resource specification ; Success response for POST. POST-OK = signature %d2 status_code location etag date_modified content_type content_body status_code = number-2 ; Response status code 2xx location = string ; Schema/type/name etag = string ; Opaque hash tag date_modified = number-8 ; Date and time modified content_type = string ; Content type content_body = longstr ; Resource contents ; Retrieve a known resource. GET = signature %d3 resource if_modified_since if_none_match content_type resource = string ; Schema/type/name if_modified_since = number-8 ; GET if more recent if_none_match = string ; GET if changed content_type = string ; Desired content type ; Success response for GET. GET-OK = signature %d4 status_code content_type content_body status_code = number-2 ; Response status code 2xx content_type = string ; Actual content type content_body = longstr ; Resource specification ; Conditional GET returned 304 Not Modified. GET-EMPTY = signature %d5 status_code status_code = number-2 ; Response status code 3xx ; Update a known resource. PUT = signature %d6 resource if_unmodified_since if_match content_type content_body resource = string ; Schema/type/name if_unmodified_since = number-8 ; Update if same date if_match = string ; Update if same ETag content_type = string ; Content type content_body = longstr ; New resource specification ; Success response for PUT. PUT-OK = signature %d7 status_code location etag date_modified status_code = number-2 ; Response status code 2xx location = string ; Schema/type/name etag = string ; Opaque hash tag date_modified = number-8 ; Date and time modified ; Remove a known resource. DELETE = signature %d8 resource if_unmodified_since if_match resource = string ; schema/type/name if_unmodified_since = number-8 ; DELETE if same date if_match = string ; DELETE if same ETag ; Success response for DELETE. DELETE-OK = signature %d9 status_code status_code = number-2 ; Response status code 2xx ; Error response for any request. ERROR = signature %d10 status_code status_text status_code = number-2 ; Response status code, 4xx or 5xx status_text = string ; Response status text ; Strings are always length + text contents string = number-1 *VCHAR longstr = number-4 *VCHAR ; Numbers are unsigned integers in network byte order number-1 = 1OCTET number-2 = 2OCTET number-4 = 4OCTET number-8 = 8OCTET

I've written this up as a draft on the ZeroMQ RFC website. There is a zproto model for the protocol if you want to experiment.

Conclusions

While it's become easy to build RPC protocols using tools like protobufs and zproto, many of these protocols tend to be fragile and expensive to evolve across a distributed system. That becomes a problem at scale. The root cause is the lack of abstraction, so that every change to business semantics causes a break in the protocol, and a new version.

One proven way to address this is to use a RESTful resource approach, instead of RPC. This provides equivalent semantics, yet is designed to handle change naturally, and cheaply.

In this article I've presented a new protocol called XRAP, which provides a RESTful approach over ZeroMQ. XRAP is still a draft specification and under development. If you want to use it, contact me and we'll work together on this.

by pieterh

The Economics of Contracts

In this section I'll explain why we need contracts and standards in software.

Definitions first: a "software contract" is an API, or a protocol, or some other formalized interaction that allows two otherwise unconnected components of a system to work together. I've written a lot about contracts, because they are so vital to making successful distributed software. Contracts let us build overall systems in parallel, in many places and at many times, and with many teams. However, contracts are a vital concept in all types of software.

All libraries and products implement contracts, explicit or implicit. Contracts may be documented (using formal prose like RFCs, ideally), or embedded in code (less than ideal). A contract may have multiple independent and competing implementations, and the more implementations, the better.

In ZeroMQ we have wire-protocol contracts like the ZMTP protocol and CurveZMQ security protocol, and the semantics of various socket patterns. We also have contracts for the API, specified as man pages.

When a contract is written and published by third parties, it becomes a standard.

The economics of standards are simple: they enable competition between suppliers, and this drives down the cost of products. Suppliers have an incentive to keep contracts undocumented, over-complex, hard to implement, or even proprietary. Customers have an incentive to turn contracts into standards, to simplify them, make them easier to implement, and bring the cost of knowledge to zero.

In markets where standards are properly regulated to remain free and open, the cost of goods does fall as it should. In markets where prices are not falling, the cause is generally lack of competition, enabled by closed, complex, or undocumented contracts. Free and open standards promote a competitive free market, whereas captive standards or captive contracts raise costs.

This is how it works in large-scale real world economies, and it is also how it works in software projects, which emulate economies (often they emulate an insane centrally planned economy, though that is another story).

In any software project, even an open source one, the incentives of competition and capture are present, even if Euros and Dollars aren't literally on the table. We are all Comcast, at times. When you're choosing a software stack, your primary questions should be, "does this stack implement public contracts?" and then, "who else implements these same contracts and are they standards?"

A contract needs several properties to work as a real public standard. Above all, it must be published, and it must be freely remixable (to prevent capture). The more we tend to formal processes and organizational ownership, the more it costs to make a contract, and the more vulnerable it is to capture.

So ideal public contracts are owned collectively, and published centrally only for convenience. Every project and product must be free to define its own contracts, either new ones, or remixes of existing works. These contracts should move, over time, towards stability and acceptance as standards by the market (if they are successful), or else should die and disappear.

The Problem With Software Versions

In this section I'll try to nullify the dominant theory of software versions.

The worst times in ZeroMQ's history were when we changed the library version to signal major and incompatible change in its contracts. Version 2.0 did not talk to version 3.0, which did not talk to (the then) 4.0. They had incompatible protocols and different APIs. These were in effect different products, each trying to define its own private (and poorly documented) contracts.

To fix this problem, we introduced the rule that new stable releases of the software must (a) talk to old stable releases and (b) they must support existing apps, without changes. We more or less succeeded with that, so ZeroMQ versions 3.2 and 4.0 work nicely with 2.2 and 2.1, for example. We slipped only once or twice, by shipping a stable release with new contracts that turned out to have flaws which needed fixing.

Initially, this rule made it harder to make stable releases, as the size and reach of the ZeroMQ contracts grew. Yet by moving to a more organic process we found that our master branch was stable most of the time. Banning breaking changes had the effect of reducing the impact of errors. Fewer large changes means fewer bugs in released code. Today, we make stable releases of core ZeroMQ projects like CZMQ by tagging master.

If you look at libzmq (the original "ZeroMQ"), you'll see a single header file that defines the API contract. This is the most significant to most users, since it is what they write code against. If you read that file, you'll see it is rather chaotic and unstructured. It ends in an "undocumented" section marked "use at your own risk".

There is no ZeroMQ "team". Change to the code base (in all ZeroMQ projects that use our C4.1 process) is driven by a distributed process that collects real world problems from users, prioritizes them, estimates their importance, and then rapidly tests solutions against them until an acceptable solution emerges. The emphasis is on low latency and accuracy.

Since change is driven by use, as a product gets wider use, change also hits a wider front. We're not talking deep, dramatic shifts (though some threads may sound like that). Rather, we're talking about tiny evolutionary changes happening in many areas.

Change to a system creates shifts in knowledge and costs, resulting in "vertical tipping points". This is when a series of answers to a whole set of problems can suddenly be recast as a new model, with new abstractions that hide much or all of the complexity that has grown up.

For example in ZeroMQ, to make a workable client-server pattern, you take DEALER and ROUTER sockets, configure them in various ways, add heartbeating and sessions and timeouts, and perhaps discovery. We start to see that this pattern happens over and over again. And suddenly we hit a tipping point where it makes sense to design "server" and "client" classes.

We accumulate change horizontally, and then we refactor vertically. This is a classic and effective pattern for building software systems. The outcome is, however, that the library is never really stable, and never finished as long as people are using it in new applications.

This seems nightmarish, and yet it is not. The trick is to understand two things. First, changes are naturally localized. Even if zmq.h looks like one long contract, it really covers several different areas. And these areas each have their natural cycle through experimentation to stability.

Here is the key understanding: a real product has multiple contracts, that change independently. Take CZMQ, which has a better-designed API than libzmq. This library has about 40 classes, each with its own contract.

When we claim "version X is stable", we are claiming either "all contracts in version X are stable". In the reality of popular products, the only way to guarantee this is to lock all contracts into synchronization, by force.

From our experience, we want a decentralized, lock-free development process. That has proven to be the fastest, most accurate, and most enjoyable way to work. This process is incompatible with coercive synchronization of all the contracts in a widely used product. Thus, it is incompatible with software versions.

Since every distributor insists on stable versions of code, there are of course answers to this puzzle. One is to reduce the size of products so they do implement only one contract, which can be coerced into stability. We get an explosion of libraries (we'd need 40 to cover CZMQ), and we start to depend on power and force for our work (something that's both distasteful and from experience, unproductive). "You cannot make this patch because we're making a stable release" is a failure, IMHO.

Before I explain another answer that does not depend on coercion or fragmentation, let's recap what we really want to achieve (apart from making packagers and Enterprise customers happy):

• We want a real-time pipeline connecting our software economy, so problems flow in one direction, and answers in the other. Annual releases just don't cut it. We want real-time, continuous delivery of small, incremental patches.

• We want guaranteed, testable backwards compatibility with stable contracts. I'd love to be testing libzmq master against the test cases from older stable releases.

• We want freedom to refactor vertically at any time and without upfront coordination. This means creating and delivering (so, testing) new contracts at any time.

• We want a coercion-free process where power is used only to regulate contracts and cheats. This is essential to get scale. Upfront coordination is the mutex of software development.

OK, we probably want a lot more than this, however it's a good list to work with.

Contract Life-Cycles

The first step to enlightenment is to see that contracts have a life-cycle. This applies to all contracts, both in software and in the real world. The contract life-cycle is not an invention, it's a feature of real world economics, and a useful one for software engineering.

Public contracts most definitely change over time. Changes are not always backwards compatible. I cannot plug a horse into the house electricity mains. My 3/4" flat-headed screw does not fit the 1/2" hole in the wall. If I plug a 110V US toaster into my 220V kitchen in Brussels, the toaster dies rapidly and dramatically.

Do I care what version the toaster is? No. Do I care what contracts it implements? Most definitely so. These are written in bold on the packaging, and those claims are wrong, I can sue the distributor and manufacturer for fraud.

To be of general use, a standards must be stable. Of course there can be variation. I can plug a 220V toaster into a 210V, 220V, 230V, or even a 240V circuit without risk. Standards can have variations, as long as these are compatible with stable versions. This goes one way: we do not usually update shipped products over time, as standards change.

How do we know when a contract is stable enough to qualify as a "standard" then? Well, for public contracts (the ones we care about and strive to make), stability is a negotiated peace between users (clients) and suppliers. Change can lower costs, yet can also create costs as the contract affects more points. So eventually the cost/benefit trade-offs drive both sides to agree that change now stops.

If you look at the CZMQ classes, you'll see some are marked "deprecated". If you look at the ZeroMQ RFCs, you'll see more elaborate tagging.

The life-cycle of a contract can be obfuscated, by not documenting it. Let's assume our contracts are written down, not embedded in the software somewhere.

A new contract usually has no implementations. Until someone signs a contract (demonstrates it in code), it's just speculation. I call these "raw" contracts. Since no-one is using a raw contract, and it's usually incomplete and immature, it changes a lot, without hurting anyone.

When you can demonstrate a contract in code, it gets some status. I call these "draft" contracts. A draft contract will change a lot, often in tandem with its implementation. This is fine, and hurts no-one.

When two or more parties implement a contract, and especially when there are third parties depending on the contract, however, things get serious. I call these "stable" contracts. Change to stable contracts needs upfront agreement. You cannot rely on this, except in young stable contracts with few users.

Eventually a stable contract gets replaced by new draft contracts. I call these "legacy" contracts. Changing legacy contracts is just an all-round bad idea. Finally, when new stable contracts emerge, the old ones become "retired".

Sometimes, new stable versions of a contract can talk to older legacy versions. Alternatively, software may implement both older and new versions of a contract.

There are three key points here:

• A realistic software system will implement many contracts, rather than just a single one. Whether or not these contracts are documented, a successful software system will, over time, add more and more contracts to the set it implements.
• Each of these contracts goes through a draft-stable-legacy-retired life-cycle, which can be less or more formally defined.
• While the life cycles of the different contracts may drive each other, they are naturally independent. To force all contracts in a system to the same state requires significant coercion and cost.

Hopefully you see why trying to force an entire product into one state does not scale well.

Defining Interoperability

Interoperability is the ability of products X and Y to work together. My WiFi router has a list of USB 3G modems it works with. This seems good, right? Actually, no, it's a rubbish approach. It's like saying, "this toaster is compatible with versions 3.1 and later of my home".

To define interoperability as a relationship between products is to assume that the set of products is limited and testable, and exists only within a given time and space. Yet real systems stretch over space and time. When I build a house, I equip it slowly, over years. The process of improving it never stops. The only way I can get the right toasters at the right price is to search widely and slowly (i.e. over space and time).

It's far better to say, "my house has a 16A 220V-50Hz electric circuit" and "this toaster draws 4A at 220V-50Hz". That is, to define an electricity contract, and then define interoperability as "respects this contract". The supply of toasters that implement that contract is huge. The supply of toasters that implement House V3.1 is small, if not tiny to the point of "only a Russian billionaire would consider this."

A list of "compatible devices" is a red flag that tells us, "this area is not standardized and thus you can expect to pay many times what you should."

Ditto for software. When you install a software product you are making a long term investment. The accounting of that decision depends on the contracts the software respects. It is interesting to know what's inside the software, for sure. Is it fast, does it crash often, can it handle many users? However it's far more interesting to know what standards it implements, and how well.

Software systems that do not respect and promote public contracts create a sense of unease and disquiet. It does not really matter whether they're open source or not. Technically and economically, the alternative to an architecture based on contracts is one based on coercion.

I believe that in removing the concept of software versions, and focusing instead on the contracts that a system implements, we can get a more accurate description of what we are getting, as consumers. Further, it's a description that is mechanically testable, and could even be held up as a legally binding statement by software producers.

The Software Bill of Materials (SBOM)

Let me introduce the Software Bill of Materials, or SBOM. The SBOM is a testable statement of the contracts (e.g. RFCs) that a software product implements. The SBOM is a claim of backwards and forwards interoperability with other products, based those product's own SBOMs, rather than product version numbers.

To implement an SBOM, a software project has a section in its README that says, for instance:

## Software Bill of Materials This software implements the following public contracts: * project/contract/name/state * ...

If an implementation supports several versions of a contract, it should list them all. Thus libzmq, for example, implements ZeroMQ RFCs 13, 15, and 23 (three versions of ZMTP).

This may seem optimistic, yet the goal should be that the SBOM defines *fully* the behavior of the software system as far as it affects other software. As in any product, there will be design trade-offs, e.g. performance vs. resource consumption. If these are relevant to the outside world, they should be defined as public contracts.

The SBOM depends on a very cheap, fast way to develop RFCs through their life-cycle. I developed and tested this starting with the Digital Standards Organization (Digistan) in 2005, and then with ZeroMQ RFCs up to the present day.

An RFC is simply a contract that's published on a public platform. Under the Digistan rules, such documents use the GPLv3 license, so can be freely forked and improved. This solves the problem of capture ("No, I won't improve my RFC to make it easier for you to implement").

We've additionally been using man pages as contracts for APIs, and this works up to a point. Both libzmq and CZMQ as well as other projects do this. It is however too easy to change the contract together with the software, and break existing applications.

Long Live Version Numbers!

In ZeroMQ we still use version numbers for two main reasons:

• We don't have fully testable or tested contracts, and so code on master still has faults that emerge over time. We reduce the risk by forking master to a separate repository and slowly fixing this, without taking new functionality from master. This emerges as major.minor.patch.

• Version numbers are powerful tools for marketing, and especially for replacing old rubbish with new, exciting, rubbish.

• If (if!) we remove support for retired contracts, the version number would make this clear. "Version X no longer supports the frongle interface."

We can still provide version numbers by tracking the SBOM. I'd simply number the SBOM changes from 1 upwards, unless we find we need more detail. Perhaps the repository commit number within the SBOM number.

Worked Example

Since I like to eat my own cooking, I'm going to start using the SBOM approach for CZMQ, and if it works there, spread it to other projects. CZMQ implements a variety of ZeroMQ RFCs, and its own API contracts.

From experience we know that a single CZMQ class can contain multiple contracts, each with its own state. E.g. zsys contains draft methods, stable methods, and retired methods. I'll to tag these API contracts as "draft", "stable", "legacy", or "retired". Further, when we change contracts (draft and stable, probably) we'll number those changes.

So the SBOM for CZMQ might contain entries like this:

* czmq/zsys/udp.1/stable * czmq/zsys/presets.1/stable * czmq/zsys/interrupt.1/retired * czmq/zsys/interrupt.2/stable * zeromq/RFC/4/stable

Where each entry in the SBOM matches a test case living either in CZMQ, or (ideally elsewhere). Incidentally, moving the tests out of the project solves the common question with contract driven development: "what if someone breaks a contract and then modifies the test case to hide that break?"

Conclusions

As I've explained, a living software product will usually deliver a mix of draft, stable, legacy, and retired contracts. Rather than try to assign a "major.minor.patch" version number to the overall mix, to indicate stability and maturity, it is more sensible to list the contracts explicitly in the living code (on git master).

Implementations of draft contracts carry no promise of interoperability. It is fair to assume that the market for draft contracts is small, and the cost of change is low. Implementations of stable contracts are guaranteed to work with all other implementations of the same stable contract. A stable contract should be testable, and the claim of a vendor to implement it could be legally binding.

By listing all contracts that a software project implements as a "Software Bill of Materials", or SBOM, we make it possible to test every single master commit against these contracts. While today we tend to run CI tests within a single project, it would be better to have external test suites, one per contract. Since people will demand version numbers of some sort, we can version the SBOM, and commits within that. So version 12.122 of a product will be SBOM version 12, commit #122.

As a worked example, I will experiment with CZMQ, one of the projects I'm mainly responsible for.

by pieterh

Don't Panic

We don't remove functionality that people are using. The existing multipart functionality is defined by ZeroMQ RFCs 23 (ZMTP) and 28 (the request-reply pattern). There is no plan to stop supporting these RFCs, nor to remove the multipart API contracts. Our strategy for shifting ZeroMQ's functionality is to develop new contracts over time, and when these are stable and proven, deprecate old ones. This happens over many years, with new code existing in parallel to old code.

So the focus of this experiment are mainly in new code: new language bindings, new protocols, and new stacks. The overall goal, as I'll explain, is to make things simpler, and easier to understand.

What's the Problem?

Multipart messages aim to solve a number of problems, and in turn create a number of problems. At the core of our shift is the understanding that the costs now appear to outweigh the benefits.

First, let's collect the problems that multipart solves:

• It allows zero-copy for high-volume pub-sub data, where the application sends a routing key, followed by a body. With one part, the body must be copied into a buffer. With two parts, the body can be sent directly from an existing buffer (thus, no copying).

• It provides a mechanism for multi-hop request-reply, where routing identifiers are pushed to an "envelope", and popped on the return path.

• It provides applications with a simple serialization mechanism, either to avoid using additional libraries, or so that brokers can add and remove information to messages they are forwarding.

Now, let's list the main problems that we've seen with the multipart model in ZeroMQ:

• It is confusing, as a "message" means both a part, and a list of parts. We've had to add new terms like "frame" to compensate.

• The multipart request-reply envelope is complex to understand, and yet any realistic application has to use it. This creates a real learning cost.

• ZeroMQ servers exploit the request-reply envelope to do server-client routing (using ROUTER sockets). This always confuses new users.

• ZeroMQ implementations cannot realistically make sockets threadsafe, as they must serialize multiple send and recv calls to a single thread. Thus, multipart messages have stopped us making threadsafe sockets.

• Multipart messages are counter-intuitive; the "send" method does not actually send anything; it queues the message until the final part is sent. New users often try to use multipart to send large files. This does not work.

• The two main sockets that exploit the multipart mechanism (REQ and REP) cannot recover from failures, and rarely used by real applications.

• Supporting multipart messages has raised complexity and performance in all bindings and tools. It is significantly harder and more work to support a list of messages than to support a single frame.

• Multipart data is incompatible with other layers that treat data as a stream of bytes or blobs. For example, the Go I/O interface expects a transport to read and write blobs.

The Identity Problem

The ZMQ_IDENTITY socket setting is one of the more confused, and in my view dangerous, semantics in ZeroMQ. It allows a DEALER/REQ/REP peer to tell a ROUTER socket what value to use for its internal routing ID. "User-defined identities" arrive as a magical first frame that can be up to 255 bytes long.

It was a mistake to mix node identities with routing information. These are two separate layers (contracts) and mixing them creates a confused hybrid contract. What happens when two clients connect with the same identity? At what stage is the identity checked? How do we correlate such identities with application-level identities? How do we proxying traffic?

It has been clear for some time that:

• We should stop using the term "identity" for server-side routing information (which is essentially the handle of a pipe).
• We should delegate identity management (authentication and access control) to higher layers, where different decisions can be made depending on the case.
• We should in fact hide the routing information in simple request-reply servers.

The Multihop Fallacy

The multipart request-reply envelope solves what appears to be a fallacy, a fake problem. That is: we want to be able to carry a single message across many hops, from originator to recipient, and then carry a reply back along the same path.

This scenario does not seem to happen in practice. There is a very good reason for this, stemming from the essential need of a distributed system to work in layers, and hide the internals of every layer from other layers.

Put this another way, the extended request-reply scenario assumes, or tries to enforce, a single contract that spans across many layers. And this is a failing pattern. Success needs specific contracts between each layer, and the flexibility to change these at need, without affecting other layers.

Take an example, the Majordomo protocol that takes client work requests, and delivers them to workers. This is practically the simplest kind of load balancing we can do.

In Majordomo we have two contracts. One defines the interaction between clients and the broker. The second defines the interaction between workers and the broker. These two layers are separate, and it is profitable to keep them so. It lets us evolve the worker contract (adding heartbeating, timeouts, recovery) and the client contract (e.g. adding scheduling and quotas) separately. It lets us experiment with multiple worker-broker contracts without affecting client applications.

Clearly we do carry payloads across both layers. However there is an intelligent piece of code switching between the two layers, and translating messages from one contract to another.

So it looks like multihop request-reply can be falsified as a theory, and thus the use case for request-reply envelopes can also be falsified.

A Better Framing Approach

In the last few years the ZeroMQ community has developed its own tool for framing, namely zproto. This generates a fast binary codec for a protocol, in arbitrary languages (Go, Clojure, Java, C, C# at this stage).

The advantage of zproto over framing is that you get a properly documented protocol with robust error checking at client and server side. For example, ZeroMQ RFC 7 (the Majordomo protocol), uses multipart framing for its protocol, which makes it expensive to implement the protocol, or evolve and improve it.

A Better Client-Server Pattern

While the standard pattern for client-server cases is DEALER-ROUTER, these sockets are clumsy and often confusing to newcomers. Their names are unclear. The ROUTER socket uses multipart messages for routing, which is semantically and technically confusing.

There is a much simpler, more obvious client-server pattern:

• CLIENT and SERVER socket types that are separate from the request-reply pattern.
• CLIENT can implement, immediately, necessary logic for heartbeating, and recovery from server restarts.
• SERVER can implement client sessions, recovery from network failures, and disconnection of idle clients.

There are several possibilities for a SERVER socket design:

• Just like ROUTER today, except the routing information is not provided as a multipart frame. Instead, it is a message property (it cannot be a socket property, if sockets are to be threadsafe).

• A higher level socket that automatically manages client sessions. This means, identifying new connections and re-connections (when the same client reconnects on a different network socket), responding to heartbeats, and expiring idle clients. The API here would add a "session" object, corresponding to a single client session.

Similarly, CLIENT could be very similar to DEALER, or something more sophisticated that deals with the requirements we always have in client-server scenarios.

Heartbeating and Sessions

Looking at several years of building client-server stacks with ZeroMQ, a consistent pattern emerges for reliable session management. We build this today using DEALER-ROUTER and zproto for protocol commands. It could profitably be built into the CLIENT-SERVER sockets, and the ZMTP protocol:

• The server must detect new client sessions. Ideally, a session identifies a CLIENT socket instance, with the same lifecycle. Today we use the routing ID, which means we cannot recover a client session after a network break.
• The client must send heartbeats, when idle. The heartbeat interval is typically one or several seconds, depending on client-side power considerations. A heartbeat command ("PING") carries no data.
• The server must repond to heartbeats, simply by sending a heartbeat reply ("PING-OK"), again carrying no data.
• Clients must send at most a few PINGs, if they are not getting PING-OK replies. Otherwise they will fill up their outgoing pipes with a thousand PING commands.
• Servers must expire clients that have sent nothing within a certain timeout (typically 10 to 30 seconds, depending on the network).

For session management, the client must send a unique ID that can be used as session ID. Today we use the routing ID for this. Some applications use the ZMQ_IDENTITY setting for this. Neither are ideal. The best solution is that a CLIENT socket automatically generate its own unique session ID, which cannot be over-ridden or used at any layer except this one.

The client sends its session ID when connecting to the server. The server holds known client sessions in a table. If the client connects with an existing session ID, the server attaches that new connection to the existing session. If not, it creates a new session. Sessions expire when they have no activity for 10-30 seconds (per heartbeating).

This session reconnection is invisible to the application. It sees, at worst, a brief lull in traffic, which then resumes.

Separately, the SERVER socket provides a routing ID for each message. This is an internal short integer that identifies the session. The SERVER does NOT expose the session ID. If applications want their own client identifiers, they build these themselves.

The SERVER's routing ID is a socket (and perhaps also message) property that the caller can get (after receiving a message) and set (before sending a message). It is trivial to write simple servers, which can just be a recv/send loop, like today's REP sockets. Unlike REP sockets, a SERVER socket has no state machine and can trivially recover from errors.

Why Threadsafe Sockets?

One of the unexpected costs of multipart messages was that sockets could not be threadsafe. It's obvious why: a socket has to serialize an unknown number of "send" commands from a single caller thread. If receiving and sending is an atomic operation, the socket can be threadsafe, else not.

Threadsafe sockets resolve a major, long-standing problem, of using ZeroMQ from languages that spawn many threads. This is such a common use case: the UI spawns a thread which wants to send a message. Today it must create a new socket, connect it, send its message, and destroy the socket. This is not fast.

Ideally, the new thread would grab the socket from shared memory, send to it, and die happily. Note that this is for a one-way message flow. If threads need a reply from their socket, they must use (today) a DEALER-ROUTER model, and (in our new model) a CLIENT-SERVER model.

There have been abortive attempts to make ZeroMQ sockets threadsafe. They did not, however, make the codebase simpler. Removing multipart messages (even if this is a compile time option), makes threadsafe sockets possible.

CZMQ Changes

As a worked example, let's see how this could change the CZMQ API:

• Deprecation of the zmsg and zframe classes, and promotion of zchunk (or byte * + size_t) as the basic message type.
• New zclient and zserver classes, which implement CLIENT and SERVER sockets, as described above. Initially these could be emulated over DEALER and ROUTER, later over real CLIENT and SERVER.
• Deprecation of all functionality that supports identities and multipart.
• A new "reply" semantic, for SERVER sockets, which replies to a previously received message. This can be implicit (reply to socket), if the socket is thread local, or explicit (reply to message or to message routing id) if the socket is shared between multiple threads.
• Change of actors and pipes to use CLIENT-SERVER instead of PAIR-PAIR. This lets actors reply to caller requests, where each caller would get its own client socket to speak to the actor.

How This Would Happen

Clearly some of the goals we have — simplifying the internals of ZeroMQ stacks — are not compatible with existing applications.

We cannot break existing application code, and so future ZeroMQ stacks will need to continue supporting multipart data. Some options for doing this are:

• To build a new socket types (client, server) that are threadsafe and that work on single parts only. This would leave the existing codebases largely untouched.
• To fork the codebase. Not perhaps the nicest option.
• To support both models, and allow newer applications to disable multipart, to get the eventual advantages.

Conclusions

It's unclear we can make these shifts successfully. However, if we can, then here are some potential outcomes:

• The REQ-REP-DEALER-ROUTER socket types will disappear (be deprecated).
• The ZMQ_IDENTITY concept will disappear.
• New CLIENT and SERVER sockets will replace REQ-REP-DEALER-ROUTER.
• Messages will be single opaque blobs of data.
• All other names for messages (frames and parts) will be deprecated.
• All "recv more" and "send more" semantics will be deprecated.
• CLIENTs and SERVERs may eventually do heartbeating and session management automatically.
• All socket types that no longer accept multipart messages can probably be made thread safe.

Comments and discussion welcome, please use zeromq-dev for this.

by pieterh

Pause, Rewind, Play

In 1999 I wrote an article for Dr. Dobb's Journal on "Template Based Code Generation". The PDF whitepaper says, "Template-based code generation is a powerful way to reduce the cost and effort of writing code. It's a technique that's underused by most developers, because there is little literature on this subject, and there are few decent code generation tools."

In 1999 I'd already been writing and using template based code generators for 15 years. Jonathan Schultz and I worked for about ten years to refine these techniques, ending in 2005 or so with GSL v4, which we used heavily to build the OpenAMQ messaging system (the first, and the fastest, AMQP server and client).

OpenAMQ taught us a few things, not all happy lessons. One lesson was especially harsh: meta hurts. In 2005 we abandoned our entire technology stack, code generators and all, believing the world was not ready for it, and never would be. When we started building ZeroMQ, it was in C++ (ugh) and by hand, with zero code generation (oh god why?!). We chucked GSL into hibernation and I stopped coding for a while, it was too painful to step back so far into the past.

It seemed, for years, that no-one understood the power and beauty of code generation. It was too different from the industry standard perception of how we make and value code. Code was, and mostly still is, seen as an asset rather than a liability. The truth is that in software design, as in any domain, all that matters are our models. Good models are simple, abstract, accurate, and tethered to reality. Bad models are fuzzy, complex, and based on falsehoods.

Here is a bad model: "Freedom is about having more flavors of ice-cream." Here is a good model: "Freedom is about doing interesting things with other people." The difference should be immediately obvious. If you're confused here, you need to review your priorities in life. A bad model confuses and traps us. A good model empowers us.

Code generation isn't about replacing the hapless coder with a robot. It's about teaching that coder to build models instead of code, to think in terms of simple, clean, accurate abstractions. That is, to learn to develop simple domain specific languages, with the tools to design and test and implement them cheaply and rapidly.

I used to call this "Model Oriented Programming", before shutting down our code generation machines and focusing instead on social architecture, aka "messing with people on a large scale, for their own benefit of course, cause otherwise that'd be Machiavellian and anti-social and even oh noes very profitable."

Well, code generation seems to be in fashion. Yay! Finally. GSL — the original and the best — found a new home on GitHub.

I'm happy to say that a slice of the ZeroMQ community has embraced model oriented code generation with a passion. About seven months ago I announced zproto, a tool for generating ZeroMQ servers, and clients.

While I like zproto and use it every day, the more interesting tool, because it solves far more horrid problems, is zproject.

Both zproto and zproject are reboots of old tools from the OpenAMQ toolkit (ASL and Boom, if you care, which you don't, as we both know :)

Since I've already talked about zproto, let me explain zproject.

The Problem with Makefiles

Once, long ago, a smart young person invented Make. He (for even in those days the technical edge of our field was very gender biased) had a chance to save the world. Or at least make it a much better place. Instead, he declared (according to the friend of a friend), "I realize the Makefile syntax is horrid, yet I already have seven users, so changing it is impossible!"

The rest is history. Every system has its own way of compiling and linking code. It's such fun to write a new project makefile specification that companies make one every year or two. Microsoft — for once I'm not joking when I mention their name — has different formats for Visual Studio 2008, 2010, 2012, and 2013.

Worse, the abstraction is largely useless. Sure, compiling and linking is one thing we want to do. However we also want to generate documentation. We want to build test frameworks, and run them each time the code changes. We want to target arbitrary platforms, today and in the future. We want to wrap our precious C APIs in other languages. We want to do endless things with our project, and above all, escape the limits and conditions set by our tool vendors.

This isn't hard to do. It just demands a higher level of abstraction.

Here is another good model:

<project name = "zyre" description = "an open-source framework for proximity-based P2P apps" script = "zproject.gsl" > <include filename = "license.xml" /> <version major = "1" minor = "1" patch = "0" /> <use project = "czmq" /> <class name = "zyre" /> <class name = "zyre_event" /> <class name = "zre_msg" /> <class name = "zyre_peer" private = "1" /> <class name = "zyre_group" private = "1" /> <class name = "zyre_node" private = "1" /> <model name = "zre_msg" /> </project>

And here is a bad model for the same knowledge:

AC_PREREQ(2.61) AC_INIT([zyre],[m4_esyscmd([./version.sh zyre])],[zeromq-dev@lists.zeromq.org]) AC_CONFIG_AUX_DIR(config) AC_CONFIG_MACRO_DIR(config) AC_CONFIG_HEADERS([src/platform.h]) AM_INIT_AUTOMAKE([subdir-objects tar-ustar foreign]) m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) PV_MAJOR=`echo $PACKAGE_VERSION | cut -d . -f 1` PV_MINOR=`echo $PACKAGE_VERSION | cut -d . -f 2` PV_PATCH=`echo $PACKAGE_VERSION | cut -d . -f 3` AC_DEFINE_UNQUOTED([PACKAGE_VERSION_MAJOR],[$PV_MAJOR], [ZYRE major version]) AC_DEFINE_UNQUOTED([PACKAGE_VERSION_MINOR],[$PV_MINOR], [ZYRE minor version]) AC_DEFINE_UNQUOTED([PACKAGE_VERSION_PATCH],[$PV_PATCH], [ZYRE patchlevel]) AC_SUBST(PACKAGE_VERSION) LTVER="2:0:1" AC_SUBST(LTVER) ZYRE_ORIG_CFLAGS="${CFLAGS:-none}" AC_PROG_CC AC_PROG_CC_C99 AM_PROG_CC_C_O AC_LIBTOOL_WIN32_DLL AC_PROG_LIBTOOL AC_PROG_SED AC_PROG_AWK PKG_PROG_PKG_CONFIG ...

Which goes on forever, or 262 lines, whichever comes first. Learning autoconf is about as much fun as falling off your kick bike and smashing your face on cheap concrete pavement. Actually, having done both, I prefer the latter. Not that the alternatives are better. CMake, you say? Yeah, CMake is like smashing your face on high quality pavement. You can admire the stonework while the blood pours down your face.

Here's the equivalent CMakeLists.txt (again I truncate to reduce your empathic pain):

cmake_minimum_required(VERSION 2.8) project(zyre) enable_language(C) enable_testing() set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}) set(BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}) foreach(which MAJOR MINOR PATCH) file(STRINGS "${SOURCE_DIR}/include/zyre.h" ZYRE_blah blah string(REGEX MATCH "#define ZYRE_VERSION_${which} ([0-9_]+) blah if (NOT ZYRE_REGEX_MATCH) message(FATAL_ERROR "failed to parse ZYRE_VERSION_ blah blah endif() set(ZYRE_${which}_VERSION ${CMAKE_MATCH_1}) endforeach(which) set(ZYRE_VERSION ${ZYRE_MAJOR_VERSION}.${ZYRE_MINOR_VERSION}. blah include(CheckIncludeFile) CHECK_INCLUDE_FILE("linux/wireless.h" HAVE_LINUX_WIRELESS_H) CHECK_INCLUDE_FILE("net/if_media.h" HAVE_NET_IF_MEDIA_H) include(CheckFunctionExists) CHECK_FUNCTION_EXISTS("getifaddrs" HAVE_GETIFADDRS) CHECK_FUNCTION_EXISTS("freeifaddrs" HAVE_FREEIFADDRS) include(CheckIncludeFiles) check_include_files("sys/socket.h;net/if.h" HAVE_NET_IF_H) if (NOT HAVE_NET_IF_H) CHECK_INCLUDE_FILE("net/if.h" HAVE_NET_IF_H) endif() file(WRITE ${BINARY_DIR}/platform.h.in " #cmakedefine HAVE_LINUX_WIRELESS_H #cmakedefine HAVE_NET_IF_H #cmakedefine HAVE_NET_IF_MEDIA_H #cmakedefine HAVE_GETIFADDRS #cmakedefine HAVE_FREEIFADDRS ") configure_file(${BINARY_DIR}/platform.h.in ${BINARY_DIR}/platform.h)

Sorry, I removed some blank lines that were loitering around to pretend that is a "language" instead of a slowly rotting heap of fish heads and unwanted celery sticks. Did I see some bits of Perl 3.x in there? I'm sure some people write really pretty CMake scripts. Meh.

I've expressed my abundant affection for autoconf and CMake, which leaves about ten other random make systems to mock. Let me summarize my experience: all make systems are rubbish, and some are worse than others.

There's a book I want to write, on how to use C properly. By "properly" I mean as we do in CZMQ and Zyre and our other C projects. Actors, classes, neat abstractions for concurrency, and APIs, and useful packages of functionality, that are obvious and trouble-free and accurate, and properly anchored to reality. Perhaps this will be "Code Connected Volume 2" or perhaps a book on "Scalable C".

Anyhow, before zproject, I would have to spend a full chapter (don't mock it, one chapter is equal to fifteen Imperial weekends) simply explaining how to write a Makefile, or autoconf voodoo. What use are elegantly crafted examples if your readers cannot build and run them?

With zproject, we are starting to develop that abstraction and test it with real cases. zproject takes a project model, and turns it into makefiles and project files for MSVC (four flavors), Android, MinGW, QT, autoconf, Travis CI, and CMake. It generates a full project skeleton (in C) and maintains this for you as you add and remove classes in your project. It calls code generators, and builds man pages. It even builds bindings for your C libraries, in QML, and soon Ruby and other languages.

And we started zproject less than a month ago, on 20 October 2014. Kevin Sapper, Joe Eli McIlvain and Phillip E. Mienk have done most of the heavy work. It started out as part of CZMQ, then grew an independent life.

Using zproject

To use zproject, fork and clone the GitHub repo, then read the README. You will need to build and install GSL v4. You don't need to learn GSL to use zproject, though if you want to contribute, you do.

zproject is a community project, like most ZeroMQ projects, built using the C4.1 process, and licensed under MPL v2. It solves the Makefile problem really well. It is unashamedly for C, and more pointedly, for that modern C dialect we call CLASS. CLASS is the Minecraft of C: fun, easy, playful, mind-opening, and social.

by pieterh

Quick Background to zproto

zproto is based on work I did for Chapter 7 of the ZeroMQ book, used in FileMQ, Zyre, and several other projects. It's a collection of code generation tools that take models and turn them into perfect code.

iMatix used to do code generation as our main business. We got… very good at it. There are lots of ways to generate code, and the most powerful and sophisticated code generator ever built by mankind lives on Github.com at imatix/gsl. It's an interpreter for programs that eat models (self-describing documents) and spew out text of any shape and form.

The only problem with sophisticated magic like GSL is that it quickly excludes other people. So in ZeroMQ I've been very careful to not do a lot of code generation, only opening that mysterious black box when there was real need.

The first case was in CZMQ, to generate the classes for ZeroMQ socket options. Then in CZMQ, to generate project files (for various build systems) from a single project.xml file. Yes, we still use XML models. It's actually a good use case for simple schema-free XML.

Then I started generating binary codecs for protocols, starting with FILEMQ. We used these codecs for a few different projects and they started to be quite solid. Something like a protobufs for ZeroMQ. You can see that the generated code looks as good as hand-written code. It's actually better: more consistent, with fewer errors.

Finally, I drew back on an even older iMatix speciality, which was state machines. My first free software tool was Libero, a great tool for designing code as state machines and producing lovely, solid engines in pretty much any language. Libero predates GSL, so isn't as flexible. However it uses a very elegant and simple state-event-action model.

The Libero model is especially good at designing server-side logic, where you want to capture the exact state machine for a client connection, from start to end. This happens to be one of the more solid problems in ZeroMQ architecture: how to build capable protocol servers. We do a lot of this, it turns out. You can do only so much with low-level patterns like pub-sub and push-pull. Quite soon you need to implement stateful services.

So this is what I made: a GSL code generator that takes a finite-state machine model inspired by Libero, and turns out a full working server. The current code generator produces C (that builds on CZMQ). In this article I'll explain briefly how this works, and how to use it.

The State Machine Model

State machines are a little unusual, conceptually. If you're not familiar with them it'll take a few days before they click. The Libero model is fairly simple and high level, meant to be designed and understood by humans:

• The machine exists in one of a number of named states. By convention the machine starts in the first state.

• In each state, the machine accepts one of a set of named events. Unhandled events are either ignored or cause the machine to die, depending on your taste.

• Given an event in a state, the machine executes a list of actions, which correspond to your code.

• After executing the actions the machine moves to the next state. An empty next state means "stay in same state".

• In the next state, the machine either continues with an internal event produced by the previous actions, or waits for an external event coming as a protocol command.

• Any action can raise an exception event that interrupts the flow through the action list and to the next state.

• A defaults state collects events that may occur in any state, together with their actions.

The zproto Server Model

The zproto_server_c.gsl code generator outputs a single .h file called an engine that does the hard work. If needed, it'll also generate you a skeleton .c file for your server, which you edit and build. It doesn't re-create that file again, though it will append new action stubs.

The server is a class that exposes an API like this (taken from the zeromq/zbroker zpipes_server, a good example):

// Create a new zpipes_server zpipes_server_t * zpipes_server_new (void); // Destroy the zpipes_server void zpipes_server_destroy (zpipes_server_t **self_p); // Load server configuration data void zpipes_server_configure (zpipes_server_t *self, const char *config_file); // Set one configuration path value void zpipes_server_setoption (zpipes_server_t *self, const char *path, const char *value); // Binds the server to an endpoint, formatted as printf string long zpipes_server_bind (zpipes_server_t *self, const char *format, ...); // Self test of this class void zpipes_server_test (bool verbose);

Rather than run the server as a main program, you write a main program that creates and works with server objects. These run as background services, accepting clients on a ZMQ ROUTER port. The bind method exposes that port to the outside world.

Here's the state machine for the ZPIPES server:

<class name = "zpipes_server" title = "ZPIPES server" proto = "zpipes_msg" script = "zproto_server_c" > This is a server implementation for the ZPIPES protocol <include filename = "license.xml" /> <!-- State machine for a client connection --> <state name = "start"> <event name = "INPUT" next = "reading"> <action name = "open pipe for input" /> <action name = "send" message = "INPUT OK" /> </event> <event name = "OUTPUT" next = "writing"> <action name = "open pipe for output" /> <action name = "send" message = "OUTPUT OK" /> </event> </state> This state allows two protocol commands, READ and CLOSE. Names of states, events, and actions are case insensitive. By convention we use uppercase for protocol events: <state name = "reading"> <event name = "READ" next = "expecting chunk"> <action name = "expect chunk on pipe" /> </event> <event name = "CLOSE"> <action name = "close pipe" /> <action name = "send" message = "CLOSE OK" /> <action name = "terminate" /> </event> </state> This internal state shows how we make a decision in the state machine. These are three internal events; they can happen come immediately from 'expect chunk on pipe', or come later from various places. The server can ask for a wakeup event (timeout expired). We can also get events from other clients' machines. <state name = "expecting chunk"> <event name = "have chunk" next = "reading"> <action name = "clear pending reads" /> <action name = "fetch chunk from pipe" /> <action name = "send" message = "READ OK" /> </event> <event name = "pipe terminated" next = "reading"> <action name = "clear pending reads" /> <action name = "send" message = "END OF PIPE" /> </event> <event name = "timeout expired" next = "reading"> <action name = "clear pending reads" /> <action name = "send" message = "TIMEOUT" /> </event> </state> <state name = "writing"> <event name = "WRITE" next = "writing"> <action name = "store chunk to pipe" /> <action name = "send" message = "WRITE OK" /> </event> <event name = "CLOSE"> <action name = "close pipe" /> <action name = "send" message = "CLOSE OK" /> <action name = "terminate" /> </event> </state> The defaults state handles an 'exception_event' (sends FAILED back to the client). It also specifies that any unrecognized protocol command (an '$other' event) will also cause the engine to send FAILED back to the client. <state name = "defaults"> <event name = "exception"> <action name = "send" message = "FAILED" /> <action name = "terminate" /> </event> <event name = "$other"> <action name = "send" message = "FAILED" /> </event> </state> We can define API methods to pass commands back from the application through to the engine. <!-- API methods <method name = "somename" return = "number"> One-line description of method here <argument name = "some value" type = "string">What is this argument?</argument> <argument name = "some value" type = "number">What is this argument?</argument> </method> --> </class>

The generated server manages clients automatically. There's various options to expire dead clients, and so on — read the zproto_server_c.gsl code to understand more. The state machine applies to *single client connection* from start to finish.

There are two predefined actions: "send" and "terminate". Other actions map to functions in your own code. The engine calls these functions as needed, passing the client context:

static void store_chunk_to_pipe (client_t *self) { // State machine guarantees we're in a valid state for writing assert (self->pipe); assert (self->writing); // Always store chunk on list, even to pass to pending reader zlist_append (self->pipe->queue, zpipes_msg_get_chunk (self->request)); if (self->pipe->reader) { send_event (self->pipe->reader, have_chunk_event); assert (zlist_size (self->pipe->queue) == 0); } }

There are a set of methods your actions can call:

// Set the next event, needed in at least one action in an internal // state; otherwise the state machine will wait for a message on the // router socket and treat that as the event. static void set_next_event (client_t *self, event_t event); // Raise an exception with 'event', halting any actions in progress. // Continues execution of actions defined for the exception event. static void raise_exception (client_t *self, event_t event); // Set wakeup alarm after 'delay' msecs. The next state should // handle the wakeup event. The alarm is cancelled on any other // event. static void set_wakeup_event (client_t *self, size_t delay, event_t event); // Execute 'event' on specified client. Use this to send events to // other clients. Cancels any wakeup alarm on that client. static void send_event (client_t *self, event_t event);

For More Information

Though the Libero documentation is quite old now, it's useful as a guide to what's possible with state machines. The Libero model added superstates, substates, and other useful ways to manage larger state machines.

The current working example of the zproto server generator is the zeromq/zbroker project, and specifically the zpipes_server class.

You can find GSL on Github and there's a old backgrounder for the so-called "model oriented programming" we used at iMatix.

If you have something to say, comment below or come discuss on zeromq-dev.

by pieterh

The REQ/REP load balancing example is using a small cluster with 4 nodes. A dummy service with ID=1000 is spawned on three nodes (1,2,3). Requests to this service ID are sent from Node 4.

The dummy service is just listening for a message and prints it out once it arrives. It also sends a response message back to the client, telling it which host instance the message was handled by.

First, load TIPC, assign node addresses and enable eth0 as the bearer interface. This is part of system configuration and is normally done by initscripts, but I'm including the steps for completeness.

Node1# modprobe tipc && tipc-config -a=1.1.1 -be=eth:eth0

Node2# modprobe tipc && tipc-config -a=1.1.2 -be=eth:eth0

Node3# modprobe tipc && tipc-config -a=1.1.3 -be=eth:eth0

Node4# modprobe tipc && tipc-config -a=1.1.4 -be=eth:eth0

In this example, the nodes are KVM instances with the eth0 devices connected to a network bridge on the host machine.

Starting the 0MQ service on Node 1-3:

Node1# ./srv -s tipc://{1000,0,10} server tipc://{1000,0,10} running

Node2# ./srv -s tipc://{1000,0,10} server tipc://{1000,0,10} running

Node3# ./srv -s tipc://{1000,0,10} server tipc://{1000,0,10} running

Before sending any traffic, let's check how it looks like on our client Node4. Using tipc-config -nt, we can dump
the whole, or parts of the name table.

Node4# tipc-config -nt=1000 Type Lower Upper Port Identity Publication Scope 1000 0 10 <1.1.3:2118189060> 2118189061 cluster <1.1.2:3767599108> 3767599109 cluster <1.1.1:1451425794> 1451425795 cluster

Looks like we're good to go, Node4 knows that service 1000 is hosted by three nodes. Start sending some requests from Node4 to service 1000. Any instance number in the published range (0-10) can be used.

Node4# ./cli -c {1000,4} Node4 sending message to: tipc://{1000,4} client sent message 'hello world from Node4', rc=25 Your request have been served by Node3 Node4# ./cli -c tipc://{1000,4} Node4 sending message to: tipc://{1000,4} client sent message 'hello world from Node4', rc=25 Your request have been served by Node2 Node4# ./cli -c tipc://{1000,4} Node4 sending message to: tipc://{1000,4} client sent message 'hello world from Node4', rc=25 Your request have been served by Node1 Node4# ./cli -c {1000,4} Node4 sending message to: tipc://{1000,4} client sent message 'hello world from Node4', rc=25 Your request have been served by Node3

Here we can see that each subsequent request are directed in a round robin manner between the instances running on Node1-3.

Client applications can define their own policies/algorithms how to spread the instance numbers of the connections. By increasing/reducing the width of the server publications (the lower/upper numbers) for some of the blades, we can increase/reduce the number of connections handled by these blades.

There are some rules that limits the TIPC name overlapping, these can be found in the TIPC programmers guide.

by pieterh

Though ZeroMQ is great in many ways, it isn't easy to plug in different transports. So when we see a new transport like this, it's a sign of world class software engineering. I asked Erik a few questions, to learn more about TIPC, and why Ericsson added this to ZeroMQ.

Pieter: First off, what's TIPC?

Erik: TIPC is a cluster IPC protocol developed by Ericsson, it was initially a proprietary closed source solution, but both the specification and source code was released on Sourceforge early 2000, and picked up by the Linux kernel from v2.6.16.

How does TIPC compare, for example, to TCP?

TIPC provides a socket based API, supporting SOCK_STREAM, SOCK_RDM and SOCK_SEQPACKET. The ZeroMQ TIPC transport currently supports connection-oriented stream transmission (SOCK_STREAM), so the behaviour and characteristics of this are similar to that of TCP.

Each node in a cluster establish logical links to its peers. These links are supervised actively with seamless failover to redundant links.

TIPC guarantees in-order, sequential delivery on the link-level, even for connectionless (SOCK_RDM) and multicast traffic, given that the serving application is capable of keeping up with the load (we have no flow control for connectionless traffic).

It comes with a distributed topology service that keeps track of all published services (bound sockets) in the cluster.

Finally, TIPC can run over Ethernet and/or Infiniband.

This sounds… extremely powerful. This means we can create clusters in just a few lines of code, using all the ZeroMQ patterns. I see you made a whole set of test cases for TIPC. Request-reply, pub-sub, the whole lot. Can you explain briefly how to use TIPC then?

Sure. To use the TIPC transport, the serving ZeroMQ application needs to bind to a TIPC name sequence, a 3-tuple {type, lower, upper}. The 'type' is an ID that uniquely identifies your service in the cluster. The 'lower' and 'upper' are a service partitioning range that can be used for basic load balancing purposes. For example:

zmq_bind (sb, "tipc://{5560,0,100}");

And then for the client application to be able to connect, it specifies an ID in the range we defined above, for example:

zmq_connect (sc, "tipc://{5560,30}");

This is a Linux-only transport, which is fine for most people. How do I install TIPC?

If you want to try it out, a Linux machine with kernel >= 3.8 with CONFIG_TIPC=m is required.

I'm going to have to play with this at some time and see how it handles failures.

I'd like to emphasize here that you won't see a performance boost by switching to TIPC. There is no support for GRO (Generic Receive offload) yet, and the kernel code is not as mature as that of TCP/IP. In my opinion this starts to get really useful in a cluster environment once we have the RDM/Multicast support in place.

Right, so the TIPC transport in ZeroMQ is about reliability, not performance.

Yes, failover and load-balancing. You can do these on top of ZeroMQ, but when we can delegate it to TIPC, the results are better and it's invisible to the programmer. I'll embed it in a simple example using two qemu nodes, and simulating a failure on the primary link. This have almost the same effect as if a switch/port/cable fails.

First, I need to assign node addresses, worth noting is that TIPC does not use per-interface addresses, but rather a node-global one.

Node1# tipc-config -a=1.1.1 [...] tipc: Started in network mode [...] tipc: Own node address <1.1.1>, network identity 4711

I do the same thing on the second node, with the address "1.1.2". Then, I tell TIPC to use Ethernet interfaces 'vlan100' and 'vlan200' (same thing done on the second node):

Node2# tipc-config -be=eth:vlan100 [...] tipc: Enabled bearer <eth:vlan100>, discovery domain <1.1.0>, priority 10 [...] tipc: Established link <1.1.2:vlan100-1.1.1:vlan100> on network plane A Node2# tipc-config -be=eth:vlan200 [...] tipc: Enabled bearer <eth:vlan200>, discovery domain <1.1.0>, priority 10 [...] tipc: Established link <1.1.2:vlan200-1.1.1:vlan200> on network plane B

Now we start a netperf session from Node1 to Node2:

Node1# netperf -H Node2 -t TIPC_STREAM -l 100 # /opt/netperf/netperf -H 192.168.1.78 -t TIPC_STREAM -l 100 TIPC STREAM TEST to <1.1.2:3310419987> [test running]

Here is how we check the link statistics on Node2 to see which link is currently the primary one:

Link <1.1.2:vlan100-1.1.1:vlan100> ACTIVE MTU:1500 Priority:10 Tolerance:1500 ms Window:50 packets RX packets:298078 fragments:0/0 bundles:0/0 [...] Link <1.1.2:vlan200-1.1.1:vlan200> ACTIVE MTU:1500 Priority:10 Tolerance:1500 ms Window:50 packets RX packets:0 fragments:0/0 bundles:0/0 [...]

Th top one is primary (see the RX packets), so let's bring that device down, simulating a crash:

Node2# ip link set vlan100 down [...] tipc: Blocking bearer <eth:vlan100> [...] tipc: Lost link <1.1.2:vlan100-1.1.1:vlan100> on network plane A

Node1 reacts to this immediately and does a changeover to link 2:

[...] tipc: Resetting link <1.1.1:vlan100-1.1.2:vlan100>, changeover initiated by peer [...] tipc: Lost link <1.1.1:vlan100-1.1.2:vlan100> on network plane A

Despite the failure and recovery, our transmission continues happily, unacked messages that were already sent out on the failed link will be retransmitted on the new primary one after the changeover:

Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 34120200 212992 212992 100.00 648.45

Regarding load balancing, I'll write a demo application for that soon. In the meantime, here's an example in pseudocode that will help you to understand the concept.

The same service is published on both node1 and node2, using partially overlapping names:

Node1: [...] zmq_bind (sd, "tipc://{1000,0,1} [...] Node2: [...] zmq_bind (sd "tipc://{1000,1,2} [...] [[/node]] Now, a client on Node3 wanting to use this service can do so by connecting to either: [[code]] A = zmq_connect (sc, "tipc://{1000,0}") B = zmq_connect (sc, "tipc://{1000,1}") C = zmq_connect (sc, "tipc://{1000,2}")

Load balancing works this way:

• A only matches the service published by Node1.
• C only matches the service published by Node2.
• B matches both Node1 and Node2. TIPC will do round-robin selection between these two, spreading out the connections evenly.

There's still a missing feature in the code that's on libzmq master right now. By default, TIPC sets the "lookup domain" to closest first. This means that if there is a published service on the local machine that matches your request, that will be used for all requests, even when you spread your service over multiple nodes.

So, either I will add a way for the application to specify the lookup domain in the connect string, or I'll set the default lookup domain to always do round-robin over the entire cluster.

The TIPC 2.0 Programmer's Guide explains a little about how name sequences work.

Thank you so much, Erik, for this excellent addition to ZeroMQ, and your explanations. What are your plans for TIPC support in future versions of ZeroMQ?

Looking forward, we'd like to extend this with a RDM multicast transport module as well, which will make it very fast. This would add the ability to reliably send a series of messages to 0..n recipients. We are also investigating how to expose the topology service to ZeroMQ applications in a portable way.

I'm looking forward to this. Incidentally, the TIPC transport will appear in ZeroMQ v4.1

by pieterh

Goals and Requirements

These are the requirements I wrote down when designing this certificate format:

• The core goal is to give people a format they can reuse, rather than ask every project that uses ZeroMQ security to invent their own format. A single format lets different applications interoperate, and lets us make a few 10-day projects that work together instead of 100 2-day projects that cannot.

• The format we use has to be simple and human readable. It has to be portable to any system and obvious to implement, and language-neutral.

• A certificate must be capable of carrying at least the CURVE security keys, but should be extensible to any security mechanism, given that ZeroMQ is itself extensible to new security mechanisms.

• A certificate may be symmetrically encrypted with a passphrase, if it holds secret information. We should use a standard encryption algorithm to ensure interoperability.

• A certificate may be asymmetrically encrypted to a recipient, when it will be sent across an untrusted channel. We should use a standard authentication and encryption mechanism to ensure interoperability.

• The certificate must be safe to copy/paste into email, and it must be possible to automatically extract one or more certificates from an email message.

• The certificate should be easy to verify by hand (e.g. over a voice call), using a text fingerprint.

Use Cases

These are the use cases I know about for the CURVE security mechanism:

• A server generates a public + secret keypair, stores the secret key securely, using a passphrase, and stores the public key in clear text. This creates two certificate files, one meant to be kept locally, and one meant to be published.

• A client receives a public certificate from a server. It stores this in a location where it can access it at run time, when it needs that server's public key.

• A client generates a public + secret keypair, stores the secret key securely, using a passphrase, and encrypts and signs the public key from its own secret key to the server's public key.

• A server receives a signed and encrypted public certificate from a client. It stores this in a location where it can access it at run time, when the client makes a new connection.

Argumentation

One Envelope, many Contents

To support arbitrary security mechanisms, it seems useful to split the envelope (how the certificate is represented) from the content (the security keys and metadata). Specifically:

• The envelope will always be in clear text, while the content may be encrypted.
• The envelope will have a small set of standard properties, while the content should be fully extensible.
• The envelope will rarely change, while the content will change frequently over time as we refine our security mechanisms.
• We would create and process the envelope, and contents, at two different layers, a generic certificate codec, and a specific layer for each different mechanism.

Figure 1 - Two-layer Certificate

So, in this article I'll aim towards (a) a certificate envelope format and (b) a certificate content format for CURVE.

Email Armoring

This is a common problem with a well-known solution. To let us copy/paste a certificate into an email, and extract certificates from emails, we put a special marker before and after the certificate. Let's use this style:

-----BEGIN ZEROMQ CERTIFICATE----- ... -----END ZEROMQ CERTIFICATE-----

Further, lines must be at most 72 characters long, and we should use only 7-bit ASCII.

Envelope Format

In the envelope we will want to note:

• The version number for the envelope, so we can assure forward compatibility.
• The mechanism name, so that the codec can pass decrypted content to the correct mechanism.
• Whether the content is encrypted, and if so, using what algorithm.
• Whether the content is signed, and if so, using what algorithm and public key.

A simple and well-understood format for the envelope is HTTP-style headers, e.g.:

Version: 0.1 Mechanism: CURVE Comment: The certificate content is signed by the sender using their s\ ecret key and encrypted to the recipient's public key. We put both pub\ lic keys into the envelope so the reader can decrypt using the right s\ ecret key. Here, the "Content-security" header is actually redundant. Content-security: signed Content-signed-by: Yne@$w-vo<fVvi]a<NY6T1ed:M$fCG*[IaLV{hID Content-signed-to: rq:rM>}U?@Lns47E1%kR.o@n%FcmmsL/@{H8]yf7

Note that we're splitting long lines at column 72 and using a backwards slash to indicate that the line continues. This makes the text safe to send in emails.

Content Format

ZeroMQ is notable for transporting opaque frames without trying to define their internal structure. We can do the same here and reuse the successful pattern where the content consists of zero or more opaque frames. Having said that, we should also standardize the content for at least the CURVE mechanism we want to support today. Presumably in two separate RFCs.

We will want a consistent syntax for content framing so the codec can do its work. Since we're building a text format, we can define a frame as a single line of text. We can use the same continuation line style as we did with headers.

In ZeroMQ's security model, all mechanisms have metadata, which consists of zero or more name=value pairs. We can specify that the metadata sits in the first frame, and that other mechanism-specific properties follow in further frames. To indicate an empty frame, let's use a single hyphen, "-".

Here is the sketch of a PLAIN certificate with metadata, name, and password as three frames:

-----BEGIN ZEROMQ CERTIFICATE----- Version: 0.1 Mechanism: PLAIN Uuid=b19e19a0-34aa-11e3-aa6e-0800200c9a66;Date-created=14 October, 2013\ ;Created-by=Pieter Hintjens;Email=ph@imatix.com admin topsecret -----END ZEROMQ CERTIFICATE-----

Note that the metadata is using a name=value format, with semicolons to separate pairs. In ZMQ RFC 23, metadata names are case-insensitive, so we'll use that convention here too.

Overall this looks somewhat like the SSH 2 public key format (IETF RFC 4716), but is adapted to the ZeroMQ extensible security model. As in IETF RFC 4716 we do not use a blank line between headers and content since it serves no purpose. The headers continue as long as there is a line containing ": ".

Binary Content Encoding

When we encrypt the content, all frames have to be encrypted the same way. We could encrypt each frame individually but this exposes the frame structure to anyone who reads the certificate. Instead, we'll write the whole content (all frames) into a buffer, including line endings, and then encrypt that whole buffer. We will then use the ZMQ RFC 32 (Z85) encoding to represent the binary data safely as text. Z85 is more compact than Base16 or Base64, and the ZeroMQ core API provides encoding and decoding methods. Like all ASCII armoring, Z85 requires padding. To allow safe decoding of Z85 data we need to know the size of the data, excluding any padding. For double-checking we also need the padded length, i.e. the actual number of characters in the encoded content.

To authenticate a certificate, we need to verify the encrypted content, e.g. over the phone. This is not practical for typically long encrypted content. We thus use the SSH standard for generating an MD5 digest of the content. MD5 has known security weaknesses: you can modify a content and generate the same hash, if you can insert 128 bytes of data at some point. Since our encrypted content has a structure that is unknowable without decryption, it cannot be successfully modified. Thus, MD5 provides an acceptable signature. We will use the SSH-style signature consisting of a sequence of 16 octets printed as lower-case hexadecimal and separated by colons.

We can place the content sizes and signature in a first frame, and the armored text into a second frame. As before, we can continue long frames over multiple lines. So this gives us, for example:

381,384,c1:b1:30:29:d7:b8:de:6c:97:77:10:d7:46:41:63:87 nP2U)xK@r8zF9)4zF78kwNPQ?xDQ]9AV!^kzE[2mv}xX5x>z6?vru66w]zZbvrrS7z/M$7w\ PzY-q7J?YazUX3yH}30z!pb2ax@J3Bz%n]fly(FwNPa4xkI6{wId[wxd)[mvqYTezE){lz/\ PRCDs.bnze?gswP?T3fetjsgCQ7<x>7Y%y?X67y<vdBxMvV)wft/daBmQtz/6FCwmY{&aA}\ KwxDRwmzdK&izY&FawftG5AX5tHx([DkwGUz]A+e*0wO#PvayPd#az+$rB0bykazCv$vrj4\ znP2U)xLzJgvrcE7v@bZ0nHFs<aAhvjB7GlhayMylBAg/laz>L8aARp9BrCKlz/xJ7ay!?$\ ayX[<Bs[WUra]?=w]zZbvrrSaB0a}2B7F(8z/cXtxK@q<xMOuniXJdgCxUVNwPRJpz/PFzA\ +n/mA+n/rh.^iyBA?)Iq=ujxaAhEjy?Wx4yH}=lw]zZbvrt^c3ig5a

To ensure that the same content survives decoding and re-encoding across multiple platforms, we will specify that the buffered content may only contain LF as its end of line marker. Carriage return characters in the decoded content should be treated as a fatal error, and the certificate rejected.

Technical Proposal - Certificate Envelope Format

Syntax and Grammar

Here's a ABNF grammar for the certificate envelope format:

certificate = begin *header content end begin = "-----BEGIN ZEROMQ CERTIFICATE-----" eoln eoln = CR | LF | CRLF header = header-name ": " header-value eoln header-name = defined-header / extension-header defined-header = "Version" / "Mechanism" / "Content-security" / "Content-signed-by" / "Content-signed-to" / "Comment" extension-header = "X-" 1*62(LETTER / DIGIT / "-") header-value = *header-cont header-last header-cont = 1*CHAR "\" eoln header-last = 1*CHAR eoln content = *clear-frame / content-crypt clear-frame = "-" / (*clear-cont clear-last) clear-cont = 1*71CHAR "\" eoln clear-last = 1*72CHAR eoln content-crypt = length-frame armored-frame length-frame = 1*DIGIT "," 1*DIGIT "," signature signature = 15(signature-octet ":") signature-octet signature-octet = 2(DIGIT / "a" / "b" / "c" / "d" / "e" / "f" ) armored-frame = *armored-cont armored-last armored-cont = 1*71armored-char "\" eoln armored-last = 1*72armored-char eoln armored-char = DIGIT / LETTER / "." / "-" / ":" / "+" / "=" / "^" / "!" / "/" / "*" / "?" / "&" / "<" / ">" / "(" / ")" / "[" / "]" / "{" / "}" / "@" / "%" / "$" / "#" end = "-----END ZEROMQ CERTIFICATE-----" eoln

Notes on this syntax:

• Header names are case-insensitive, while header values are case-sensitive.
• The order of headers is not relevant except that a second header with the same name as the first will override the first header.
• Header values are limited to 1,024 characters. RFC 4716 allows quoted values but this adds nothing so I've not allowed that.
• The content starts on the first line not containing ": ".

The Version Header

Specifies the version of the certificate format used. This header is mandatory. It may have these values:

• 0.1 - specifies draft version 1. Draft versions do not provide any guarantee of backwards compatibility.

The Mechanism Header

Specifies the security mechanism that the certificate implements. This header is mandatory. It may have these values:

• PLAIN - implements the ZeroMQ PLAIN mechanism, as defined by ZMQ RFCs 24.
• CURVE - implements the ZeroMQ CURVE mechanism, as defined by ZMQ RFCs 25 and 26.

The content depends on the mechanism and will be defined separately for each standardized mechanism.

The Content-security Header

Specifies whether the content is encrypted and how. This header is optional. It may have these values:

• clear - the content is carried as clear text.
• password - the content is encrypted using a symmetric key derived from a user-supplied password, using the scrypt algorithm. Note: this needs more study and explanation.
• signed - the content is authenticated and encrypted using the crypto_box function of NaCl or libsodium.

If the Content-signed-by and Content-signed-to headers are present, Content-security defaults to "signed", else to "plain".

The Content-signed-by Header

When Content-security is "signed", specifies the sender's public key used to authenticate the content. The key is provided in Z85 format and must be 40 characters long. The recipient uses this to authenticate the certificate content (not the certificate itself, which must be authenticated by some other channel).

The Content-signed-to Header

When Content-security is "signed", specifies the recipient's public key used to encrypt the content. The key is provided in Z85 format and must be 40 characters long. The recipient uses this to find the correct secret key to decrypt the content.

The Comment Header

Contains a user-specified comment. The comment MAY be displayed when using the key. The comment MUST be stored with the certificate and copied if the certificate is recreated on disk.

Validating a Certificate

When reading a certificate, these are the minimum set of validations the code should do. A certificate that fails any of these checks can be considered invalid:

• The certificate has valid begin and end lines.
• The version and mechanism headers are present, and have valid values.
• The content, if encrypted, has a valid size and fingerprint.

Technical Proposal - PLAIN Certificate Content Format

The PLAIN mechanism uses these frames:

• Frame 1 is metadata, as with all mechanisms.
• Frame 2 is the username.
• Frame 3 is the password.

A PLAIN certificate:

• MAY use signed content security when it is sent across insecure channels.

Technical Proposal - CURVE Certificate Content Format

The CURVE mechanism uses these frames:

• Frame 1 is metadata, as with all mechanisms.
• Frame 2 is the public key, and is always present.
• Frame 3 is the matching secret key, and is optional.

A CURVE certificate:

• MUST use signed content security, and MUST not contain a secret key, when it is sent across insecure channels.
• MAY use password security, when it is held for a peer's own use.
• MUST use clear security when it is meant to be published openly to all recipients.

Conclusions

I've presented a hopefully simple certificate format that can store anything we need to implement ZeroMQ mechanisms at server or client, and which can be extended to new mechanisms. What's missing here is some proposal for standard filenames and certificate locations on disk. This should be part of a standard specification, I think.

by pieterh

The following list is meant for discussion. Based on that, I'll edit this post into shape and come up with some proposals that we can beat into shape.

Syntax vs. Semantics

To start with, do we need a single standard file format? I think no, it doesn't matter what format a certificate is stored in, as long as it's convertible. That means consistent semantics. One name for a thing, where the name is formally defined. But whether we use JSON, XML, YAML, or hand-crafted 13-bit binary formats, does not really matter.

Extensibility

My pet mechanism for ZeroMQ is CURVE, which implements the CurveZMQ handshake. However people are already starting to design other mechanisms for ZeroMQ, which has an extensible security model. Do we want to have a new certificate discussion for every mechanism? I think it would be better to have one format that different mechanisms can reuse (if they want to, since some mechanisms will already have their own certificate formats). Since each mechanism will have its own requirements, how about an extensible certificate format?

Metadata Headers

Certificates probably need some place to store metadata, such as the name of the person who this certificate identifies, partly to make management easier, perhaps also for authentication and access control. Let's say metadata consists of a set of named properties, without hierarchy. Let's make the names case-insensitive to avoid mistakes when people type them by hand.

Multilevel Certificates

At least for CURVE, we need two certificates; a public one (metadata + public key), and a secret one (metadata + public key + secret key). For the application, this could maybe be transparent, or abstracted in some way.

A Standard API

Speaking of applications, if we have a standard certificate format for ZeroMQ applications, shouldn't we also have a standard API to work with them? That could even be embedded into the core libzmq API so that it's available to all language bindings. By "work with", I mean at least creating but also possibly parsing certificates.

Encryption of Secret Keys

Having a secret key in plain view is troublesome since it's too easy to leave a directory readable to other processes. Even if applications check that secret certificates have the right permissions, that doesn't guard against any exploit that gets root access. So perhaps we need passphrase encryption of the certificate, where the client application can prompt the user for the passphrase when starting up.

Certificate Signing

I don't see a way to safely share a certificate without some shared secret, or resorting to a third party, CA-style. Even if I encrypt the certificate with the recipient's public key, they can't authenticate that without knowing my public key in advance. Is there a simple answer to this?

Public Infrastructure

Some people will certainly build certificate servers, but do we want this to be part of the certificate design? I think that's a strong "no" since it raises the barrier for even simple cases. A certificate format should be simple for simple case.

by pieterh

Requirements & Kicking Off

For this tutorial you will ZeroMQ v4.0, CZMQ v2.0, and Sodium (the crypto library). We will build from source, since we're going to be coding anyhow. You can grab these either as download packages (here's Sodium, ZeroMQ, and CZMQ), or from GitHub (here's Sodium, ZeroMQ, and CZMQ).

Sodium is a packaging of the NaCl crypto library, which implements the Curve25519 elliptic curve cryptography (ECC) algorithm. It doesn't matter for this tutorial, but when you use ECC in real work, people will ask you (a) isn't ECC backdoored by the NSA?, and (b) how secure is this, and (c) how fast is it, and (d) who else uses it?

The answers are, at least today, (a) some ECC curves may be, and older crypto like RSA probably is, but Curve25519 is not, (b) it provides 256 bit ECC keys which is as strong as 3072 RSA keys, and (c) it's one of the fastest elliptic curves out there. Also, (d) it's not exactly mainstream, but is gaining popularity and Google is using it for their Quik protocol.

I'll assume you're working on Linux or OS/X and have gcc or similar installed, and some idea of how to compile and link C programs. The API I'll use is that provided by CZMQ. If you are working in another language, it may or may not provide the same level of abstraction. I'll explain what CZMQ is doing, in any case.

Here's how to build from GitHub (building from packages is very similar, you don't clone a repo but unpack a tarball):

git clone git://github.com/jedisct1/libsodium.git cd libsodium ./autogen.sh ./configure && make check sudo make install sudo ldconfig cd .. git clone git://github.com/zeromq/libzmq.git cd libzmq ./autogen.sh ./configure && make check sudo make install sudo ldconfig cd .. git clone git://github.com/zeromq/czmq.git cd czmq ./autogen.sh ./configure && make check sudo make install sudo ldconfig cd ..

Next, let's check that things are working, with a minimal example:

Here's how I build and run this:

gcc -o hello hello.c -lczmq -lzmq -lsodium ./hello

If that prints "Hello, Curve", then you're on the road to success. If not, check you have the necessary packages. On Debian/Ubuntu, you'll need build-essential, for instance.

What We'll Make

We'll make a minimal server-to-client PUSH-PULL flow with one server and one client. To make things really simple, the server will bind, the client will connect, and the server will send one message to the client:

Figure 1 - Minimal Server to Client

I'm going to make this in five steps, so you can see the different security patterns that are possible with today's ZeroMQ. To make it easy to remember these five patterns, I've given them mnemonic names:

• Grasslands, which is plain text with no security at all. All connections are accepted, there is no authentication, and no privacy. This is how ZeroMQ always worked until we build security into the wire protocol in early 2013. Internally, it uses a security mechanism called "NULL", but you don't even see that.

• Strawhouse, which is plain text with filtering on IP addresses. We still use the NULL mechanism, but we install an authentication hook (I'll explain this in a second) that checks the IP address against a whitelist or blacklist and allows or denies it accordingly.

• Woodhouse, which extends Strawhouse with a name and password check. For this, we use a mechanism called "PLAIN" (which does plain-text username and password authentication). It's still really not secure, and anyone sniffing the network (trivial with WiFi) can capture passwords and then login as if they were the real Joe.

• Stonehouse, which switches to the "CURVE" security mechanism, giving us strong encryption on data, and (as far as we know) unbreakable authentication. Stonehouse is the minimum you would use over public networks, and assures clients that they are speaking to an authentic server, while allowing any client to connect.

• Ironhouse, which extends Stonehouse with client public key authentication. This is the strongest security model we have today, protecting against every attack we know about, except end-point attacks (where an attacker plants spyware on a machine to capture data before it's encrypted, or after it's decrypted).

As I said in my last article, privacy is about cost, and these patterns get increasingly expensive to use. Stonehouse and Ironhouse are not as easy to deploy as TLS, for instance. But they are also much more expensive to break, and for most ZeroMQ users that is a good tradeoff.

The Grasslands Pattern

We'll kick off with the simplest possible example, which looks just like classic ZeroMQ code, because it is:

To keep the code short and focus on the API calls, I'm not handling errors here. This is fun, but dangerous, like driving a motorbike without a seatbelt. In practice, since your code will fail in lots of unexpected places, it's wise to catch at least the more basic problems early on.

When you start to use the ZeroMQ and CZMQ security APIs, the main symptom of confusion will be "nothing happens", in other words, there's no connection when you expect one. To make sure it's not a basic bind/connect error, at least check the return codes from those two calls. For example:

int rc = zsocket_bind (server, "tcp://*:9000"); // If the bind fails, don't even attempt to recover assert (rc != -1);

There are people who hate asserts, but they're an extremely useful and brutal tool for forcing your code through a single sane path, and giving you accurate and early proof of errors.

The Strawhouse Pattern

Our next model takes the same code and adds address checking. I'll explain what we're doing and how it works in detail, first here is the application code:

When you use CZMQ, all authentication happens through a zauth object, aka an "authenticator". In the Strawhouse code we create an authenticator, we enable verbose tracing (during development, at least), and we whitelist one address. Then we do the same work as Grasslands, and finally we destroy the authenticator:

zauth_t *auth = zauth_new (ctx); zauth_set_verbose (auth, true); zauth_allow (auth, "127.0.0.1"); ... zauth_destroy (&auth);

Internally, the authenticator talks to libzmq across a protocol called ZAP, aka RFC 27. Every single time a client tries to connect to a server, libzmq asks the ZAP handler (the authenticator), if there is one installed, to allow or deny the connection.

We tell libzmq that the server socket is in fact a "server" in the sense that it authenticates clients, by using this call:

zsocket_set_zap_domain (server, "global");

The ZAP domain concept lets you define authentication groups. For instance a set of server sockets could say, "We're in a domain called "test" and we only accept localhost IP addresses". CZMQ doesn't implement domains yet, and the "global" domain we use here has no significance at all. Remember that in this example we're using the NULL security mechanism, and by default libzmq just accepts NULL connections, period. But the simple fact of telling libzmq, "Please set domain X on this socket (with the default NULL security mechanism)" will cause it to act as a server, and call the ZAP handler for every client request.

You can mix different kinds of security in one application, by using different sockets, and then configuring each security mechanism separately in the authenticator. So it will handle PLAIN in one way, CURVE another way, and so on.

Now, to CZMQ's IP address filtering. This uses whitelisting and blacklisting, and works thus:

• You can whitelist one or more addresses, or blacklist one or more addresses. We do this through the zauth_allow() and zauth_deny() calls, rather than loading external files.

• The whitelist or blacklist applies to all requests that the authenticator gets from libzmq. That is, any NULL socket with a ZAP domain set, any PLAIN server socket, or any CURVE server socket, on that ZeroMQ context.

• If you have a whitelist (one or more addresses), that means, "reject any address that's not in the whitelist". Think of a private party, a very large bouncer, and a sheet of paper with names on it. "Sorry, buddy, you're not on the list. Please move out of the way."

• If you have a blacklist (one or more addresses), that means, "reject any address that's in the blacklist". Think of a public function with a couple of very large bouncers and a sheet of paper with names on it. "Hey buddy, didn't we tell you to never show your face here again?" You won't use both a whitelist and a blacklist, unless you want to make the bouncers confused, and angry.

• If not denied, and the security mechanism is NULL, then the connection is allowed.

• If not denied, and the mechanism is PLAIN or CURVE, then the authenticator carries on with its checking. So perhaps you're on the whitelist, or not on the blacklist, but now we need to see some identification, please.

So in our example, where we allow "127.0.0.1", we effectively deny every other client address.

The Woodhouse Pattern

Next, let's see how to do username and password authentication. Here is the full example:

Setting up the authenticator and the whitelist is the same as in Strawhouse, and I won't explain that again. The new API calls happen in three places in the code:

// Tell the authenticator how to handle PLAIN requests zauth_configure_plain (auth, "*", "passwords"); ... // Before binding the server zsocket_set_plain_server (server, 1); ... // Before connecting the client zsocket_set_plain_username (client, "admin"); zsocket_set_plain_password (client, "secret");

Bear this in mind: you have to configure a socket before doing a _bind or _connect on it. If you run the Woodhouse example without a password file, it'll never complete. Since we enabled tracing, you'll see it print out this:

I: PASSED (whitelist) address=127.0.0.1 I: DENIED (PLAIN) username=admin password=secret

The message in fact repeats because ZeroMQ re-tries the connection over and over. We mught change this behavior but it's not significant here. Let's create a password file, "passwords", in the current directory, with some entries, one per line:

guest=guest tourist=1234 admin=secret

Then run the Woodhouse example again, and it'll print this:

I: PASSED (whitelist) address=127.0.0.1 I: ALLOWED (PLAIN) username=admin password=secret Woodhouse test OK

The Stonehouse Pattern

Plain text is what it is. Anyone with half a malicious intent can sniff traffic and impersonate real clients, and make a mockery of your so-called security. That may be fine, or it may be a problem. The next level up is Stonehouse, which changes the game somewhat. Here's the example code:

Stonehouse starts with a single call to the authenticator that switches on the CURVE_ALLOW_ANY feature:

zauth_configure_curve (auth, "*", CURVE_ALLOW_ANY);

Here we do that for the domain "*", meaning all domains. Recall that CZMQ doesn't deal with domains yet, so if you use Stonehouse on one server socket, it will apply to all server sockets in the same context.

Now, let's talk about certificates. For any CURVE security you need a "long-term public / secret keypair" for the client, and for the server. CZMQ wraps up a keypair along with metadata (a name, email address, etc.) as a "certificate", which can be an in-memory object, or a disk file. Here is the explanation from the zcert class:

The zcert class provides a way to create and work with security certificates for the ZMQ CURVE mechanism. A certificate contains a public + secret key pair, plus metadata. It can be used as a temporary object in memory, or persisted to disk. On disk, a certificate is stored as two files. One is public and contains only the public key. The second is secret and contains both keys. The two have the same filename, with the secret file adding "_secret". To exchange certificates, send the public file via some secure route. Certificates are not signed but are text files that can be verified by eye.

Certificates are stored in the ZPL (ZMQ RFC 4) format. They have two sections, "metadata" and "curve". The first contains a list of 'name = value' pairs, one per line. Values may be enclosed in quotes. The curve section has a 'public-key = keyvalue' and, for secret certificates, a 'secret-key = keyvalue' line. The keyvalue is a Z85-encoded CURVE key."

What you would do in practice is generate certificates up-front (and CZMQ provides a simple tool to do this, in addons/makecert), and then put them carefully on each node in your network. It's like handing out neat little membership cards for an exclusive club, up front. Makes life easier for guests and security alike.

In the Ironhouse example we'll see how to work with "real" certificates on disk, using a so-called "certificate store". However for this example, we can generate certificates on-the-fly, in memory, and pretend we got them from disk.

We need a client certificate, a server certificate, and then we also have to pass the server's public key to the client. This is the total setup to use the Stonehouse pattern. CZMQ provides a zcert class that generates a new certificate, and which we can "apply" to a socket in order to enable it for CURVE security. Finally, we have to tell the client socket what our server public key is. Here are the relevant API calls:

// Generate certificates at runtime, since this is a test zcert_t *client_cert = zcert_new (); zcert_t *server_cert = zcert_new (); char *server_key = zcert_public_txt (server_cert); ... // Before binding the server zcert_apply (server_cert, server); zsocket_set_curve_server (server, 1); ... // Before connecting the client zcert_apply (client_cert, client); zsocket_set_curve_serverkey (client, server_key); ... // Destroy these in-memory objects when done zcert_destroy (&client_cert); zcert_destroy (&server_cert);

The zcert_new() call generates a public/secret key pair. We can ornament that with metadata, e.g. name, email, and organization, which can be helpful when managing certificates. Metadata doesn't play any role in authentication however.

The Ironhouse Pattern

Now, let's explore full authentication of clients. I call this the Ironhouse pattern, just to remind us that security is never total or perfect. It's always about cost. Even an iron house may have holes in the floor, or the roof, and your security is only as good as the systems it runs on. If you run Ironhouse on a box with an insecure or outdated operating system, don't expect privacy.

Here's the Ironhouse example:

Ironhouse is very similar to Stonehouse except it uses a "certificate store". That is a directory somewhere on the server's file system that contains a set of client public certificates. This is the API we use:

// Tell authenticator to use the certificate store in .curve zauth_configure_curve (auth, "*", ".curve"); ... // Stick the client public certificate into the directory zcert_t *client_cert = zcert_new (); zcert_save_public (client_cert, ".curve/testcert.pub");

For the client, nothing changes — it's the same code as for Stonehouse, as you'd expect. If this wasn't clear already, the zauth object belongs to the server, not the client.

A word about certificate stores. This is a CZMQ concept, and implemented by the zcertstore class. Normally you won't use that class directly, but like this example, allow the authenticator to manage it. All we do is (a) tell the authenticator what directory will hold certificates, and then make sure our clients' public certificates end up safely there.

In practice a server application will be a long-running process, so the certificate store reloads itself automatically if there is a change. This means we can copy new certificates into it at any time, to allow new clients, or remove certificates to reject clients.

Certificate filenames don't matter. Here we use "testcert.pub" but in practice I'd use a filename which is based of the certificate public key, for instance, to ensure it's unique. In our example we create the directory, and add some meta data, before saving the certificate:

// Create the directory zsys_dir_create (".curve"); zcert_set_meta (client_cert, "name", "Client test certificate"); zcert_save_public (client_cert, ".curve/testcert.pub");

When you save a certificate using the zcert_save_public (or zcert_save) method, it looks like this:

# **** Generated on 2013-09-19 21:25:34 by CZMQ **** # ZeroMQ CURVE Public Certificate # Exchange securely, or use a secure mechanism to verify the contents # of this file after exchange. Store public certificates in your home # directory, in the .curve subdirectory. metadata name = "Client test certificate" curve public-key = "I7[{YV4[}q[9a)]b&d>bisoT]UXa/7b$Tp:6yoyq"

In own applications I'm using $HOME/.curve to hold certificates, as zcert recommends, but you can put them anywhere. Some guidelines do apply:

• A certificate produced by makecert has a secret and a public file. Never share the secret file with anyone.

• When you transfer public certificates, use a secure route. If an attacker can modify this data without you knowing about it, they can construct a man-in-the-middle attack.

• For the same reason, make sure public certificates are not writeable by anyone except the current user.

The Ironhouse Pattern, Model 2

When you read the examples so far, you might be tempted to copy the code for your own applications. However it's not meant for that. It doesn't have a clean separation into client and server, and doesn't do any error checking. Here's a reworking of the Ironhouse example that is much more explicit. It's also twice the amount of code:

Here we see how to save and load client certificates from disk, to create truly persistent certificates:

zcert_t *client_cert = zcert_new (); zcert_save (client_cert, "client_cert.txt"); zcert_destroy (&client_cert);

And then, in a different place and time, or at least a separate thread:

// Load our persistent certificate from disk zcert_t *client_cert = zcert_load ("client_cert.txt");

Wrapping Up

OK, that's enough code. After half a dozen examples, we start to get code blindness. What is striking however is how short the code examples are, even in C. For me, this is success.

There's one last thing I want to discuss about security, which is the constant harping on about "server" and "client", which you may know are not ZeroMQ native concepts at all. The closest we get in ZeroMQ is having one side binding to an endpoint, and the other side connecting to it.

When using the PLAIN or CURVE security mechanisms (wood, stone, or iron), you can bind the client and connect the server. You can try this in stonehouse.c for example. Simply switch the bind and connect statements (change the socket arguments :) and observe that it still works. The server still authenticates the client.

You'll get the best results by typing the code, because muscle memory is a fantastic thing for programming. But if you're in a hurry, it is in the CZMQ project, in the examples/security directory. These examples are licensed under MIT/X11 so you can reuse the code in your own applications.

Let me know how you get on. Comment below, or come to the zeromq-dev mailing list. If you need commercial support in using these new security tools, or you're an open source project that wants to use them, drop me an email at moc.xitami|hp#moc.xitami|hp and tell me your use case.

by pieterh

The Case for Privacy

The summer of 2013 was when privacy on the Internet died. Or rather, when our belief in privacy died, because the surveillance of our phone calls, emails, web searches, and chats has been going on for many years, if not since the start of the telegraph era.

Privacy is a cost issue. An organization can secure its data centers from surveillance (by spies of all colors, whether they are foreign intelligence services, competitors, or criminals) by building its own physical networks and isolating these from the public Internet. However that is extraordinarily costly. More typically, we'll use virtual private networks (VPNs) to create secure lines that cross public networks.

VPNs are not reliably secure, however. They use long term keys that can be broken. They are complex to configure and to upgrade. The often use aging algorithms like RC4 that are considered crackable.

Even SSL/TLS are not immune from attack. The NSA themselves point out that the mainstream cryptography is over 20 years old, and state that "the best assured group of new public key techniques is built on the arithmetic of elliptic curves."

Without the shield of cheap, trustable privacy, an organization cannot safely use the public Internet. What the NSA can break today, any competitor or foreign intelligence service can break tomorrow. In our connected world, this throws a pool of ice cold water over the growth of low-friction Internet commerce.

Goals for ZeroMQ's Security Layers

Just as ZeroMQ brought us cheap, standardized connectivity for distributed systems, our goals with the security layers (which I'll explain in a minute) are to bring cheap, standardized privacy and authentication for distributed systems. The focus here is on cost. Cryptography has not stood still since the 1970s and there are many excellent, strong security options. However they remain costly to learn, understand, and safely use.

Here is our shortlist of goals:

• It must be very simple to use, and impossible to get wrong. Complexity is the number one risk with cryptography and the number one vulnerability. Every additional option is a way to get it wrong and end up with an insecure system.

• It must be fast enough for real work. If security makes a system too slow to use, people will switch it off, because the pragmatic need of being able to work today beats the risk of intrusion tomorrow.

• It must be based on standardized protocols, so that it can be re-implemented by any team, verified independently, and improved outside of the software stack.

• It must have free and open source implementations, so that the code can be inspected and fixed. One cannot trust a closed-source security stack, given the risk of secret backdoors.

• It must be free of patent risk, so that it cannot be blocked on grounds of intellectual property violations.

• It must work with all ZeroMQ socket types, and more generally, be interoperable with clear text messaging. A typical architecture will use clear text for internal messages, and security on external traffic.

• It must be built-in to the core libzmq library, so that it is available to all language bindings. Full-native stacks like JeroMQ should be able to re-implement the security protocols.

We started working on this in March of 2013, and the first formal publication will be in the forthcoming ZeroMQ/4.0.0 release. You can always get the latest version from the ZeroMQ GitHub project.

How ZeroMQ Security Works

The security architecture is broken into several layers:

• A new wire protocol, ZMTP 3.0, that adds a security handshake to all ZeroMQ connections.

• A new security protocol, CurveZMQ, that implements "perfect forward security" between two ZeroMQ peers over a TCP connection. I'll explain CurveZMQ below.

• A set of security mechanisms for ZMTP: NULL, PLAIN, and CURVE, each described by their own RFCs. NULL is essentially what we had before. PLAIN allows simple username and password authentication. CURVE implements the CurveZMQ protocol.

• An extended API for libzmq that lets you choose and configure the security mechanism for any socket. The default is NULL. You can then upgrade the socket to PLAIN or CURVE, either as a client (that connects securely to one server), or a server (which authenticates any number of clients).

• A protocol (ZAP) that lets you install arbitrary authentication servers. ZeroMQ lets you choose how you authenticate clients, be it via configuration files, LDAP, databases, etc.

• Finally, a separate security library, Curve, that implements CurveZMQ on top of ZeroMQ. Curve is a reference implementation for the CurveZMQ protocol, and a way to get that functionality on older versions of ZeroMQ. It will work with ZeroMQ/2.x and ZeroMQ/3.x (but not interoperate with the ZeroMQ/4.x CURVE mechanism, that works at a different level).

Let's walk through how we do secure PUB-SUB from one publisher to a number of subscribers, using the CURVE mechanism:

• We generate a long term keypair for the publisher. ZeroMQ provides a tool (curve_keygen) to do this. We store the resulting keypair safely.

• We manually distribute the public part of the keypair. This is a short text string. ZeroMQ does not do key exchange.

• If we want to authenticate subscribers, we generate a keypair for each subscriber, store the public+secret keypair safely at each subscriber, and collect all public subscriber keys on the publisher.

• Now, we can create a PUB socket and configure it with the long term keypair, and tell ZeroMQ that it is a server. This requires three zmq_setsockopt calls.

• We can then create any number of SUB sockets, and configure each one with its own keypair, and the server's public key. Again, three calls to zmq_setsockopt.

• We then bind the PUB socket, and connect the SUB sockets as usual.

• The ZeroMQ messaging API is unchanged. We send and recv messages as before.

One aspect that surprises some people is that one socket can have exactly one security level. You cannot have, for instance, PLAIN and CURVE subscribers connected to the same publisher. This would be technically feasible but would add extra work to applications to check the security level of every incoming message. It would make it easy to "get it wrong".

The Curve Story

Elliptic curve cryptography is a controversial topic. It has the reputation of being "backdoored" by the NSA, and heavily patented. There is no single curve, but an infinite number, and different curves have different strengths and weaknesses. Some of the "official" curves proposed by the NIST are suspect.

We use Curve25519 as designed by Daniel J. Berstein, which is the basis for the NaCl cryptographic library, more widely used as libsodium.

Curve25519 together with the other algorithms used in NaCl has some attractive features:

• It is fast.
• It has no known patent issues.
• It is considered strong.

Curve25519 provides encryption and authentication with 256-bit keys, equivalent to 3072-bit RSA keys.

This isn't sufficient guard against the simplest threat, where an attacker records encrypted data today, and then decrypts that tomorrow, after getting the keys by force, subterfuge, or warrant.

To prevent that, we need "perfect forward security", in which the two peers create temporary session keys and exchange those safely using the long-term keys. When the session is over, they discard their keys, and the encrypted data is then unreadable by anyone.

D. J. Bernstein developed the CurveCP protocol for secure networking over UDP, part of which was a secure handshake that provides perfect forward security using Curve25519.

The ZeroMQ CurveZMQ protocol is essentially a fork of CurveCP, adapted for use over TCP, and written as a single RFC.

Clients and servers have long-term permanent keys, and for each connection, they create and securely exchange short-term transient keys. Each key is a public/secret keypair, following the elliptic curve security model.

To start a secure connection the client needs the server permanent public key. It then generates a transient key pair and sends a HELLO command to the server that contains its short term public key. The HELLO command is worthless to an attacker; it doesn't identify the client.

The server, when it gets a HELLO, generates its own short term key pair (one connection uses four keys in total), and encodes this new private key in a "cookie", which it sends back to the client as a WELCOME command. It also sends its short term public key, encrypted so only the client can read it. It then discards this short term key pair.

At this stage, the server hasn't stored any state for the client. It has generated a keypair, sent that back to the client in a way only the client can read, and thrown it away.

The client comes back with an INITIATE command that provides the server with its cookie back, and the client permanent public key, encrypted as a "vouch" so only the server can read it. As far as the client is concerned, the server is now authenticated, so it can also send back metadata in the command.

The server reads the INITIATE and can now authenticate the client permanent public key. It also unpacks the cookie and gets its short term key pair for the connection. As far as the server is now concerned, the client is now authenticated, so the server can send its metadata safely. Both sides can then send messages and commands.

This handshake provides a number of protections but mainly, perfect forward security (you cannot crack data encrypted using the short term keys even if you record it, and then later get access to the permanent keys) and protection of client identity (the client permanent public key is not sent in clear-text).

Multilayer Security Using Libcurve

ZeroMQ's built-in Curve security can protect only one hop. For a B-to-C use-case this is ideal, and acts as a replacement for TLS/SSL, without the certificate authorities, but with the added cost of pre-distribution of public keys.

For C-to-B-to-C cases however, it is not sufficient, just as TLS/SSL is not sufficient. The cloud in the middle gets unencrypted traffic, and surveillance at that point is trivial.

To solve this we need security at a higher layer (instead of, or in addition to, the transport-layer security).

In effect this means encrypting messages before passing them to ZeroMQ, and then decrypting them after receiving them from ZeroMQ. The same requirement for perfect forward security applies, otherwise the data is vulnerable to key theft attacks.

With our security architecture, we use the Curve library (libcurve) to handle this higher layer. Curve provides both low-level codecs (to encrypt and decrypt) messages, and high-level "virtual sockets" (that act like ZeroMQ sockets but do security behind the scenes).

Coming Up

Security is complex, but if done right, will protect your business secrets from attackers. In my next article I'll start show example code that does secure messaging, over ZeroMQ. If you want to contact me directly about using ZeroMQ's security in your architectures, email me at moc.xitami|hp#moc.xitami|hp.

by pieterh

First, the Physics

I'm not a physicist, but my kind hosts — Emmanuel Taurel at ESRF, and then Wojtek Sliwinski at CERN — explained the basics to me. ESRF and CERN both have large circular accelerators, though "large" is relative. ESRF's is the size of a small airport. CERN's is the size of a small town. They also have linear accelerators (LINACs) that act as injectors for the larger circular ones. Actually CERN has a dozen different accelerators, built over time.

The major difference — from the physics point of view, not the software — is that ESRF accelerates beams of electrons, while CERN accelerates beams of protons, because the physics experiments they're conducting on both sites are very different.

ESRF scientists basically want to shoot streams of X-rays at samples to get information about their molecular structure. I assume that it's like shining a very bright light at them.

How do you get streams of X-rays? It turns out that if you take electron beams and bend them (or perhaps it's twist them, I'm not sure), they emit X-rays along the direction they wanted to go in. I've not tried this at home, but it makes sense.

So ESRF shoots electron beams (from very big cathode ray guns, like we used to have in TVs) into a ring and speeds them up, getting them to go faster and faster, and as they spin around, they emit X-rays where they get bent by the massive magnets. There are several kinds of magnets, the size of fridges. The X-rays are guided down with more magnets, filtered and bounced off mirrors, and whatnot, and focused on the samples, where high-speed cameras record the results. The ring carries forty independent beams of electrons, and along the ESRF ring there are forty corresponding experiment rooms.

In ESFR's X-ray factory, therefore, you have a whole set of devices — hundreds — that create, steer, twist, bend, and focus these electrons and X-rays. As there are forty independent beams, they manage each beam more or less separately, though I've no idea how they do this. Magic, I assume.

The control room is basically filled with large panels that show the status of the whole machine as it operates. From these panels, scientists can tune their beams, pump in more electrons, focus them, get readings, and so on, through an impressive graphical user interface. It is a "drive by wire" control system, which was innovative when it came along, for before that, scientists would literally tune their magnets and mirrors by hand.

Now, at CERN, the physics are very different. CERN's mission is to create black holes, or at least very large holes in the ground, because their largest ring, the Large Hadron Collider or LHC, is 27km around. You can't see it except from the buildings on the surface, dotted around its position.

At CERN, the X-rays are a nuisance. What the LHC does is create speeding clouds of protons, which are positively-charged particles with mass (electrons being negatively-charged wavelets that sometimes act as particles but with a tiny mass compared to protons). They speed these clouds up faster and faster, in two beams, one rotating clockwise, and one anti-clockwise. They then smash these two clouds into each other, simulating the Big Bang at the origin of the Universe, and seeing what the heck emerges from that.

The CERN experiments are basically massive collectors, and the LHC has four main ones, and three smaller ones. We visited the largest, ATLAS, buried underground. ATLAS is a "general purpose" collector, meaning it looks for many kinds of collisions. This is what makes it so large. The special purpose collectors focus on just one type of collision and can be much more compact.

So the experiments at ESRF and CERN are entirely different but the two control systems follow a similar pattern. It's these two control systems that both ESRF and CERN are rewriting to take advantage of state-of-the-art technology in messaging, namely ZeroMQ.

Second, the Architecture

The ESRF control software is called Tango, and is a collaborative effort between nine different research institutes. It's open source along the cathedral model (whereas ZeroMQ is a work of the bazaar). Tango used CORBA for its messaging and is now moving to ZeroMQ. Tango has the makings of a widely-used product, even a general-purpose industrial control system, or SCADA, since it is essentially simple and already used in multiple institutes.

The CERN control software (the "middleware") is a different animal. It's much larger and more complex, reflecting the much larger size of CERN's projects, teams, and ambitions. The CERN SCADA is also not open source. The two approaches make an interesting comparison, since there is inevitably some competition, or at least friendly rivalry, between institutes around the world.

The architecture consists of several layers:

• A set of physical devices (lasers, magnets, mirrors, rotating pigeons, and magic hats), aka the "front ends". These run some embedded stack or Linux, and talk over an array of weird and wonderful interfaces such as serial lines.

• A set of "device services" that manage a small group of devices, with custom drivers for each type of device. These device servers expose the devices to the world as addressable objects with properties and some methods. This is the CORBA view of the world as objects. It's not entirely sane, but not totally insane either.

• A name service, which holds a registry of devices and their network addresses (rather, that of the device server which talks to them).

• A set of interactive or batch applications that talk to the device servers using a set of patterns: asynchronous pub-sub, in which devices publish their status; asynchronous request-reply, in which applications query or set device properties, and synchronous request-reply, which is the same but blocking.

There are two API layers that hide the whole messaging stack from normal developers: one for those writing device servers, and one for those writing applications. In both projects the goal is to replace CORBA without modifying the APIs too heavily. CERN has rather more freedom to break the old APIs since the LHC is currently in shutdown, so it's okay to write new applications.

Other Thoughts

What are the biggest risks and constraints in these projects? Obviously, as in any SCADA, things can't break in ways that damage the whole system. Individual devices will, sometimes, fail. People will write buggy code. But the messaging layer has to be entirely invisible.

This means, mainly, no brokers. While CERN, at least, has a lot of brokers (such as ActiveMQ and RabbitMQ), they don't use these in the LHC control middleware, but for other projects. Each project team makes its own choices, and some people just really like STOMP and JMS, for example.

Throughput and latency are not yet problems, since in both projects the ambition is to replace a well-known technology with a conceptually similar one. ZeroMQ does have higher latency for the synchronous request-reply case but this doesn't matter (it never does; in any real low-latency design you will always work asynchronously).

However as ESRF and CERN gain more experience and confidence with ZeroMQ, I would expect them to move to more interesting use cases. Multicast is an obvious one since there are thousands of devices talking to hundreds of applications. But there are a few other areas where the ZeroMQ community could help a lot over time:

• Building a reusable name server (this would be so useful I plan to make one anyhow, so that "bind" automatically registers a new service, and "connect" automatically looks it up).

• Standardizing on protocols based on ZMTP and ZRE to talk to devices so that the device server layer could be removed.

• Helping ESRF and CERN with the process of building community around their open source layers. At CERN I was fortunate to be able to present our community-building process to a hundred or so developers. At ESRF I presented much the same arguments to a packed room of thirty or forty developers.

Conclusions

It's nice to see such significant organizations as ESRF and CERN choosing ZeroMQ for their most important control systems. I'd expected a little more competition from other products but there's nothing out there with the same mix of scale (the sheer number of projects in and around ZeroMQ is impressive), maturity, and appeal to developers.

With a little patience and attention, we could see ZeroMQ become the technology of choice not just in synchrotrons and accelerators but also factories and airports and more, and ZeroMQ's protocol — ZMTP — could become the "Protocol of Things" for the Internet of Things. Which would be fitting, since the Web was invented at CERN.

by pieterh

Preamble

Let me start by stating for the record that I'm not a security expert. I've got many… talents, such as knowing immediately if someone has sipped from my glass of beer, or making too many friends in too many places. But a deep and confident knowledge of how to stop the NSA reading my most private communications isn't one of them. However, the goal of this exercise is indeed to give us secure communications using ZeroMQ.

What is "security"? It's not one thing but a lot of answers to a lot of problems. At the core if A says something confidential to B, then A wants to know that C can't read it, can't modify it, can't pretend to be B, and can't imitate A either. But it goes further than this. Sometimes C will simply steal B's laptop, or kidnap B and torture him to give up his secret passwords.

If you study this long enough, you realize no-one is really a security expert. There are cryptographers and mathematicians, protocol designers and malware specialists, operating system security experts, code breakers and forensic cryptographers and on and on. I don't think any single person or team can build "security" or can even really know all the angles. What use is the best security library in the world if the application allows trivial SQL injections? Or if the operating system has known vulnerabilities? Or if the system admin is corrupt?

So this brings us back to something I repeat often in my writing and talks: it takes a big, diverse community to solve the hardest problems decently and cheaply. Luckily ZeroMQ is just that, which means given the "what" and the "who", I can segue into the "how".

How to solve a Very Difficult Problem

To solve a VDP, one starts by building a Minimal Plausible Solution, or MPS, and throwing this into the circus where the crowd can gather to watch it being torn apart by wild animals and religious fanatics. The crowd then realizes they can do better, so they make improvements and changes to the MPS and in turn throw these better versions into the ring. Over time the sport develops into a wild and furious game where the crowd split into factions each cheering different variations, and the noise attracts yet more wild animals and fanatics.

It's not quite that simple, but more or less, if you can throw a MPS into the ring and if people can improve this freely, you have a decent chance of seeing really good solutions emerge over time.

What are the characteristics of a good MPS? First, it must not take itself too seriously. You're designing Steak Tartar, remember. It has to be provocative and cheerfully cheap and simple. You're aiming for plausibility, not certainty.

Second, it has to at least work, so it can walk under its own power into the center of the ring, ready to be torn apart. We want the spectacle, the bloodshed, the drama, for it's this that pushes others to get involved. No crowd ever rioted after being shown a 50-slide PowerPoint.

Third, it has to be remixable in all directions. In the old days, when we ran punched card FORTRAN-77 on UNIVAC mainframes, we built security systems via committees of gray bearded elders who signed contracts defining who was allowed to make improvements to their Codes. These days we push everything under the (L)GPL and the use GitHub fork+pull request model. Old men are still welcome but only if they cut off their beards.

One of my talents is, I think, designing great MPSs and understanding how the circus works.

So this is what we've done for security for ZeroMQ. As you ask, "why didn't you implement SSL or TLS?", that makes you one of the wild animals. Enjoy the Steak haché tartare, it was meant for you.

Collecting the Problems

Diving down into the real grist of a minimal plausible solution, we need to collect some valid problems. It's no use producing an MPS for the wrong problems, that does nothing except waste everyone's time.

So what are the problems we need to solve first? You'd think it was something like "how do we implement a TLS handshake?" or some-such. But it turns out to be more meta than that. I started my research by looking at various security models, and making some quite elegant prototypes, but all I really learned was that choosing any particular model would be a mistake.

This is the first problem to solve: our own guaranteed incompetency when it comes to security. Even if we could build a perfect security model today (which I really doubt), it would become broken over time. It is not smart to build an expiration date into your software, especially if you are trying to make general purpose, long-lasting infrastructure, which is our goal with ZeroMQ.

There is, luckily, a known way to avoid having to choose a single security model, exemplified by SASL, an Internet security standard that's been used in quite a few protocols now. What SASL does, which is neat, is to delegate all security to a "mechanism" that client and server can implement independently of the rest of the protocol.

I find SASL overly complex, so would not implement it as-such unless someone was paying me to be inefficient. However it's quite doable to get the same effect with a simpler design. We stick a "mechanism name" into our protocol, and then depending on the mechanism the client and server propose, we call some plug-in library to handle the actual handshake.

The second problem is how to codify this so that we can get multiple independent implementations. It would be a big mistake to start by writing code (except as throw-away experiments). We need some formality that different stacks can aim at. Thus, standardization, and more specifically, in the ZeroMQ Message Transport Protocol (ZMTP) we use to tie our networks together.

Next, we need to prove our design by making a few different security mechanisms. It would be poor taste to force everyone into using full encryption, whether they need it or not. Some protocols then offer an on/off choice (e.g. insecure or secure HTTP). But it's smarter, I believe, to offer N choices, which can cover a spectrum of options. It fits with notion of extensible security and it also seems to fit what people really want, which is a sliding scale of trade-offs (usually performance and simplicity versus security).

So, let's recap our first set of problems:

• Making an extensible security model.
• Writing this up as a proper protocol specification.
• Creating a first set of security mechanisms.

We've been evolving ZMTP for some years from a minimal adolescent framing protocol to a more adult industrial-strength protocol. For example, adding a version header made it possible for newer versions of libzmq to talk safely to older ones, something that was impossible before.

ZMTP 3.0 basically breaks down into four layers or phases:

+-------------------------------+ | Version detection | +-------------------------------+ | Security handshake | +-------------------------------+ | Metadata exchange | +-------------------------------+ | Messages and commands | +-------------------------------+

The two peers exchange signatures and agree on a version; they then exchange credentials and agree on their security; they then exchange meta-data; and they then exchange commands and messages. The security handshake happens as early as possible to avoid leaking any information (the metadata) about peers.

I'm not going to explain ZMTP in much detail since there is a detailed RFC. But I can explain some of the design decisions. The mechanism is a single string. The two peers have to agree on this, or they can't work together. This is one big deviation from SASL, which allows peers to negotiate security dynamically. That could work in ZeroMQ but would force us to make the message API more complex (to indicate the level of security of the sender). We can't prove we need SASL's extra flexibility. So we allow just a single security mechanism.

ZMTP then specifies three default security mechanisms, NULL, PLAIN, and CURVE. The last two are specified by their own RFCs. They all use a somewhat similar handshake. The libzmq library implements the three mechanisms as separate classes.

So extensibility is now simple: create a new RFC that documents a new mechanism, and add the necessary class to libzmq. It's probably a week or two of work to add a new security mechanism like DTLS.

I'll briefly explain the three mechanisms. NULL means the client doesn't provide any credentials. We can check things like the IP address. PLAIN means the client provides a name and password, without encryption. It's meant to stop us doing stupid things like sending test data to production servers. Finally, CURVE implements an elliptic curve handshake called CurveZMQ, which is based on CurveCP. I'll explain this next.

Elliptic Curve Security, CurveCP, and CurveZMQ

CurveZMQ is a MPS for "full security" that was a lot more fun to make, and plausibly much more valuable, than an older asymmetric encryption model. I'm not a security expert, but elliptic curve cryptography, or ECC, works with shorter keys and uses less CPU than conventional cryptography.

However that's almost incidental. The real reason I chose to make CurveZMQ is because of something called CurveCP, a protocol and stack designed by mathematician and cryptographer Daniel J. Bernstein, author of the NaCl library. CurveCP was announced at the 27th Chaos Communication Congress on 28 December 2010 and CurveCP implementations are still considered experimental. I'm not going to claim CurveCP has any magical properties, is superior, or even works as-such. Indeed, the CurveCP protocol and software really could not work with ZeroMQ in any easy way, since it's designed to work over UDP.

Nonetheless, CurveCP starts with a handshake between client and server, and this handshake turns out, after we chop and cook it in various ways, to work very well both over TCP and over ZeroMQ DEALER-ROUTER, and to fit into our SASL-lite security model. There's a detailed RFC for CurveZMQ which explains the precise chopping and cooking I had to do.

CurveZMQ is essentially the CurveCP handshake, trimmed and cleaned-up to work over any bidirectional message transport, and documented as an RFC. I'm not yet sure how good or bad this is, but recall that we can't afford to take any particular mechanism too seriously. This is a minimal plausible solution for encryption, no more or less.

What immediately caught my attention was DJB's stated goal of making his APIs simple. The NaCl API is beautifully simple. I've worked with OpenSSL and it uses the industry standard approach of "expose every possible choice upfront with zero hints as to what is really vital, and what you will never ever use".

Whereas NaCl hides everything it can, on the theory that offering choices to people like me about what what key sizes or hashing algorithms to use is just asking for trouble. And I totally agree. NaCl is really like ZeroMQ, it hides choices we usually only get wrong.

Love at first sight! Whether or not the thing actually works, it was fun to learn and use and that's a powerful indicator. NaCl is in fact wrapped up as libsodium, which is the library we use.

So I wrote up CurveZMQ as a proper RFC and made a reference implementation. This library serves a double purpose. It acts as a guide to people implementing CurveZMQ and it can also be used by applications to do CurveZMQ encoding above ZeroMQ, to get end-to-end encryption. This is a little complex, and I'll make a real example app sometime to demonstrate it.

Helicopter View of CurveZMQ

To start a secure connection the client needs the server long term key. It then generates a short term key pair and sends a HELLO command to the server that contains its short term public key. The HELLO command is worthless to an attacker; it doesn't identify the client.

The server, when it gets a HELLO, generates its own short term key pair (one connection uses four keys in total), and encodes this new private key in a "cookie", which it sends back to the client as a WELCOME command. It also sends its short term public key, encrypted so only the client can read it. It then discards this short term key pair.

At this stage, the server hasn't stored any state for the client. It's generated a keypair, sent that back to the client in a way only the client can read, and thrown it away.

The client comes back with an INITIATE command that provides the server with its cookie back, and the client long term public key, encrypted so only the server can read it. As far as the client is concerned, the server is now authenticated, so it can also send back metadata in the command.

The server reads the INITIATE and can now authenticate the client long term key. It also unpacks the cookie and gets its short term key pair for the connection. As far as the server is now concerned, the client is now authenticated, so the server can send its metadata safely. Both sides can then send messages and commands.

This handshake provides a number of protections but mainly, perfect forward security (you cannot crack data encrypted using the short term keys even if you record it, and then later get access to the long term keys) and protection of client identity (the client long term key is not sent in clear-text).

Security Goals for CurveZMQ

Despite treating CurveZMQ as a disposable MPS, I want it to be solid against at least these attacks (this list is based on the CurveCP documentation):

• Eavesdropping, in which an attacker sitting between the client and server monitors the data. We send all in boxes that only the authentic recipient, knowing the necessary secret key, can open.

• Fraudulent data, in which an attacker creates packets that claim to come from the server or client. Only the authentic sender, knowing the necessary secret key, can create a valid box, and thus a valid packet.

• Altering data, in which an attacker sitting between client and server manipulates packets in specific ways. If a packet is modified in any way, the box it contains will not open, so the recipient knows there has been an attack and discards the packet.

• Replaying data, in which an attacker captures packets and re-sends them later to imitate a valid peer. We encrypt every box with a unique nonce. An attacker cannot replay any packet except a Hello packet, which has no impact on the server or client. Any other replayed packet will be discarded by its recipient.

• Amplification attacks, in which an attacker makes many small unauthenticated requests but uses a fake origin address so large responses are sent to that innocent party, overwhelming it. Only a Hello packets are unauthenticated, and they are padded to be larger than Cookie packets.

• Man-in-the-middle attacks, in which an attacker sitting between client and server substitutes its own keys so it can play "server" to the client and play "client" to the server, and thus be able to open all boxes. Since long-term keys are known beforehand, an attacker cannot successfully imitate a server, nor a client if the server does client authentication.

• Key theft attacks, in which an attacker records encrypted data, then seizes the secret keys at some later stage, perhaps by physically attacking the client or server computer. Client and server use connection keys that they throw away when they close the connection.

• Identifying the client, in which an attacker traces the identity of a client by extracting its public long-term key from packets. The client sends this only in an Initiate packet, protected by the connection keys.

• Denial-of-Service attacks, in which an attacker forces the server to perform expensive operations before authentication, thus exhausting the server. CurveCP uses high-speed high-security elliptic-curve cryptography so that a typical CPU can perform public-key operations faster than a typical Internet connection can ask for those operations. The server does not allocate memory until a client sends the Initiate packet.

Client Authentication

When we get a new connection into a server, we may have to check whether the client is allowed to connect or not. It's one of those problems that gets complex and messy quite quickly. Do we support password files, key files, databases, LDAP, PAM, GSSAPI, etc.?

As usual my favorite way to solve complex problems is to cheat and make this "someone else's problem". For authentication, what I wanted was a way to not implement any of the above list in the libzmq library, and still make it simple to support any of the above.

The answer in this case is to use ZeroMQ itself. The library, getting a new connection, sends a message requesting a yes/no decision, and someone gets the message, does the work, and sends a reply:

+----------+ | | | Client | | | +----------+ | | TCP connection V +----------+ | | | Server | | | +----------+ | REQ | +----------+ inproc:// | REP | +----------+ | | | Handler | | | +----------+

We can use the REQ-REP pattern over inproc://. If the process is built without any handler, the server won't authenticate clients (will allow them all). A handler can also proxy requests to out-of-process handlers, over local networking or ipc://.

I wrote this up as an RFC, called the ZeroMQ Authentication Protocol, or ZAP. Here's an example of a ZAP authentication request sent by a server to a handler, for a client using the PLAIN mechanism:

+-+ | | Empty delimiter frame +-+---+ | 1.0 | Version number +-----++ | 0001 | Request ID, for example "0001" +------+ | test | Domain, empty in this case +------+-------+ | 192.168.55.1 | Address +-------+------+ | PLAIN | Mechanism +-------+ | admin | Username +-------++ | secret | Password +--------+

This shows an example reply from the handler to the server:

+-+ | | Empty delimiter frame +-+---+ | 1.0 | Version number +-----++ | 0001 | Request ID echoed from request +-----++ | 200 | Status code +----++ | OK | Status text +----++ | joe | User id +-+---+ | | Metadata, empty +-+

It's a neat design because it works entirely out-of-band. That is, authentication doesn't affect message flow, can take as long as it has to, and can be handled by components that are fully independent of the application.

The ZeroMQ Security API

The security API is a set of new options that you apply to client or server sockets.

Here's how we configure a server socket for PLAIN security:

as_server = 1; zmq_setsockopt (server, ZMQ_PLAIN_SERVER, &as_server, sizeof (int));

And here's how we configure a client socket, which needs to know a username and password to send to the server:

strcpy (username, "admin"); strcpy (password, "password"); zmq_setsockopt (client, ZMQ_PLAIN_USERNAME, username, strlen (username)); zmq_setsockopt (client, ZMQ_PLAIN_PASSWORD, password, strlen (password));

Here's how we configure a server socket for CURVE security:

// Test keys from the zmq_curve man page char server_secret [] = "JTKVSB%%)wK0E.X)V>+}o?pNmC{O&4W4b!Ni{Lh6"; as_server = 1; zmq_setsockopt (server, ZMQ_CURVE_SERVER, &as_server, sizeof (int)); zmq_setsockopt (server, ZMQ_CURVE_SECRETKEY, server_secret, 40);

And here's how we configure a client socket, which needs to know three keys:

// Test keys from the zmq_curve man page char server_public [] = "rq:rM>}U?@Lns47E1%kR.o@n%FcmmsL/@{H8]yf7"; char client_public [] = "Yne@$w-vo<fVvi]a<NY6T1ed:M$fCG*[IaLV{hID"; char client_secret [] = "D:)Q[IlAW!ahhC2ac:9*A}h:p?([4%wOTJ%JR%cs"; zmq_setsockopt (client, ZMQ_CURVE_SERVERKEY, server_public, 40); zmq_setsockopt (client, ZMQ_CURVE_PUBLICKEY, client_public, 40); zmq_setsockopt (client, ZMQ_CURVE_SECRETKEY, client_secret, 40);

The application has to handle key exchange. For now the simplest model is that clients are given the server public key, and servers do not authenticate clients.

The Z85 Encoding Standard

The nice thing about standards is that you have so much choice. Wikipedia lists more than a dozen Base64 variations of dubious interoperability.

Why am I talking about Base64? Because exchanging binary keys is a royal pain. Almost as soon as we tried using CURVE security we hit the problem of how to represent keys in source code. For sure, you can encode binary as escape sequences but that's not portable between languages and doesn't work in configuration files, on the command line, or in emails.

Base64 is the common "standard" but it has several things about it that are nasty. First, there is no single standard so there's no real guarantee that Base64 implementations will work together. Second, it is a clumsy fit for binary data, and this is one reason there are so many variations. Base64 encodes 3 bytes of binary data as 4 characters. Yet most chunks of binary data are multiples of 4 or 8 bytes. That "divide by 3" aspect makes Base64 implementations more complex and fragile than such a simple problem deserves.

So I took an existing little-used encoding (Ascii85) and fixed it to make it safe to use in strings (so, also in XML, in JSON, on the command line, etc.) The result is Z85, which is "more compact than Base16, more reliable than Base64, and more usable than Ascii85".

I like Z85, because the code turned out to be simple and clean. That's always a good sign. The codec I wrote uses a precomputed table to rapidly convert from Base85 (Z85) to Base256 (binary).

Some people of course didn't like the introduction of Yet Another RFC, and they're right in some respects. I don't think it matters too much: I enjoyed making this little package; if it becomes a problem to others, someone will make a Base 64 encoding for the libzmq API.

The trick is when you do a setsockopt, if the key length is 32, it assumes binary. If the length is 40, it assumes Z85. If the key length is 44 (I think… or 43 perhaps?) you could assume Base 64.

Current Status

We got the security test cases running on June 21/22 at our ZeroMQ Developers' Meeting in Brussels, thanks to Ian Barber and Martin Hurton (who did the bulk of the work in libzmq). It's all on libzmq master today, and supported in the CZMQ binding. Other bindings will start to implement the security API options soon.

This all works, which is somewhat shocking.

It's also remarkably simple, which is also a pleasant surprise. We have, for applications:

• Three new options for PLAIN security.
• Five new options for CURVE security.
• One inproc:// plug-in endpoint for authentication handlers.
• One tool that generates new keypairs (in tools/curve_keygen).

And for those who want to get their hands dirty:

• Extensible classes in libzmq so anyone can try their hand at new mechanisms.
• A whole bunch of new RFCs along with reference implementations.

Conclusions

Of course no story is over until the main characters have been slaughtered in an orgy of bloodletting that leaves the audience shocked and traumatized and asking "why?" I'm sure CurveZMQ and ZAP and Z85 will get their Red Wedding in time. When it happens, I'd enjoy wielding the knife but it really doesn't matter as long as we make space for further interesting characters over time.

I'll be using the new security mechanisms in FileMQ, Zyre, and some other applications. There are a few other things still to fix in libzmq before this is ready for Internet use, and perhaps we'll print this as a new major version. I think it deserves it. But that's another story, for another day.

by pieterh

The Good, the Bad, and the Ugly

First, a confession: I'm an Ugly, writing my first distributed fabric in 1990. This was for a pan-European tour operator app, running on Digital ACMS, and it ran for 20 years. It was quite a shock to see formal messaging systems when we started building AMQP for JPMorganChase around 2004. I've come to appreciate the abstractions but it took a long time.

The Good, when they see ZeroMQ, don't come with preconceptions. They open a magic box, learn the different tools, and for the most part, experience the kind of euphoric empowerment we dream of when we make software for other people to use. The Good are the people I cherish the most, because their lack of previous experience keeps them open to doing the most exciting, aka crazy, things with ZeroMQ. They also often come from a generation that's used to GitHub, which enjoys making pull requests, and so on.

The Bad, when they see ZeroMQ, immediately try to fit it into their view of How Things Work. I've called this the "Fire vs. Electricity" paradox. When you're used to Fire, and you enjoy your grilled mammoth burger, it's kind of freaky to hear someone explaining electricity. The more experience you have with Fire, the more you will dislike this new form of energy. When someone explains, "Sure, you can make a grill, but you can also power your smartphone," the inclination is to hit the speaker over the head with the chewed-up remnants of a mammoth rib bone. The Bad tend to come to the zeromq-dev mailing list with long and unanswerable questions of the form, "How do I use ZeroMQ to do such-and-such?" which is like asking, "how do I use this thing you call 'electricity' to set fire to the plains so that the mammoth will run over the cliff?"

Finally, we have the Ugly. We uglies are experienced technical developers, wise enough to not trust technology, especially technology that is boastful and arrogant enough to claim some kind of uniqueness. We uglies know that most technology is bullshit, and the louder the marketing, the worse it is. We make a lot of stuff ourselves because we like things to work for 20 years. When we come to ZeroMQ, we start trying to fit the Guide to our problems, and soon we give up in a mix of frustration and anger.

Now I'm not calling Tavis Rudd "ugly", and indeed he looks quite handsome in his Twitter profile pic, but as he says, "finding the ZeroMQ guide a very frustrating read. Great software, but the guide is too chatty, too boastful and poorly structured. True root of my frustration w the zeromq guide was wanting to use req/rep for something I'm much better off doing with plain bsd sockets".

A good designer always takes the blame. As ZeroMQ gets more popular, we get a bigger spread of reactions to the learning curve. Early users were a subset of the Bad (let's call them the Evil), who had already figured out why they wanted decentralized messaging. For them, ZeroMQ was a godsend, and they would have read the source code to figure out how to use it. The Guide was therefore aimed squarely at the other early users, the Good, who were new to messaging and wanted a full story to buy into. Good and Evil, the ingredients of every great creation myth.

ZeroMQ Is Just Like BSD Sockets, But Better

The other essential ingredients of a creation myth are lies and deception. ZeroMQ is nothing at all like BSD sockets despite very insistent attempts from its early designers to make that. Yes, the API is vaguely socket-like. APIs are not the same as semantics. ZeroMQ patterns are weird and wonderful and delicate but they are not, and I'll repeat this, even marginally close to the BSD "stream of bytes from sender to recipient" pattern.

It's worse than that. The closest match in ZeroMQ is the PAIR socket, which is kind of broken in libzmq, and which we use only for inter-thread pipes. So innocent Uglies like Tavis start with request-reply sockets, and soon discover that not only is ZeroMQ not like BSD sockets "on steroids" but the very simplest ZeroMQ patterns are useless for them. Request-reply isn't just contrived and weird, it's also broken in interesting ways and is really just a teaching tool for people to get into the real ZeroMQ socket patterns: publish-subscribe, push-pull, and router-dealer.

Happily, sometime in 2012, one of the random passing geniuses who contributes to ZeroMQ came up with a patch that gave us raw STREAM sockets. It's about as sane as adding a lighter to your smartphone in an age where most people are just switching from coal to gas. The lighter is surprisingly useful when interfacing with primitive fire-based systems. It makes zero sense in an electricity-based world, but software is more like something from a steampunk comic, a mix of technologies of all ages.

When we explain ZeroMQ to the Good, we start at the beginning with the request-reply pattern and move through hundreds of pages to the real gold. I can really imagine how frustrating it must be to come with decades of experience of networking and have to wade through hundreds of pages of explanation to come to the right answer.

Of course, if everyone asked the right questions up-front, life would be simpler.

The question Tavis didn't really ask was (afaics), "how do I replace plain BSD sockets with ZeroMQ, gradually moving to more interesting use-cases?"

Which I'll try to answer, thanks to hshardeesi and his RAW patch. Let's assume we're talking about TCP, because UDP is special in its own cute ways.

Moving from BSD Sockets to ZeroMQ Sockets

Step One: STREAM sockets

First, we'll get a ZeroMQ socket talking to a classic TCP socket. Then we'll morph that into a ZeroMQ-to-ZeroMQ connection. Let's break this down into steps:

• Forget REQ, REP, and PAIR sockets, We're going to use a ZMQ_STREAM socket. I'll show you code you can cut/paste in a second.

• The stream socket acts as a server, so bind it to a tcp:// endpoint that you expect clients to connect to.

• You can talk to clients after they talk to you. Just like a TCP server socket. The RAW option doesn't let you talk to a TCP peer until it's talked to you.

• You read messages from the socket one after the other, either using blocking zmq_recv calls, or using zmq_poll. Each message has two frames. The first frame is the handle of the client (ZeroMQ calls this the "identity"). The second frame is the received data, which is 1 or more bytes.

• To send data back to a specific client, you first send the handle, and then the data, as two frames. If you want to close a client connection, send the handle and then an empty frame.

That's it. The stream socket takes care of all the multiplexing, error handling, polling, and so on, and does this in a background I/O thread so it's fast.

What kind of havoc can we create using a stream socket? How about a web server? That's always fun. OK, here's a 30-line HTTP server in C. This little animal doesn't do much real work. It reads requests, logs them, chucks them out, and answers "Hello, World!":

Get the code.

To make this into a real server we'd have to add at least a few things:

• Read a full HTTP request header, which means keep reading and accumulating data until we find a blank line (two carriage-returns) in the data.

• Parse the HTTP request to see if the client has asked us for something we can handle, e.g. to GET a resource.

• Map that resource name into a real filename (if that's what we're trying to do).

• Open the file, and send its contents to the client.

But that's classic RFC2616 stuff and not relevant to ZeroMQ, so not really worth talking about more. 2616ing used to be a full-time hobby but honestly, ZeroMQ is way more fun.

Step Two: ROUTER to DEALER

So, you've now seen how ZeroMQ can talk to BSD sockets. It's not perfect: for one thing, we can't write clients like this. That sucks. If anyone wants to fix this, perhaps allow outgoing connections on a STREAM socket, resulting in a null message back to the caller, with an identity and zero body.

Anyhow, what's the next step? Let's replace the BSD TCP socket with a DEALER socket. It's not a perfect match but it's close enough to make a good party. If we connect DEALER to exactly one ROUTER, the DEALER just reads and writes like a TCP socket, with some small but oh-so-valuable differences:

• We can write any amount of data as a single message and it will arrive as a single message. This is awesome because using a classic TCP protocol like HTTP means doing a lot of dancing around the pot to figure out when a request or response is finished. Framing, young man, the future is in framing!

• The DEALER socket will connect opportunistically. Again, a small but awesome improvement on the BSD style of networking. Just connect one time, then use the socket for ages. It'll disconnect and reconnect as the network shifts. Perhaps the ROUTER socket will decide it wants to flush connections. No problem! The DEALER will reconnect later.

DEALER does quite a lot more than this. It can connect to many ROUTERs and will distribute requests among them. Load-balancing, for free. It does fair queuing on input, so if you're getting a gigabyte of messages from one ROUTER, and another sends you a 10-byte urgent message, that will sneak in front.

Doing a DEALER-to-ROUTER pseudo-HTTP is of course weird, it's just an exercise. We'll call the protocol WTFP, and it looks (nothing at all) like HTTP over ZeroMQ framing:

Get the code.

Satori

So now we've seen how ZeroMQ can roughly imitate BSD sockets, talk to them, and then fark them up beyond all recognition. DEALER-ROUTER is what we use in most real large-scale architectures, because it lets us build real bidirectional protocols. But there's also PUSH-PULL and PUB-SUB, which are fun. Not to mention using ZeroMQ for in-process multithreading. When you think of distribution not just as "outer" but also "inner", you start to see just how ambitious and arrogant this library is.

Hope you enjoyed the article. Let me know what you want me to write about next.

by pieterh

Real Protocols are Worth It

ZeroMQ used to have a sweet but unrealistic story: "our API looks like plain sockets, and we have practically no protocol". As it turns out, this was great marketing but not great technical design. The plain BSD socket API doesn't work for messaging (Contexts? Multipart messages? Zero-copy? Events? Threads?), and a non-protocol doesn't provide the number one guarantee users want, which is interoperability (you need version numbers).

ZMTP v1.0 was simple: each peer connected and then started blasting messages to each other. This is the protocol that libzmq v2.x speaks. The later release, v3.1 speaks a dialect of this protocol where subscribers send subscriptions upstream to publishers. You can't mix v2.x and v3.1 pub-sub on the same network.

We designed and implemented ZMTP v2.0 about a year ago. It added version numbering, and fixed the framing to be more sensible. This is the protocol the current libzmq v3.2 speaks (it will also downgrade to speak to older 2.x peers). It was a minimal change, and didn't address a bunch of other problems in ZMTP.

By 2013 it was clear that the number one problem with ZMTP is the lack of security. As I've shown in previous articles, it's not technically so difficult. It just needs to be done. At the same time, since adding security means big changes, we (Martin Hurton and myself) decided to address a few of the other long-standing problems in the protocol. These are mostly just questions the protocol should answer, but didn't.

As you read these changes, understand the "Cheap and Nasty" pattern. We connect once, and then exchange millions of messages over hours or days. The message transfer is asynchronous, changes very little over time, and has to be as efficient as possible. But creating a connection is synchronous, changes a lot over time, and efficiency is utterly irrelevant.

When you get this, you begin to see how to design what I'd consider a real messaging protocol. It breaks into two parts: a chatty negotiation at the start, and a high-speed asynchronous data stream thereafter. The negotiation needs to define commands with structure, and extensibility in the form of dictionaries (binary, or text, it doesn't really matter). ZMTP v2.0 didn't even start to go into this direction. In fact it broke that pattern by for instance sending meta data (identities and subscriptions) as messages.

Goals

A good way to understand ZMTP is to start with a plain TCP connection and then see what problems we need to solve in order to get a working message transport:

• TCP carries a stream of bytes with no delimiters, but we want to send and receive discrete messages. Thus, ZMTP reads and writes frames consisting of a size and a body.

• We need to carry metadata on each frame (such as, whether the frame is part of a multipart message, or not). ZMTP provides a flags field in each frame for metadata.

• We need to be able to talk to older implementations, so that our framing can evolve without breaking existing implementations. ZMTP defines a greeting that announces the implemented version number, and specifies a method for version negotiation.

• We need security so that peers can be sure of the identity of the peers they talk to, and so that messages cannot be tampered with, nor inspected, by third parties. ZMTP defines a security handshake that allows peers to create a secure connection.

• We need a range of security protocols, from clear text (no security, but fast) to fully authenticated and encrypted (secure, but slow). Further, we need the freedom to add new security protocols over time. ZMTP defines a way for peers to agree on an extensible security mechanism.

• We need a way to carry metadata about the connection, after the security handshake. ZMTP defines a standard set of metadata properties (socket type, identity, etc.) that peers exchange after the security mechanism.

• We want to allow multiple tasks to share a single external unique interface and port, to decrease system administration costs. ZMTP defines a resource metadata property that lets any number of tasks share a single interface and port.

Overall Behavior

A ZMTP connection goes through these main stages:

• The two peers agree on the version and security mechanism of the connection by sending each other data and either continuing the discussion, or closing the connection.
• The two peers handshake the security mechanism by exchanging zero or more commands. If the security handshake is successful, the peers continue the discussion, otherwise one or both peers closes the connection.
• Each peer then sends the other metadata about the connection as a final command. The peers may check the metadata and each peer decides either to continue, or to close the connection.
• Each peer is then able to send the other messages. Either peer may at any moment close the connection.

Version Negotiation

ZMTP v3.0 makes it much clearer how to do version negotiation. Each peer sends part of the greeting, and the higher version number can decide to downgrade to the lower version number. Thus implementations can choose to be backwards compatible (as libzmq is), but do not need to try to be forwards compatible (which is kind of hard). If you're a v3.0 peer and you get a greeting from a v4.0 peer, you just continue. If the 4.0 peer doesn't want to downgrade to your protocol, it'll close the connection.

Commands, Messages and Frames

Following the greeting, which has a fixed size of 64 octets, all further data is sent as size-specified frames. Command frames are used in the security handshake, and message frames are used to carry clear text application data. Commands always consist of a single frame. Messages consist of one or more frames.

A ZMTP connection is either clear text, or encrypted, according to the security mechanism. On a clear text connection, message data is always sent as message frames, using a single consistent format, so that traffic analysis tools don't need to know the specific details of a clear text mechanism. On an encrypted connection, message data is always encoded as commands so that wire analysis is not possible. (You'll never see message frames on a secure connection, period.)

The frame has the same format as in v2.0 (like I said, the asynchronous high-volume part of a protocol rarely changes once you get it right). It has a 1 byte flags field, followed by one or eight bytes of size, followed by body data. The only change we made in v3.0 was to add another flag bit to mark command frames.

People have proposed various other flag fields in the past, e.g. marking the first and last frame of a message rather than "more". Or, adding a "label" bit to mark message envelopes separately from message content (this would get rid of the null delimiter frame that request-reply still uses).

Right now, ZMTP still doesn't know much about actual socket semantics, but we could start to add this. I'd like, for example, to be more explicit about XSUB/XPUB subscribe and unsubscribe commands, which are now sent as messages with a special first byte. We could also do heartbeating and more. Actually it's not hard to add features. The real cost is not even implementing them, it's ensuring existing applications can migrate safely onto new stacks.

Authentication and Confidentiality

Now we come to the meat of ZMTP v3.0, which is security. More specifically, do we know who we're talking to, and are our conversations fully private?

Making secure protocols is complex, dangerous, and thankless work unless you find a way to cheat and make it all Someone Else's Problem. For ZMTP I borrowed from the IETF's Simple Authentication and Security Layer (SASL), aka RFC 4422, and the NaCl CurveCP security handshake. (I've explained my proposal for creative abuse of CurveCP in previous articles.)

If you haven't studied SASL, and you care even a little about security protocols, you should go off and read my latest book, called "SASL Made Simple" (1,350 pages, from iMatix University Press, $95.50 plus S&H). Just kidding… we can boil SASL down to a few key design elements:

• The server proposes, and the client accepts, a "security mechanism", which is a string.
• The implementations use the security mechanism to lookup a library that implements it.
• The server and client exchange challenges and responses, which are binary blobs.
• These blobs are passed to the security library, which digests them and possibly produces a new challenge or response.
• Eventually the security library at each side says, "OK, ready", and the server and client can now communicate.

It's a little more subtle than this, but that's the basic idea. After the two peers shake hands, they can send each other messages encrypted by the security library. Message I/O is still handled by the same code, with the security library acting as a filter (it encodes and decodes messages as needed).

What SASL does is pretty magic. First, we can add security mechanisms arbitrarily. The mechanism I want to use for ZMTP is CurveZMQ, but perhaps someone else will make a DTLS mechanism. Remember, each mechanism is a plug in library that doesn't need to know how data gets passed around. It takes blobs, produces blobs, and holds state about the connection. In my previous article I showed how a such a filter works.

Splitting the security work off makes it much easier to solve. We can test the mechanisms in isolation, before we plug them into real messaging. It also leaves the protocol totally naive about security. That means that as we switch to newer, faster security models, we don't have to change the protocol.

A peer announces precisely one security mechanism, unlike SASL, which lets servers announce multiple security mechanisms. Security in ZMTP is assertive in that all peers on a given socket have the same, required level of security. This prevents downgrade attacks and simplifies implementations.

When we use this approach, every connection always uses a security mechanism. Of course for most traffic will still be clear text. So ZMTP comes with three mechanisms: NULL (no security), PLAIN (clear text authentication) and CURVE (full security). PLAIN is there for two reasons. One, it provides a decent model for other mechanisms. I'll explain that model in a tick. Two, clear text authentication is pretty useful on internal networks. You don't want that test client connecting to the production server and sending 1M test messages by accident.

The NULL Security Mechanism

The NULL mechanism is easiest to understand - it gives us pretty much what we already have with ZMTP v2.0. The big difference is that there's an explicit READY command to send metadata (what you'd call "headers" in HTTP). Metadata is one of those things you really, really want to send in an extensible dictionary, since this is where 80% of the experimentation and profit comes from in protocols.

In the NULL mechanism, each peer sends off a READY to the other, and can then send or receive messages. They can check each others' metadata, e.g. to validate socket types. If either peer doesn't want to continue it simply closes the connection.

There's something about designing protocols to work like real human interactions that pleases me, and feels right. So there's no error reporting in ZMTP. If the peer closes the connection during handshaking, that's a strong rejection (take the hint, and don't connect again!), whereas if the peer closes the connection later, that's a temporary blip (sorry, got distracted, please connect again).

Here's the grammar for the NULL mechanism:

null = ready *message ready = command-size "READY " *property property = name value name = OCTET *name-char name-char = ALPHA | DIGIT | "-" | "_" | "." | "+" value = 4OCTET *OCTET ; Size in network byte order

Commands always start with a fixed-space 8 character name, and then descend into whatever form of madness makes the command work. For READY, we are sending a metadata dictionary. I decided to go with short text names and long binary values. 2GB should be enough for anyone. This format seems easy to create or parse in any language.

The PLAIN Mechanism

I'll explain how ZMTP's security works, because if we're going to use this for real work, we need criticism. I've some ideas for encouraging people to try to break the CURVE mechanism, but that's for later.

Since I've not finished documenting CURVE, we'll look at PLAIN. These two mechanisms have RFCs already, and you'll see that they use the precise same command grammar:

C:hello S:welcome C:initiate S:ready *( C:message | S:message )

This was deliberate. The handshake is synchronous and consists of two requests from the client and two responses from the server. For PLAIN, we could collapse this to one request and one reply. For CURVE it has to be split up, since the initiate-ready step depends on security already established by hello-welcome. (Note that in CurveCP, which is the original design for the CURVE mechanism, the command is called COOKIE, not WELCOME).

So conceptually, we have three steps:

• Establish security (HELLO, WELCOME).
• Send meta data (INITIATE, READY).
• Send messages.

While the first four commands are synchronous, messages are async. That is, the client can start sending messages as soon as it's received READY, and the server can send messages right after it's sent READY. INITIATE and READY carry metadata, like the NULL mechanism's READY command, and using the same dictionary format.

The CURVE Mechanism

Roughly, the CURVE mechanism works like PLAIN except that it exchanges encryption keys. CURVE in fact implements CurveZMQ, which is a generic security handshake I'm designing that is taken from CurveCP, a UDP-based security and traffic control protocol designed by cryptographer and mathematician D. J. Bernstein. CurveCP is based on NaCl, the security library from the same djb.

The security handshake in CurveCP is elegant and relatively simple (compared to other security designs), and comes with a very good pedigree. Whether it really is as robust as djb claims we won't know for a while. However as well as security, CurveCP aims to replace TCP entirely, doing its own traffic control over UDP.

To be honest, I think CurveCP (over UDP) would make a great mechanism in itself for ZMTP but that's a serious challenge. I've tried to keep ZMTP semantically compatible with that idea. In the more pragmatic short term, it turns out that stripping down CurveCP to just the security aspects, assuming TCP is going to be here for some time yet, is feasible.

This is CurveZMQ: the CurveCP handshake taken out and stripped down a little. There's a website, curvezmq.org, but it's still empty. My plan is to take material from curvecp.org, and remix that into a single clean RFC.

It turns out that it's very useful to do end-to-end security on top of 0MQ, as an alternative to doing security in the library. It means you can route over untrusted peers. Imagine I want to make a secure chat application. Two phones, talking to a server in the middle. We don't want the server to ever get clear text data. So the phones do end-to-end security. If we use security in ZMTP, that won't work.

End-to-end security only works with a ROUTER-DEALER combo (or another mix of sockets that can imitate it), since it needs that chatty request-reply handshake. But this is how we build most large-scale services anyhow.

Meaning, at some point I'll write a CurveZMQ mechanism in CZMQ, and show how to use this in a protocol like http://rfc.zeromq.org/spec:19 rfc.zeromq.org/spec:19/FILEMQ or rfc.zeromq.org/spec:20/ZRE.

Pluggable Authentication

A protocol shouldn't have an opinion on implementations except when it affects how they work together. But as ZeroMQ users we should be asking, "how does my application plug into this security model?"

The answer is a little subtle. Clearly as peers come and go, we have to invoke code that authenticates them. ZeroMQ can't do this itself in a realistic way. For instance even the PLAIN mechanism could do a remote username / password lookup to some other server.

Our idea is that libzmq provides an inproc socket for authentication and other events. Presumably this would happen on a per-context basis. You then write your authentication logic as a classic ZeroMQ task: connect to a socket, get the events, process them, and send back OK / NOT-OK replies. Meanwhile the "real" application logic continues to run ignorant of the ongoing authentication. It just gets messages and sends messages like today.

Connection Metadata

Adding a dictionary to the security handshake gives us extensible metadata. Right off, this let me move the socket type and identity out of limbo into a safer home. It also provides a place for more information about the connection.

There's really only one reason to put information upfront in the greeting, and that is when we absolutely need it for security negotiation. You'll see one flag ("as-server") that tells the mechanism whether the peer is acting as client or server. It means we don't need to rely on the bind/connect order. But all other metadata sits in this dictionary.

As well as socket type and identity, I defined "resource", which lets us do virtual addressing.

Virtual Addressing

One of the missing features in ZMTP was virtual addresses. Physical addresses (TCP endpoints) can be costly in some architectures, e.g. when crossing a firewall. If the protocol provides a place for a peer to request a virtual address, during connection negotiation, we can quite easily allow multiple tasks to bind to the same TCP endpoint.

In ZMTP we use the concept of resource, because it fits the schema we already use for zmq_bind and zmq_connect endpoints. Here's how a peer makes a resource available by binding to it:

zmq_bind (publisher, "tcp://*:9001/my/resource/name");

Here's how a peer connects to a specific resource on another peer:

zmq_connect (subscriber, "tcp://192.168.55.23:9001/my/resource/name");

The implementation will only create one listening TCP socket, but will keep a list of resources available for that listening socket. When a peer connects, the resource it specifies is then used to route messages to the right task.

In ZMTP the resource name comes after the security handshake, so it's encrypted and tamper-proof when we use secure mechanisms (I'll come to mechanisms in a second). Putting the resource name after security does have some implications. It means we cannot use the resource to select credentials. In effect, all connections on a physical endpoint must share the same credentials, e.g. the same server keys.

This isn't really a limitation since TCP in any case does not allow multiple listeners on a port. It's quite plausible to make a mulitiprocess design in which arbitrary processes on a box share a single physical endpoint, by each specifying a different resource. However, there is going to be a single process that does all the zmq_bind commands, acting as proxy for the other processes. And it's sensible that one process has one set of security credentials.

Security Considerations

I'll collect possible attacks and strategies to defend against them (if you have more suggestions, let's hear them!) Even attacks we can't defend against are worth documenting. So ZMTP makes some recommendations to implementors:

• Attackers may overwhelm a peer by repeated connection attempts. Thus, a peer MAY log failed accesses, and MAY detect and block repeated failed connections from specific originating IP addresses.

• Attackers may try to cause memory exhaustion in a peer by holding open connections. Thus, a peer MAY allocate memory to a connection only after the security handshake is complete, and MAY limit the number and cost of in-progress handshakes.

• Attackers may try to make small requests that generate large responses back to faked originating addresses (the real target of the attack). This is called an "amplification attack". Thus, peers MAY limit the number of in-progress handshakes per originating IP address.

• Attackers may try to uncover vulnerable versions of an implementation from its metadata. Thus, ZMTP sends metadata after security handshaking.

• Attackers can use the size of messages (even without breaking encryption) to collect information about the meaning. Thus, on an encrypted connection, message data SHOULD be padded to a randomized minimum size.

• Attackers can use the simple presence or absence of traffic to collect information about the peer ("Person X has come on line"). Thus, on an encrypted connection, the peer SHOULD send random garbage data ("noise") when there is no other traffic. To fully mask traffic, a peer MAY throttle messages so there is no visible difference between noise and real data.

Conclusions

We're still some way off from a secure ZeroMQ, but we're seeing the mountain clearly now. The next step is to make simple reference implementations of the core protocol and each mechanism. Then, we can implement the new protocol in libzmq, with some minimal authentication hooks. One step at a time…

So what do you think? Have you tried NaCl or libsodium, and did you like it? Are you more of a DTLS guy, and if so, what do you need to start designing your plugin mechanism for ZMTP/3.0? Are you one of those crazy geniuses who has reimplemented ZMTP/2.0 in Java, C#, Erlang, or Bash? How does this new protocol feel to you? Come and discuss on zeromq-dev.

by pieterh

The Census Pattern

The Census Pattern is an evolution of Meerkat that solves the problem of monitoring or querying a set of nodes. The question part looks like a PUB-SUB broadcast and the answer part is similar to a PUSH-PULL sink in that it's a many-to-one flow. You can often simply hard-code the question so it's implied. For instance instead of asking a set of nodes, "who is doing the least work right now?" (to know who to send new work to), nodes could broadcast their availability without being asked.

The advantage of explicitly asking the question is, of course, that you can do more sophisticated group queries. For instance, you could send a search pattern to a set of engines, and get back the first reply or all replies. So Census is like group request-reply, where the replies can be processed according to the use-case.

Traditionally, we'd design Census using a PUB-SUB pattern for the question, and a PUSH-PULL pattern for the replies. In the old days this the advice we gave ØMQ developers: break a complex pattern into separate flows, use a specific socket type for each one.

However it turns out that multiple sets of sockets gets complex and confusing. It's hard to synchronize things when you have multiple independent paths. It's hard to imagine a security model working over mixed socket flows. Using a single socket pair is, it turns out, simpler. And simpler is always better. So that's what we'll aim for: a single socket flow for group request-reply.

Don't confuse "censuses" (if that is a proper word) with "surveys". A census is what you do when you ask your kids, "who wants burgers?" A survey is what you do when you stop random people at the 11th Annual Vegan Life Festival and ask innocently, "who wants burgers?" (The answer might astonish you. Or not. I've never tried, but YouTubing ridiculous surveys conducted on self-selected samples suddenly seems quite fun. Next up, we survey the Papal convention for their opinions on whether burger makes a valid pizza topping, and then we extrapolate their answers to the whole world population.)

Popes aside, here are some ground rules for conducting a census:

• We generally know in advance how many people we're speaking to. People may come and go randomly but unlike a survey, our goal is to count the heads we can see, not the ones we can't.

• We expect either an answer, or silence, within a certain time. On a computer network as in the real world, there's no such thing as perfection.

• Based on the results, we take some decision, which may be to repeat the question, ask a different question, to jump to a bogus forgone conclusion, etc.

So parameters for a census are: the question we ask, the allowed answers, the time we allow for answers, and the total size of our target population. That last piece is, of course, where Meerkat comes in since it lets us conduct a census of a dynamic population that may be leaving and joining the network randomly.

Census Over XPUB-XSUB

Clearly the PUB-SUB pattern is a good place to start with a census since we want to broadcast the question to everyone. We do have to use XPUB and XSUB since they give us extra superpowers we need. Two superpowers, in fact:

• We can send and receive subscription messages, which lets us do the Meerkat part.

• XSUB sockets can talk back to XPUB sockets, which lets us do the Census part.

John Muehlhausen's libzmq d32e39 commit from 7 January 2013 is worth studying. If I taught computer science, I'd spend a few hours just boring my poor students with this one. His 14-line patch hacked XPUB and XSUB into becoming a two-way highway for messages. It's so simple and so very beautiful. I like the evolution of PUB and SUB from a one-way purist design (from the original one-way PGM-style financial data distribution) into more tactile and pragmatic two-way sockets.

The patch is a minimal change, evolution in action. A subscription message starts with 0x01. An unsubscription message starts with 0x00. We discard any other message, right? That's how XPUB and XSUB were designed. But hang on, no, hang on… Let's not throw other messages away, let's pass them on up instead. Suddenly the gate opens to subscribers sending stuff back to their publishers.

So, literally (and I mean this in the non-figurative sense), you can send a "normal" message to an XSUB socket, and you can receive "normal" messages off XPUB sockets. XPUB works like a sink, and will fair-queue incoming messages from its subscribers, while XSUB works like a copier, and will copy the message to each XPUB it's connected to. You don't need any special socket options for this, it's how the sockets work out of the box. If you don't want the functionality, just discard "normal" messages in code that reads from XPUB sockets.

Figure 1 - The Census Pattern

Enough ado already, let's take this pie out of the oven and see how it tastes. Here's my first attempt at the Census pattern:

It's not huge, 150 lines of code in C. The counter task has two steps. If all nodes are up and running, the whole thing runs with no delay. If there's a node broken or too busy to answer, each step will wait for a while before timing out. A census will still be valid if there are missing nodes. In a realistic implementation we could continue to ask more questions, and not need to wait for nodes to join again each time.

Census Over ROUTER-DEALER

In the Big Black Book of Mad Science (originally it was a kind of light green but multiple incendiary near-misses gave it a charred exterior, and the general consensus was that "The Big Kind of Light Green Book" didn't have quite the same ring anyhow), rule number three is: "If at first you succeed, try again with more gunpowder". (Rules one and two are, since you're asking, "Bald is the new Evil", and "If you laugh loudly, you can get away with almost anything". Protective eye-wear only merits rule number 24. Rule number five says, enigmatically, "If you have to ask what rule #5 is, you can't afford it.")

So, more gunpowder! XPUB-XSUB is neat, but how about ROUTER-DEALER? After all, the Census pattern requires a two-way many-to-one dialog, and ROUTER-DEALER are pretty good at that. As an experiment, let's rewrite the same Census example using ROUTER-DEALER, and see whether it's simpler or more complex.

Figure 2 - Census with More Gunpowder

Here's the code:

Quite shockingly, this worked first time and is only two-thirds the size of the XPUB-XSUB version. The code is quite a lot simpler. This teaches us a few things:

• ROUTER-DEALER is a powerful tool.

• Asynchronous beats synchronous. The main cost in the XPUB-XSUB model is the getting everyone to pay attention and listen to the question at the same time. This doubles the size of the publisher task and doubles the worst-case time for the census.

• Always try again, with more gunpowder.

The next and final article in this little series of PUB-SUB Hacks will cover the Repeater pattern, which is an "any-to-all pub-sub for Very Large Networks" pattern.

by pieterh

The Meerkat Pattern

One of the common questions from ØMQ developers (until they learn that there's no good answer and retreat into a kind of Stockholmesque "if it hurts it must be for our own good" acceptance) is "how can a publisher know when subscribers are present?" However as I've written elsewhere, pain is not a good sign.

It is true that for certain use cases, the answer is, "you don't want to know". It's true that for very large scale publishers, the raw cost of tracking individual subscribers in real time is not worth the effort. The BBC doesn't wait until everyone's ready with their telly before starting Blue Peter. So, the argument goes, why should we?

But I'm suspicious of people who tell me what I should think. Particularly since, as a writer, propaganda is my job. And I know just how much rubbish people say when they're being especially serious. But kidding aside, the question is valid, and the "no, you don't want to do this" answer seems bogus and somewhat lazy. We know that publishers can benefit greatly from tracking individual subscribers. This was after all the major improvement in version 3 of ØMQ: much faster PUB-SUB thanks to "upstreaming", where subscribers actually tell publishers what data they want.

If it's good for XPUB and XSUB sockets, why not for applications too? This was the question that innocently entered my mind one spring morning in Brussels, as we watched yet another snowstorm unfold. Innocence is all relative, of course, as they say in Rome. The question hooked into some unfinished thoughts and formed a flash mob of irritation that forced me to stop working briefly on more interesting topics. The Meerkat Pattern is what came out.

This is one of the real pleasures of using a technology: to twist and distort it into shapes its original designers never thought of, and see what comes out. So let's distort and abuse the XPUB-XSUB upstreaming functionality to answer that question and give publishers a way to know what subscribers are around.

I once had a friend and mentor, the larger-than-life Leif Svalgaard, who would work secretly on some amazing code and then show the finished results to general astonishment. When we asked him how he did it, he invariably answered with a huge Santa Clause grin, palms up innocently, saying, "it's magic!" Once Leif wrote a tiny but working multitasking system for DOS over the weekend, just to disprove Microsoft's claims that OS/2 (yes, this was some time ago) needed 8 Megabytes of memory "because it did multitasking". As he started a compilation in one window, and edited a file in another, I asked him, "how did you do this?" "Ah, it's magic!" he answered.

Incidentally, that tiny multitasking system (1,000 lines of x86 assembly) inspired my multithreading SMT kernel in 1995, which became the heart of our server technology and then OpenAMQ, which led the way to ZeroMQ. So it goes in the history of software, a chain of inspiration and culture leading back to the first time primitive man told his wife, "just going down to the cave to paint some pr0… ugh… hunting scenes, honey. BRB!"

Figure 1 - The Meerkat Pattern

Step 1: Counting Subscriptions

What if a publisher (suitably wearing an "X" on its torso to mark it with XPUB superpowers) simply counted the number of subscriptions it received? Perhaps surprisingly, this works for all values of N on the condition that N is equal to one. To say this in English, an XPUB tracks subscriptions and only tells you of the first one. So, on to attempt 2.

Step 2: Using the Verbose Option

A few months ago we added a neat little option (ZMQ_XPUB_VERBOSE) to XPUB sockets that disables its filtering of duplicate subscriptions. This now works for any number of subscribers. We use this as follows:

However we still have a problem. Subscribers may subscribe any number of times. We need some devious trick to distinguish different kinds of subscriptions.

Step 3: Sending 'Special' Subscriptions

My first idea was to use the empty subscription, but of course this doesn't play well with real use. Let's say I have two subscribers that subscribe to a series of topics:

• A subscribes to "USD", "JPY", and "EUR"
• B subscribes to "JPY" and "KRW"

How do I know how many subscribers I have?

My answer, which is not very beautiful but works, is to send a "special" subscription that doesn't match any real data but which tells my application what it needs to know:

• A subscribes to "USD", "JPY", "EUR", and "Meerkat"
• B subscribes to "JPY", "KRW", and "Meerkat"

So this brings us to an example of Meerkat in action:

Get the code.

When you run this code you'll see it starts a random number of subscribers, waits until they all wake up, and then sends "Hello, World" to everyone.

Step 4: Ridiculous Optimization

If you look a little closer at the code you'll see a possible optimization. Instead of sending a subscription message to signal presence, how about an unsubscription message?

It's a great idea except (at time of writing), XPUB sockets don't verbosely send you unsubscribe messages. There are reasons for this (there are always reasons, he said darkly, to no-one in particular) but it's minor. Perfection is, sometimes, just a waste of time, and sending subscribe messages seems fine.

Step 5: Robustness

The Meerkat example will occasionally fail in the real world because it doesn't distinguish between a subscriber saying "I'm here" for the first time from one who comes back several times. We're basically counting the heads that pop-up, not the bodies that are really present on the field. The same head can pop-up over and over, and like idiots we'll count it multiple times.

There's an answer—there always is—but it makes things more complex. Each subscriber has to identify itself uniquely and the server has to track this. Subscribers could for instance send a UUID as part of their "special" subscription. The server would track these in a hash table, and count only new entries.

It's the kind of boring grunt work we invented ØMQ to get away from, so I'm not going to feed you an example. Let's just agree that making things more complex is easy. My next article will cover the Census pattern, which extends Meerkat into a group query pattern.

by pieterh

Overview of CurveZMQ

The CurveZMQ code uses the CLASS conventions which make the C code look like something nicer. Here's the self-test code, which sends "Hello" and expects "World" back. CurveZMQ uses a single module, zcurve:

zmq_msg_t *msg; // Start server peer zcurve_t *server = zcurve_new (); msg = zcurve_start (server, NULL); assert (msg == NULL); // Start client peer and ping/pong through the handshake zcurve_t *client = zcurve_new (); msg = zcurve_start (client, zcurve_public_key (server)); while (msg) { // Send encoded message from client to server // ... // Receive encoded message in server msg = zcurve_accept (server, msg); if (msg) // Send encoded message from server to client // ... // Receive encoded message in client msg = zcurve_accept (client, msg); } msg = zcurve_encode (client, (byte *) "Hello", 5); assert (msg); // Send encoded message over the wire // ... // Receive encoded message from the wire msg = zcurve_accept (server, msg); assert (msg); assert (memcmp (zmq_msg_data (msg), "Hello", 5) == 0); msg = zcurve_encode (server, (byte *) "World", 5); assert (msg); // Send encoded message over the wire // ... // Receive encoded message from the wire msg = zcurve_accept (client, msg); assert (msg); assert (memcmp (zmq_msg_data (msg), "World", 5) == 0); zcurve_destroy (&server); zcurve_destroy (&client);

We work with two zcurve instances; one acts as client and one as server. In practice you'd create a zcurve instance for each 1-to-1 connection. The security protocol, which I'll explain, assumes that the underlying transport is TCP. There's an easy tweak to make this work over disconnected protocols (PGM, UDP, or even non-network protocols like shared disk). But since that isn't our use case, I'll explain the tweak and then forget about it.

The zcurve API reminds us a little of SASL, with its "I'll consume and produce opaque blobs until I'm satisfied". The blobs in our case are 0MQ messages. In the test code above we don't send them anywhere, we simply pass them between two zcurve instances until the handshaking is done. There are three critical methods in the API:

• zcurve_start () - which starts the state machine and may issue a challenge command.
• zcurve_accept () - which accepts a command and may issue a response.
• zcurve_encode () - which turns a plain text payload into an encoded blob.

In fact the API still needs work: right now the accept method's response message does double duty. If we're still handshaking, it has to go back to the peer. If we're finished hand-shaking, it's a decoded payload. I'll fix this in the next prototype.

But one step at a time. Don't make stuff you don't need. This brings me to the CurveZMQ protocol, which is derived from the CurveCP handshake and uses the same concepts like cookies, nonces, and vouches, but is simpler. The original CurveCP handshake is designed for UDP and includes a variety of non-security related features like virtual hosts. Trimming off this and other unnecessary fields gives us a minimal Curve-derived protocol.

The zcurve module uses a state machine to decide what's valid and what's not. In this prototype, anything invalid causes an assertion failure. It is not meant to be production code, yet. In practice invalid messages will cause the connection to be dropped.

Building and Testing

To build this you first want to install and build libsodium. Use the standard commands:

./configure make && make check && make install

You also want libzmq and CZMQ. Then take the CurveZMQ code from the gist and compile and link zcurve_test:

git clone https://gist.github.com/5224958.git CurveZMQ cd CurveZMQ gcc -g -o zcurve_test zcurve_test.c zcurve.c -lczmq -lzmq -lsodium

And run the test program:

$ ./zcurve_test Running self tests... * zcurve: C:HELLO: OK S:COOKIE: OK C:INITIATE: OK -:MESSAGE: OK -:MESSAGE: OK Tests passed OK

Security Models

CurveZMQ allows three possible security models:

• Anonymous clients, known server: where the server key is known to clients, but the server does not validate client keys.
• Pre-shared clients, known server: where the server key is known to clients, and all clients share a single key.
• Fully authenticated: where the full set of allowed clients is known to the server.

We don't yet have a hook in zcurve to authenticate a client; it would be trivial to add, e.g. using a callback that checks a given client long-term public key.

Differences between CurveZMQ and the CurveCP Handshake

To understand this section you should at least briefly read the CurveCP website and protocol description.

While CurveCP runs over UDP and uses a number of techniques to harden the server (such as not allocating memory until the client has successfully echoed a cookie), CurveZMQ runs over a connected transport (TCP or IPC) and cannot use, and does not need all the functionality of CurveCP:

• We do not use a minute key for cookies; instead, each connection creates a unique cookie key which is valid until the client sends an INITIATE command. The server could also check that INITIATE comes within 60 seconds of HELLO.

• We do not send the client short-term key in MESSAGE commands. This would be useful over disconnected transports but for our use case, TCP and IPC, it's pointless. If we added that, we'd add it both for client and server MESSAGE commands, so that clients could work with multiple servers.

• For simplicity, INITIATE does not contain message data.

• Thus, server and client MESSAGE commands have the same structure and are the only carriers for payloads.

• We don't have a hostname in the INITIATE command.

And we don't use any of the CurveCP traffic control protocol fields. CurveZMQ assumes a reliable connected transport.

The current CurveZMQ prototype still uses a fixed message size, maximum of 100 bytes. The next iteration will need variable message encoding. There are a number of other simplifications that are more or less safe:

• Long term server nonces are not segregated but we'd want to do this in practice: start the server nonce at a random number and then increment by 1.

• Short term nonces are always generated from random numbers. The chance of a collision is low enough that this is robust for production use.

CurveZMQ Internals

Here are the command structures we use. I'll document the detailed specification of each field later; the CurveCP website does explain them already:

typedef struct { byte id [8]; // Command ID byte cn_client [32]; // Client short-term public key C' byte cn_nonce [8]; // Client short-term nonce byte box [80]; // Box [64 * %x0](C'->S) byte padding [64]; // Anti-amplification padding } c_hello_t; typedef struct { byte id [8]; // Command ID byte nonce [16]; // Server long-term nonce byte box [144]; // Box [S' + cookie](C'->S) } s_cookie_t; typedef struct { byte id [8]; // Command ID byte cn_client [32]; // Client short-term public key C' byte cn_cookie [96]; // Server connection cookie byte cn_nonce [8]; // Client short-term nonce byte box [112]; // Box [C + nonce + vouch](C'->S') } c_initiate_t; typedef struct { byte id [8]; // Command ID byte cn_nonce [8]; // Server short-term nonce byte box [116]; // Box [M](A'->B') (from A to B) } cs_message_t;

The flow is quite simple:

• Client sends a HELLO command to server
• Server responds with a COOKIE command
• Client responds with an INITIATE command
• Client can now send MESSAGE commands
• When server receives the INITIATE command, it can send messages.

Next Steps

As next step we'll implement CurveZMQ in the libzmq core and document the protocol properly as an RFC. The C implementation will act as a reference implementation for the protocol.

by pieterh

The short story is: Sodium delivers. This is the library I want to build ØMQ security on.

There's so much to like about Sodium and only one downside that I'll come to. The project lives on GitHub so I forked and cloned it. It uses autotools, so builds just like all our projects do:

./autogen.sh ./configure make && make check && make install

This will make it easy to integrate into FileMQ or CZMQ when the time comes. When I recall how difficult it used to be to build OpenSSL, this is gratifying. Autotools may be one of the least friendly toolsets ever but it's established a standard for C library projects that just works.

So I cleaned out my NaCl installation and installed Sodium. Sodium uses the same approach as CZMQ and libzmq, which is to provide a single header file (sodium.h) for the whole library. Awesome!

Allow me a little rant. When you make a library, please do like Sodium does, and export a single header file. Why give the user extra work with no benefit? The old technique of "if you want functionality X, include header file Y" was a great idea in 1970 when compilers took minutes per file, but is pathological in 2013.

When I include czmq.h in a program, that pulls in several dozens of the commonly-used system header files as well as the whole CZMQ interface. It adds about 0.1 seconds to a compilation. Even in 1990 when this trick added 2-3 seconds, this was a good trade-off because it gave me simpler, more consistent code.

It's very poor practice to start a C source file with a bunch of includes. Libzmq does this and it's been a source of error and confusion. "Oh yes, you have to include winsock.h but after that other file, otherwise you get a bad definition for the maximum sockets." Bleh. Create a project header file that sets-up the whole compile environment for you and then include that in each source.

And export your API as one header file, please. </rant>

So here's a minimal Sodium example:

#include <sodium.h> int main (void) { unsigned char public_key [crypto_box_PUBLICKEYBYTES]; unsigned char secret_key [crypto_box_SECRETKEYBYTES]; crypto_box_keypair (public_key, secret_key); return 0; }

Here's the next thing I love about Sodium: it is 100% compatible with the NaCl API. Not 80%, or 99%, but literally one hundred percent. It's hard to express the joy this gave me as a developer. NaCl's API is one of those "love at first sight" things. This is the gold standard for everyone building APIs: stick to the contracts, resist the temptation to screw over your users in the name of "doing it better this time". Try designing the API right the first time around, maybe?

Now, language bindings. Sodium reports two bindings, PyNaCl and RbNaCl. This means any design I make using Sodium will be easy to re-implement in the Python and Ruby bindings for libzmq (PyZMQ and rbzmq). Yay!

It was literally 30 seconds' work to change, recompile, and retest my zcurve proof-of-concept. It helps that I use a helper script called "c" that does smart things like searching /usr/local/lib for all libraries:

~/work/zcurve$ c -l zcurve-poc Compiling zcurve-poc... Linking zcurve-poc... ~/work/zcurve$ ./zcurve-poc C:HELLO: OK S:COOKIE: OK C:INITIATE: (host=localhost) OK C:MESSAGE: (request=Hello) OK S:MESSAGE: (reply=World) OK -> 'Hello World' test successful

Now, performance testing. Here's the performance I was getting from NaCl, compiled specifically for my AMD64 CPU:

One-step box: Encryptions: 6400/second Decryptions: 6400/second Two-step box: Encryptions: 1449300/second Decryptions: 762600/second

The one-step box does a hashed key exchange and then does an encryption using the resulting key. The key exchange is slow, so NaCl lets you do this in two steps: do the key exchange once, and the encryption repeatedly (using a different nonce each time). That gives the second result.

Now here are the figures for Sodium:

One-step box: Encryptions: 300/second Decryptions: 300/second Two-step box: Encryptions: 733400/second Decryptions: 488800/second

The lesson is clear: don't use single-step encryption unless you're really doing a one-off (as in the start of the CurveCP handshake). The two-step encryption is about half the speed of NaCl's optimized code. The Sodium team know that performance is a concern, and they say:

Optimized implementations (including NEON optimizations) will eventually be supported, but tests and benchmarks will be performed at run-time, so that the same binary package can still run everywhere.

They've gotten it right: clean packaging and portability is more important than raw performance. It'll take a few years for Sodium to replace OpenSSL as the security library of choice and by that time, there will be enough people using it to make optimizations worthwhile.

by pieterh