The session will be on Live Meeting, at http://snipr.com/virtualaltnet.

Please join us on Monday, times below:

• France/Germany/Belgium: 8:00PM
• UK is: 7:00PM
• EST in the US: 2:00PM
• PST in the US: 11:00AM

Special thanks to Stefan for thoroughly reviewing a draft of the article, Dan North for helping shape an even earlier version, and Jim Webber and Ian Cartwright for lots of conversation and encouragement.

I’ve still not written much on how to implement consumer-driven contracts, but I’ve something special cooking (slowly) in that area: expect some illustrative material to emerge over the next few months.

• Stories – in the form of Behaviour-Driven Development exemplars – describe goals and desired outcomes;
• Capabilities encapsulate the resources and abilities an organisation needs to satisfy those goals;
• Services host capabilities;
• Contracts – particularly consumer-driven contracts – assert the interactions between services.

We iterate over this model many times in the course of an engagement, emphasising different parts as we go. We concentrate on organisation-level stories and capabilities when establishing an organisation-wide context, then project-level stories, capabilities and services whilst planning projects and delivery streams, and then release- and iteration-level artefacts during specific instances of delivery.

Such an approach represents a “thin-slice” way to:

• establish context, business goals and consensus amongst stakeholders;
• create a long-term vision that joins up the business, architectural and technology views of an SOA initiative;
• describe and challenge an organisation’s goals and the benefits attached to those goals;
• describe the capabilities needed to meet those goals;
• identify the quality-of-service expectations the business has of those capabilities;
• identify services and assign capabilities to services;
• describe and test the externally visible interactions between services;
• identify, plan and develop slices of service functionality that deliver business benefits early and often.

In preparation for the launch of the ThoughtWorks Anthology, each of the authors recorded a short podcast in which they discuss their essay. Mine is online now. It’s four minutes tops – the perfect aural companion as you huddle outside with a cigarette.

If you like that, you might like this: a couple of years ago, Martin Fowler and I did an interview with Microsoft’s Ron Jacobs on the Evolution of Architecture. As per usual, I’m a one-note band…

• A Contract Vocabulary: Part 1
• A Contract Vocabulary: Part 2

In this post I’m going to address a few questions and puzzles:

• Does Consumer-Driven Contracts really achieve a separation of concerns?
• Are consumer-driven contracts applicable only in point-to-point scenarios?
• Never mind the theory, is all this feasible in practice?

A Separation of Concerns?

Is the triumvirate of provider, consumer and consumer-driven contracts a true separation of concerns? The goal of any separation of concerns is to allow functions [to] be optimized independently of other functions. Mark Baker suggests that we will have successfully separated concerns if variations in [concern] A do not induce or require a corresponding change in [concern] B (and usually, the converse). Do our three contracts pass this test?

By distinguishing between provider and consumer contracts we make explicit the fact that one half of the provider-consumer relationship ought to be able to change without inducing a corresponding change in the other half – within limits. Providers ought to be able to evolve their contracts without obliging their consumers to change – just so long as they continue to satisfy existing obligations. Consumers ought to be able to modify their consumption of a provider’s existing services without requiring either that their peers change or that the provider itself be updated – up to the point they require new functionality of the service provider.

These limit cases suggest that our concerns here are not so much separated as separating. What prevents an absolute separation is the very thing that constitutes some measure of success for any distributed system: good services are used, and in being used, become a “victim” of their own success. These limits to the separation of concerns are surfaced as a third type of contract: the consumer-driven contract. Such contracts measure the success of a service, insofar as they catalogue its obligations, whilst at the same time describing when and where a variation in concern A (the provider contract) will induce a change in concern B (one or more consumer contracts).

Service Indirection

Another question I sometimes get asked is: aren’t your three contracts really only applicable in point-to-point scenarios, where consumers communicate directly with a service provider?

It’s important here to distinguish between the accountability of a service and its location. As a consumer of services, I tend to want to delegate my business to providers I trust, even if I don’t quite know where the specific person, department or office dealing with my request is located. Accountability in this instance is really shorthand for my ability to determine whether a provider is capable of satisfying my functional and quality-of-service requirements, and thereafter to demand traceability of it in respect of any requests I make. I’d be foolish to delegate my interests to some anonymous party I know nothing about, and from whom I can’t secure a guarantee of accountability, which is why I tend to draw back from an SOA-ad-absurdum that encourages me to fling messages into the “cloud,” to be satisfied by “something,” “somewhere.”

The key point here is that consumer-driven contracts require a site of responsibility or accountability. Mediated, brokered and routed messaging scenarios are perfectly amenable to consumer-driven contracts just so long as they don’t diminish service responsibility. It may be that I delegate some of my due diligence activities to a broker, which then matches my requirements against trusted providers; but even here, where I don’t know who exactly is satisfying my requests, I have a trust relationship, an expectation-obligation relationship, with at least one party – the broker – and therefore a potential site for consumer-driven contracts.

Theory into Practice

I ended my last post with the suggestion that in many cases it’s useful for a service provider to know something of its consumer-driven contract. Assuming you do want to know more about a provider’s current contractual obligations, we then need to ask: can you in fact discover what those obligations are? Can you identify a service’s consumers?

The answers to these questions will vary according to the environment in which a service provider is situated. In an enterprise environment, one in which the landscape is well known, it is often possible to identify all consumers of a service. Outside the enterprise, things may be more difficult. Services on the Web, for example, may have a largely anonymous clientele, making it near impossible to discover “existing” contractual obligations.

That’s not the end of the problem. Even if you can identify a consumer, it may still prove difficult to establish whether a relationship between that consumer and the provider is in fact currently in force. Whilst many such relationships are long-lived, particularly in the enterprise (I’m not talking about long-lived instances of a connection or transaction, merely the fact that for some period of time we can confidently say that service X is likely to collaborate or have some relation with service Y to satisfy such-and-such a business process), others are more ad hoc and less easily identified – a function of an agent crawling a service inventory, perhaps. In these circumstances, a snapshot of contractual obligations, even if some such thing could be generated, may well have some fuzzy edges.

The twin issues of the liveness and lifetimes of consumer-driven contracts and the consumer contracts from which they are derived are of renewed interest to me today. When does a consumer-driven contract begin? When does it end? When does it change?

I’ve no doubt that all three kinds of contract – provider, consumer and consumer-driven – are more or less in play in every distributed interaction, irrespective of the architectural and implementation decisions that have helped shape that interaction. But it’s clear to me now that the separation of concerns described in my original article was influenced heavily by the relatively static nature of the Web Services stack’s contract mechanism. WSDL is an upfront (partial) declaration of a provider contract. Clients activate an application protocol – engage in a conversation with a service – by invoking the operations that have been published in the WSDL (the protocol’s not inherent in the contract, only the levers by which it can be activated). By contrast, more RESTful services allow for a certain dynamism in the conversational aspects of a provider contract: representations in and of themselves advertise the levers by which a client can advance the application protocol, and they do this without reference to some pre-existing static contract map. Provider contracts in such services are “for the now,” and are exhausted in being consumed.

How do the forces at play in a RESTful interaction affect our understanding of consumer-driven contracts? If they make it more difficult to describe consumer relations outside of a specific instance of an interaction, do they not also make it correspondingly less important to know something of the consumer relation? My thinking is not settled on these issues. Sometimes I’m inclined to think of a consumer-driven contract as a resource, as a promise to provide such-and-such, to be provisioned by the provider for a certain period of time; at other times I prefer to see it as a function of per-request content negotiation. More on this another time.

Summary

In summary, Consumer-Driven Contracts describes in some detail the forces in play in any loosely coupled relationship. It does this by proposing that three types of contract exist between service providers and consumers. Real-world implementations of distributed systems exhibit all three types of contract, but the degree to which we can describe each contract, and the need to do so, varies according to the context.

Provider Contracts

These are the contracts with which many of us imbued with service-oriented concepts will be most familiar. They’re the contracts identified in that hoary old tenet, “Services share schemas and contracts, not classes and types.” Provider contracts mark services as king of their own domain – “this is what I am, what I offer, take it or leave it.” They’re the first-line defence in the struggle to preserve service autonomy. As I suggested in my previous post, in the WS-* world provider contracts typically comprise some or all of WSDL plus XML Schema plus WS-Policy.

Consumer Contracts

These kinds of contract are really a private affair, particular to a specific consumer – to the way in which that consumer uses a service. In short, a consumer contract is a description of the parts of the service in which the consumer is really interested, the parts upon which it really depends. These might be interfaces, parts of an interface, operations, schemas, parts of schemas, quality-of-service characteristics, etc.: all those things I cheerfully labelled “a logical set of exportable business function elements.” More often than not, these specific usage characteristics remain implicit, and can be gleaned only be diving into the implementation of the consumer.

Some consumers need everything a provider exports: schemas, operations, endpoints, policies, and so on. More often than not, however, a consumer really only needs a subset of what the provider offers. Some consumers import only what is absolutely necessary, and ignore or tolerate changes in all the rest, but others will import or bind to everything, irrespective of its usefulness. Binding to everything might be appropriate in some circumstances, but at other times it’s a real barrier to evolution.

Oddly enough, I get the feeling that when people refer to consumer-driven contracts, they’re actually talking about consumer contracts. Most of the practical work in the area of service evolution seeks to reduce the amount of knowledge a consumer must have of a provider, whether it’s knowledge of a provider’s implementation (almost always a bad idea), or knowledge of parts of a provider contract that really have no bearing on the consumer. After all, knowledge is a dependency, isn’t it? Reduce the consumer’s knowledge of the provider – limit the scope of the consumer contract – and so achieve just enough coupling, no more.

I suggest it’s useful for consumers to know what their consumer contracts are.

Consumer-Driven Contracts

In the original article I also called these derived contracts, but I don’t often use that term: “consumer-driven contracts” was suggested to me by Robin Shorrock, and I much prefer it – though if you’ve been test-, behaviour-, budget- and whatnot- driven to a frenzy, you might prefer “derived.”

Consumer-driven contracts bring the contractual forces back home to the provider. Assume for a moment that you could identify all the current consumers of your service; assume also that for each consumer you could know its consumer contract. A consumer-driven contract is the aggregate of all these consumer contracts. It’s a snapshot of existing contractual relations: it’s what the provider has to do to really satisfy existing dependencies, as opposed to what it started off offering. On the one hand, the provider offers a provider contract – “all that I am” – on the other, based on some set of current consumer expectations, it’s obliged to maintain only some portion of this contract. (Note that in some cases this “some portion” might in fact be everything that a provider offers under its provider contract, in which case its provider and consumer-driven contract look similar.)

Whilst we can say that a weak form of consumer-driven contract is implicit in all provider-consumer relations, Consumer-Driven Contracts took the idea a little further, suggesting that consumers might wish to import their consumer contracts into the provider, thereby surfacing an explicit consumer-driven contract.

Now the thorny question is: should you really seek to identify a provider’s “real” contractual obligations? After all, isn’t this extra knowledge, and isn’t extra knowledge an extra dependency, a tightening of the coupling screw?

My first response is to point out that these consumer expectations, and the provider’s obligation to satisfy them, exist whether we like it or not. Change something upon which a consumer currently depends, and you might very well break that consumer. Wouldn’t it be useful to know what you can and can’t change about a provider’s exported functionality, its provider contract, based on a “current” view of consumer relations? Wouldn’t this perhaps help you plan and govern the evolution of your service in response to changing business requirements?

I suggest that in many cases it’s useful for a provider to know what its consumer-driven contract is.

Looked at from the point of view of these details, the article appears to be nothing more that a summary of some reasonably well-known validation techniques as they apply to specific schema, validation and contract languages. But I’m not sure that’s all there is to it. For me, the core focus has always been on a simple separation of concerns. Consumer-Driven Contracts was motivated by a desire to unpack the term “loosely coupled;” to turn a spotlight on the several facets of the contractual relations between services, and to name each part. I wanted to create a vocabulary for describing the roles of the participants in a loosely coupled relationship, so that we might better understand who does what when a service changes, and so articulate the knotty problem of service evolution.

Services collaborate to satisfy a business process. Where two services communicate, at least one assumes the role of provider, the other of consumer. To the extent that the consumer uses something of the provider, it is coupled to that provider. In the interests of autonomy, process agility, reuse, and so on, we seek to reduce the degree of coupling between providers and consumers. But no matter how successful we are in loosely coupling services, we recognize that for a system to do something useful, some small amount of coupling must remain.

What does it mean to be coupled, one service to another? What relations or forces are at play between provider and consumer? Between a provider and all the consumers who depend upon it?

Service Contracts

We can begin answering these questions by reviewing our understanding of service contracts. Service contracts are often described as what the requestors and providers of a service must agree on. In the WS-* universe, service contracts are expressed using commoditised, standards-based XML vocabularies:

• WSDL (Web Service Description Language) descriptions of a service’s operational behaviour – including structural definitions of the operations a service supports and the messages that go in and come out of a service – together with details of where to find a service and how to bind to it;
• XML Schema descriptions of the messages sent and received by the service;
• WS-Policy descriptions of any constraints on accessing and using the service, such as security requirements that a consumer must adhere to, or acceptable transports over which the service can be contacted. Policies separate these constraints from the service’s behaviour: if a service is relocated, perhaps to a new address or transport, its policy may change whilst its behaviour remains the same.

Such service contracts promote loose coupling by encapsulating the provider’s execution environment, thereby increasing the platform independence of the several parties in a distributed system.

A key point to note about Web Service contracts is that they are almost always explicit, being published using one or more of the contract formats described above. With the standardisation of these formats, there arose widespread tool support for generating language-specific clients and message representations. Looked at in one way, this appears to be the natural maturation of the Web Services platform; looked at another, the volume of tool support hints at the development burden imposed by a service’s relative freedom to implement an illimitable number of operations and message schemas. Operations proliferate because there are no intrinsic constraints on the number or names of operations exposed by a WS-* service: your service exposes a GetCustomerDetails operation; mine supports HandleCustomerDetailRequest. Message schemas proliferate in large part because of the de facto predominance of XML. If all you have is XML, then the choice – if you can call it a choice – of media type adds nothing to the interpretation of a message. This leads us to innovate at the level of the structure of the representations we make using that one media type: all too often, your Customer schema ends up looking nothing like my Customer schema.

In short, the Web Services platform trades standardised, commoditised and constrained contract metadata in return for an unconstrained set of operations and message schemas. From the point of view of a consumer wanting to invoke operations on and exchange messages with service providers, each service differs from every other, thereby requiring a service-specific proxy per consumer. Hence the need for explicit contracts on the provider side and toolset support for generating proxies on the consumer side.

Unlike WS-* services, RESTful services typically eschew explicit contracts: but that doesn’t mean that RESTful contracts don’t exist. Stefan Tilkov discusses this very issue in his Addressing Doubts about REST. The implicit RESTful contract is in large part captured by the HTTP specification, which describes a set of operations that can be performed on the resources exposed by a service, together with a set of response codes that guide a consumer’s interactions with those resources. Format-wise, REST encourages us to choose an appropriate IANA-recognised media type, such as XHTML, Atom, JSON, form encoding, or even XML, to structure a representation in the most economical and widely recognized manner possible. This focuses us on identifying potential structural similarities in our data, rather than essential semantic differences. A set of configuration parameters, for example, appears quite different from a customer’s contact details; leading us, perhaps, to create a Config schema for the one, a Customer schema for the other. But looked at in terms of their structure, both resources might be thought of as comprising sets of name-value pairs, making it practical to represent them in a widely recognised form-encoded format.

Whether implicit or explicit, service contracts to date have been overly provider-centric. In contrast to this monadic focus, Consumer-Driven Contracts distinguishes three types of contract: provider contracts, consumer contracts and consumer-driven contracts. I suggest that these contractual relations exist whether we like it or not. In a sense, we don’t opt to implement these contracts, least of all the consumer and consumer-driven contracts: they exist from the moment a service is up and running and being used by one or more clients. We’re not obliged to design and build to these contracts, but having named them and surfaced them as distinct from one another, we can leverage their characteristics if we so choose.

In my next post I’ll describe each of these three contracts in more detail.

A couple of years ago I wrote about some of the challenges we face when evolving services in response to changes in the business processes they implement, or to changes in the way they are used by their consumers. The resulting article, Consumer-Driven Contracts: A Service Evolution Pattern, was published on Martin Fowler’s site and shortly thereafter on MSDN. This month I’m delighted to announce that a newly revised version can be found – in somewhat humbling company – in the fabulous ThoughtWorks Anthology.

Consumer-Driven Contracts described what I perceived to be a common problem in real-world service-oriented solutions – that of the ostensibly loosely-coupled service estate being in fact extremely difficult to change on a service-by-service basis – and explained how we might make our service providers and consumers more independent of one another and better able to tolerate change. The article drew on my experience of building services in large organizations where, because of our schedule of frequent releases, we had to balance innovation with the minimum of system and organisational disruption. Besides discussing how we can engineer services that are tolerant of change, the article touched on several other topics related to the design and delivery of distributed systems – topics that interested me then, and continue to interest me today:

• Are agile software methodologies compatible with delivering organizational agility in the context of a service-oriented approach to business process automation?
• How can we align the specification and delivery of service functionality with the key business process drivers within an organisation?
• What are the roles and responsibilities of the several parties in a loosely coupled relationship?
• Under what circumstances can we depend on toolset-assisted service implementations, and when should we adopt more deliberate – sometimes more complex – implementation strategies, to ensure the longevity of a system?
• How ought we to translate the collaborations between the business functions through which an organisation realizes its core processes into our services’ externally observed behaviours and interactions?

I’ve discussed the ideas in Consumer-Driven Contracts with many friends and colleagues over the last couple of years, and have learnt much by continuing to test and refine their application at ground level. Now with the recent publication of the ThoughtWorks Anthology, I thought it appropriate to conduct something of a retrospective. Beginning with my next post, where I’ll consider Consumer-Driven Contracts in terms of its describing a contract vocabulary, I’ll reflect on what I’ve learnt, unpick some of the several strands in the article, and dive further into some of its more important themes.

Updated: The first three parts of this retrospective can be found here:

• A Contract Vocabulary: Part 1
• A Contract Vocabulary: Part 2
• A Contract Vocabulary: Part 3