Calling All Architects: It’s Time to Develop a Complex!

Explore how enterprise architects can handle complexity in the context of Microservices

I would like to discuss the role of architects in the context of Microservices in a very roundabout way…

I got an email yesterday from a software vendor telling me that their solution would “reduce complexity” for my enterprise applications. I see this a lot, especially in association with the latest IT trends. J2EE will reduce code complexity! SOA will remove integration spaghetti! Cloud computing will eliminate infrastructure complexity! We’re starting to see this with Microservices too. We’ve lived through or are living through all of these trends and have certainly benefitted from them all when they’ve been applied correctly. But have our applications or infrastructure become less complex? Should they be any less complex?

Fred Brooks is best known for his seminal book The Mythical Man Month but he followed that work up with a paper titled “No Silver Bullet” in 1986 that is an equally important read for anyone involved in software engineering. As the title implies, Brooks’ main point is that progress is made in the field of software can only be made incrementally, not through quantum leaps in productivity and innovation. He illustrates this with an inspiring analogy:

The first step toward the management of disease was replacement of demon theories and humours theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers the progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.

A silver bullet is what people are looking for and this quest is the reason we see so much magical hype at the time new trends emerge. The paper goes on to discuss why software has this constraint on progress. Specifically, Brooks examines complexity of software systems and defines two types:

  • Essential complexity The complexity of the software’s functional scope and the problems it solves (e.g. correlating and analyzing large amounts of data in real time)
  • Accidental complexity The complexity of the software’s implementation details (e.g. the languages, processes and messages used to do the work)

Brooks argues that accidental complexity can be reduced but essential complexity cannot. This is why it seems like these new trends never match their hype: even though they may do a good job in reducing accidental complexity, the essential complexity still feels overwhelming.

This brings us back to Microservices. It is refreshing to see that many in the Microservices community are embracing the fact that software is complex and is going to remain so. Sam Newman highlights this in his indispensable book Building Microservices. Industry leaders Martin Fowler , Michael Feathers and Mary Poppendieck have all written or spoken about complexity in Microservices at scale. Matt Heath shared his experience dealing with complexity in Hailo’s Microservices implementation.

At our API Academy Microservices Boot Camp last month, Ronnie Mitra gave a talk on designing a Microservices architecture. He discussed Brooks’ complexity types, Larry Tesler’s “Law of the Conservation of Complexity” and then illustrated where complexity moves in a Microservice architecture:

monolith-v2

microservices-v2

In the Microservices picture, the complexity moves to the “system”. Earlier in the Boot Camp, I had stressed the importance of considering the system aspects when defining goals and principles for Microservices. In my opinion, looking at an interconnected set of Microservices as a system rather than as a single application is important in managing its essential complexity.

System theory and systems thinking are mature schools of thought whose scope goes well beyond technology. Nevertheless, they include lessons that apply to software systems, particularly in dealing with essential complexity. Techniques such as modularity, abstraction and analogy help make complexity consumable by the human mind.

W. Brian Arthur of the Santa Fe Institute—an organization devoted to studying complexity—says the following in his excellent book The Nature of Technology: “Modularity … is to a technological economy what the division of labor is to a manufacturing one.”  This concept directly addresses productivity, the output Brooks was hoping to drive in his paper. I believe that Microservice architecture is all about applying modularity at the right level of abstraction in the system in order to make its essential complexity as easy to deal with as possible. Of course, that doesn’t make it easy.

Going back to Ronnie’s presentation, who is this “System Maintainer” of whom he speaks?  Microservices is a developer-driven movement for the most part. Many of the enterprise architects I talk to about Microservices are being pulled into the topic by momentum in their organizations originating within development groups. In many of the Microservices adoption stories I have studied—including the biggest and most mature—not enough attention is being paid to the attributes and evolvability of the system. In the link from Feathers above, he articulates this concern well.

So, to all of you technologists who have “Architect” in your title, now is your time!  This is your opportunity to step out of the weeds and out of your comfort zone. You are in the best position to understand the complexity of your organization’s software systems.  Large enterprise monolithic applications mask their own complexity. Microservice architecture lays it bare, just as germ theory did to disease in Brooks’ analogy above.  It follows that embracing complexity is the beginning of hope and the path to productivity.  You, architects, get in there and be the system champion!  Be the person in your organization who helps navigate its complexity.

For more information on this topic, check out the replay of this webcast by Mike Amundsen and me.

The Author

Matt McLarty

Team Lead

Matt McLarty is an experienced software architect who leads the API Academy at CA Technologies. He helps organizations with their strategy and architecture for APIs, microservices and enterprise integration. Matt recently co-authored the book Microservice Architecture for O’Reilly, with his API Academy colleagues.

Related Articles

Microservices

Voltaire, Vonnegut, and the Vocabulary of Microservices

The lack of common concepts and axioms is holding the software engineering industry back. This blog post explores the need for a common distributed systems vocabulary to help with that problem.

Matt McLarty on Aug 10, 2018

Microservices

Applying and Extending DHARMA

This post gives some practical examples of the DHARMA method for API Security in a Microservice Architecture, and also shares some opportunities for extending the model.

Matt McLarty on Jul 9, 2018

Microservices and APIs
API Strategy

Microservices, APIs and Innovation: The Power of APIs

Explore the role APIs play in empowering teams and enabling organizations to innovate.

Mike Amundsen on May 24, 2018

Join the Conversation