During my time as a the team lead of a SOA team my views on how integrations are built has continually evolved and changed. I am exposed to ideas through colleagues, online, at community events, through books and via experience. Every now and again I am asked a question in a meeting and I know the answer but I don't know how to do it justice with an explanation without going away and thinking it through. I am going to try and use this post to organise my thoughts and prepare for the next meeting!

(Usual disclaimer applies, ideas and thoughts I have aren't unique and certainly aren't stuff I have come up with all on my own!)

The question

I was shown the above diagram where each line is an integration and asked "Finance says all these integrations cost too much and we can't go on spending like this. How can we reduce the number of integrations?

I think this is a typical SOA question and as I have moved to thinking about a more API centered approach I think it’s the wrong question to ask.

Why is this a SOA question?

I think the actual premise of the question (reduce the number of integrations) comes from a SOA way of thinking. Reducing the number of integrations means less cost.

If you have:
- good strong central management of integrations
- well thought through reusable models
You can avoid duplicating the effort of building the same feeds twice by leveraging reuse. You have less to maintain and a simpler estate. This means much less cost - sounds simple right?

But SOA didn't work:

SOA gives us expensive integrations

• To achieve the cost savings promised by SOA you end up trying to do every integration 'properly'.
• Development process takes time and involve many people.
• We build complex data models to support this
• We have to communicate and explain these models to everyone involved

And reuse was rarely possible

• Complexity led to brittleness, the second project to use a canonical model frequently needed breaking changes

SOA team becomes the enemy of agility

• Central SOA team too slow to keep up with demand
• Endless discussions about granularity of data models ensue. (Should we have a person integration? No it should be a customer integration? How about customer addresses? Wait, customer disability information needs to be on a separate integration due to GDPR concerns? - these kinds of questions go on and on!)

So some people work around SOA, create 'under the radar' integrations in order to ensure customer success and with SOA, the spider diagram grows faster and faster. These people aren't 'baddies', in fact they are frequently the teams that are able to deliver the most value to customers!

You could argue that this is the result of lack of central governance/management of integrations and wider ICT but I don't think this is true; This story is not unique and the fact that the industry has moved on to a 3rd generation API's everywhere suggests it is more fundamental than that.

SOA seems to me to be a central authority-based design first approach where teams build what data architects determine. Conway’s law ensures this complex authority-based arrangement leads to the building of slow, complex and expensive to maintain systems.
The move to API’s allows us to push key detailed decisions down into the development teams. ICT leadership should concentre on building the people in ICT into teams and roles focused on producing the right products. The actual system designs will sort itself out as the teams are empowered to use the right tools to build the right solutions.

I don't think the SOA principles of re-usability, canonical models, etc are wrong, I just think that they are not the place to start thinking about integrations in a large organisation. The architects can never hold every detail of every integration in their head and they frequently end up in opposition with the development teams.

What question should we actually be asking?

I would argue that the question is wrong. The business doesn't care how many integrations there, rather what this all costs.

All these lines have multiple factors influencing their cost including:
- Cost to build in first place
- Cost to maintain
- Cost to remove
- Availability of appropriate tools

Not every integration needs to be enterprise quality and heavy weight. What SOA is completely missing is a method of choosing where applying SOA principles will produce a cost saving vs where it will not.

How did the people we have and the organisational structures we put them in end up producing this complexity?

Admitting this is the question means understanding that at this scale IT systems are designed by the culture ICT leadership creates not the well laid out plans of high-level architects. Once we are here we can start to work out how we go about designing a culture which is focused on the products we need to produce customer success!

Then we can go on to ask questions like how do we know where to investing time and effort in building reusable integrations?!

Two next steps

I am still thinking about all this, and there are two big areas I want to learn more about.

API Management and API Gateways

API Gateways are great tools for letting technical teams negotiate their way out of the monster picture that appears above! How can we leverage these tools across the enterprise to ensure we achieve these goals?

Taking design decisions away from architects and into product teams

I talk frequently about the spotify method of organising IT and I am a fan. I understand it is a continually evolving process and there is no actual spotify method but it is more like a spoitify culture. I want to understand more about how we can an enterprise can leverage this approach and move away from the highly centralised authority driven approaches I am used to.