BAM, SOA & Big Data

Leveraging Big Data has become a commodity for most IT departments. It’s like the mobile phone. You can’t remember the times when you couldn’t just call someone from your mobile, no matter where you are in the world, can you? Similarly, IT folks can’t remember the days when files were too big to summarize, or grep, or even just store. Setup a Hadoop cluster and everything can be stored, analyzed and made sense of. But, then I tried to ask the question, what if the data is not stored in a file? What if it was all flying around in my system?


Shown above is a setup that is not uncommon deployment of a production SOA setup. Let’s summarize briefly what each server does:

  • An ESB cluster fronts all the traffic and does some content based routing (CBR).
  • Internal and external app server clusters host apps that serve different audiences.
  • A Data Services Server cluster exposes Database operations as a service.
  • A BPS cluster coordinates a bunch of processes between the ESB, one App server cluster and the DSS cluster.

Hard to digest? Fear not. It’s a complicated system that would serve a lot of complex requirements while enhancing re-use, interoperability and all other good things SOA brings.

Now, in this kind of system whether it’s SOA enabled or not, there lies a tremendous amount of data. And No, they are not stored as files. They are transferred between your servers and systems. Tons and tons of valuable data are going through your system everyday. What if you could excavate this treasure of data and make use of all the hidden gems to derive business intelligence?

The answer to this can be achieved through Business Activity Monitoring (BAM-ing). It would involve the process of aggregating, analyzing and presenting data. SOA and BAM was always a love story. As system functions were exposed as services, monitoring these services meant you were able to monitor the whole system. Most of the time, if the system architects were smart, they used open standards, that made plugging and monitoring systems even easier.

But even with BAM, it was impossible to capture every message and every request that passed through the server. The data growth alone would be tremendous for a fairly active deployment. So, here we have a Big Data problem, but it is not a typical one. A big data problem that concerns live data. So to actually fully monitor all the data that passes through your system you need a BAM solution that is Big Data ready. In other words, to make full sense of the data and derive intelligence out of the data that passes through modern systems, we need a Business Activity Monitor that is Big Data ready.

Now, a system architect has to worry about BAM, SOA and Big Data as they are essentially interwined. A solution that delivers anything less, is well short of a visionary.

Govern your SOA deployment easily and it won’t cost you a thing…

SOA is the solution to all integration problems. Legacy apps, apps written in different eras, different platforms, languages all talk to each other through SOA. Now with SOA being the solution to everything, causes a new problem 🙂 Everyone is using SOA, means that everyone is exposing their functionality through a service, and a typical organization has a lot of functionality, resulting in a lot of, I mean “quite” a lot of services. Keeping track of all these services is a new nightmare! Is SOA actually giving more headaches than it solves? Well, not exactly. That is where governance comes in. You need to govern your SOA, and that needs to be a key factor when your planning your SOA deployment. So, when your planning you need what’s now popularly termed “Design-Time Governance”.

So, what do you need to look for in this space?
  1. Central location – To avoid the nightmare of everyone maintaining different documents, you need a central place where everyone 
  2. Separate locations – This might appear to contradict with the point above. But this refers the ability to drop documents into separate locations that are only visible to the relevant servers. For example: If you want only want one particular server to see/access the document, you drop it to one location, but if you want a set of servers in a cluster to see the document you drop it to a different location. Furthermore, if you want your whole SOA deployment to see the document you drop it to that location.
  3. Versioning – Your SOA setup is done and it is working like a charm. Your job is done. Or is it? 6 months down the line, you won’t be keeping the same functionality now, would you? No, you’ll be adding more functionality and exposing new versions of the service. So you need to support versioning of your services. It’s important to that you be able to easily create new versions, maintain old versions and restore old versions if something goes wrong.
  4. Permissions – That’s right, now you wouldn’t want everyone to see the company’s pay roll now would you? I thought so. So you need to restrict who sees which documents and control access to.
Have these essentials incorporated into your SOA deployment and I’m sure governing your SOA deployment would be a breeze. It is important remember that proper design-time governance often decided the success (or the failure) of your SOA initiative. To easily have the ability to use all these features take a look at this fully fledged commercial product available here and it’s available free of charge.