Producing middle ware that is both lean and enterprise worthy is a difficult job. It’s either non-existent or requires innovative thinking (a lot of it) and a lot of going back and forth with your implementations. Very risky business, but if you get it right, it puts you far ahead of anyone else. It’s why we thought of re-writing WSO2 BAM from scratch and taking a leap rather than chugging away slowly by iterative fixing. If you prefer to hear me rather than reading this, please catch a webinar on this at http://bit.ly/xKxm8R.
When you try to monitor your business activities, you need to plug in to your servers and capture events. It sounds easy enough, so what’s the big deal? you may ask. Here’s a few road blocks we hit with our intial BAM 1.x version:
- Performance – We plug in to our ESBs and App Servers and all metrics were perfect. It nicely showed request counts, response times, etc. It was perfect as long as the load is low. If one server starts sending 1000 events/sec, things started getting ugly. Even worse, if we plug in to a few servers and start getting 1 billion events / day, well, that would have been a nightmare from the word go. We couldn’t even fathom what would happen at that type of scale.
- Scalability – We need to store events and process them. Sadly, we discovered the hard waye this would mean is we need to scale in many different ways.
- Event load – We need to scale in terms oh handling large amounts of events. We didn’t have a high performance server, but no matter how good our performance would be, there is still a breaking point. Afterwards, you need to scale.
- Storage – If you store 1000 events a day, your data will grow. And, all of us hate to delete off old email, to get more inbox space. So naturally, everyone wants to keep their events.
- Processing power – When you want to analyze events that you collect, a single server can only give you that much of processing power. You need to scale out your analytics. Another, ‘oh, so obvious’ thing that we learnt eventually.
- Customizability – We provided a lovely set of dashboards that showed all you wanted to know about your server and API metrics. But, no one is ever satisfied with what we they have. They want more. They want to monitor their metrics and analyze their data and put up their own graphs. And, of course, they want to do it now, not in 2 months.
In May 2011, we decided to start a whole new initiative to re-write WSO2 BAM from scratch. We analyzed the problem made a few decisions. Here’s a few of them.
- Divide and conquer – We divided the problem. We have to aggregate, analyze and present data. So we built separate components for each, keeping in mind that we need to scale each individually. We mapped these into the event receiver, analyzer framework and a presentation layer. Data agents are the link between anyone who wants to send events and the BAM server. The WSO2 Carbon platform, allows us to easily uninstall a component from any server. This means we can take the BAM distro, uninstall other components just to make an Event Receiver BAM server. Or to make an Analyzer BAM server. It’s just a click of a button.
- Scalable and fast storage – We chose to use Apache Cassandra as our storage solution. I do not want to argue that it’s the best data store ever. But, it works for us well. It allows us to do fast writes to store a large amount of data, quickly. Also, it’s built to scale. Scaling up Cassandra, takes minutes, not weeks. And scaling up doesn’t mean it’s going to cost you. Also, it’s written in Java, and being a Java house, it allows us to hack around the code.
- Fast protocol – We chose to use Apache Thrift as our default protocol. There are many arguments against it, but it holds up well for us. It’s fast and it does it’s job. It allows us to maintain sessions, supports a bunch of languages. One key thing was Cassandra uses it as well, allowing us to gain more performance in streaming data into Cassandra without deserializing.
- Scalable analytics – We chose to write our own analytics language. But, if it doesn’t suit you, you can plugin your own java code. Hadoop is unavoidable when it comes to scaling analytics. So, we decided to have a Hadoop mode for large amounts of data and a non-Hadoop mode, so that anyone can just use BAM without worrying about any Hadoop cluster.
- Gadget based dashboards/reports – Drag and drop visualizations are very attractive when you don’t want to spend weeks writing code to visualize. We developed a gadget generator so you can quickly visualize your analyzed data easily.
After a couple of milestones, we were able to spin off an alpha. It’s available here: http://dist.wso2.org/products/bam/2.0.0-Alpha/wso2bam-2.0.0-ALPHA.zip. It is not the silver bullet and documentation is still WIP. But, if we haven’t already reached our destination, it’s within our reach now.