CSV Connector for WSO2 BAM

I’ve worked on a small tool to publish your spreadsheets (after converting to CSVs of course!) to WSO2 BAM.

The cool thing is you can publish 1 (or 1000s) of spread sheets to WSO2 BAM, and use the HiveAnalytics UI to slice and dice them to produce neat results.

So, you need maven to build and run this (and of course, WSO2 BAM up and running). Here are the steps:

1. Download and unzip the source from this link.

2. Run ‘mvn clean install’ at the unzipped location.

3. Now run the exec command in maven as per the following example: ‘mvn exec:java -Dexec.mainClass=org.wso2.carbon.bam.CSVAgent -DcsvFile=../ExportCustomerAccounts.csv -DstreamName=CustomerAccounts -DstreamVersion=1.2.0’

Here is what happens:

“CustomerAccounts” is the stream that will get created out of the CSV file, “ExportCustomerAccounts.csv”. All streams are versioned in BAM, so this stream will have the version “1.2.0”. Versioning means you can publish different versions of the CSV (columns deleted or added) with different versions.

The potential of this is you can publish any number of CSVs to BAM and make use of the SQL-like Hive query language to do joins and group bys to get valuable information out of your spread sheets.

WSO2 BAM 2.0.0 released!

 

The screenshots above show the final result of a Service statistics monitoring use case. Data across many servers got published to BAM, had to be analyzed and then presented on the dashboard you see above. Nothing better than a cool dashboard to make sense of all that data ūüėČ

It has been an enduring journey with an abundance of learning curves that allows the BAM team to make some great technologies work together seamlessly. After, spending almost an year on a complete re-write of the WSO2 Business Activity Monitor, we were able to put the 2.0.0 release of this product, which is a complete re-write of the 1.x product. It has been a marathon effort for the last few months, and having a great team made all the work feel like a refreshing summer breeze.

The release note I concocted should say all you need to know about the product. A major thanks to everyone who helped inside and outside WSO2 to make the final release a reality.

 

WSO2 Business Activity Monitor 2.0.0 released!

The WSO2 Business Activity Monitor (WSO2 BAM) is an enterprise-ready, fully-open source, complete solution for aggregating, analyzing and presenting information about business activities. The aggregation refers to collection of data, analysis refers to manipulation of data in order to extract information, and presentation refers to representing this data visually or in other ways such as alerts. The WSO2 BAM architecture reflects this natural flow in its design.

Since all WSO2 products are based on the component-based WSO2 Carbon platform, WSO2 BAM is lean, lightweight and consists of only the required components for efficient functioning. It does not contain unnecessary bulk, unlike many over-bloated, proprietary solutions. WSO2 BAM comprises of only required modules to give the best of performance, scalability and customizability, allowing businesses to achieve time-effective results for their solutions without sacrificing performance or the ability to scale.

The product is available for download at: http://wso2.com/products/business-activity-monitor

The documentation is available at: http://docs.wso2.org/wiki/display/BAM200/WSO2+Business+Activity+Monitor+Documentation

Key Features

  • Collect & Store any Type of Business Events

    • Events are named, versioned and typed by event source
    • Event structure consists of (name, value) tuples of business data, metadata and correlation data
  • High Performance Data Capture Framework

    • High performance, low latency API for receiving large volumes of business events over various transports including Apache Thrift, REST, HTTP and Web services
    • Scalable event storage into Apache Cassandra using columns families per event type
    • Non-blocking, multi-threaded, low impact Java Agent SDK for publishing events from any Java based system
    • Use of Thrift, HTTP and Web services allows event publishing from any language or platform
    • Horizontally scalable with load balancing and high available deployment
  • Pre-Built Data Agents for all WSO2 Products

  • Scalable Data Analysis Powered by Apache Hadoop

    • SQL-like flexibility for writing analysis algorithms via Apache Hive
    • Extensibility via analysis algorithms implemented in Java
    • Schedulable analysis tasks
    • Results from analysis can be stored flexibly, including in Apache Cassandra, a relational database or a file system
  • Powerful Dashboards and Reports

    • Tools for creating customized dashboards with zero code
    • Ability to write arbitrary dashboards powered by Google Gadgets and {JaggeryJS}
  • Installable Toolboxes

    • Installable artifacts to cover complete use cases
    • One click install to deploy all artifacts for a use case
Issues Fixed in This Release
All fixed issues have been recorded at –¬†http://bit.ly/Tzb1VP
Known Issues in This Release
All known issues have been recorded at –¬†http://bit.ly/TzberZ

Engaging with Community

Mailing Lists

Join our mailing list and correspond with the developers directly.

Reporting Issues

WSO2 encourages you to report issues, enhancements and feature requests for WSO2 BAM. Use the issue tracker for reporting issues.

Discussion Forums

We encourage you to use stackoverflow (with the wso2 tag) to engage with developers as well as other users.

Training

WSO2 Inc. offers a variety of professional Training Programs, including training on general Web services as well as WSO2 Business Activity Monitor and number of other products. For additional support information please refer to http://wso2.com/training/

Support

We are committed to ensuring that your enterprise middleware deployment is completely supported from evaluation to production. Our unique approach ensures that all support leverages our open development methodology and is provided by the very same engineers who build the technology.

For additional support information please refer tohttp://wso2.com/support/

For more information on WSO2 BAM, and other products from WSO2, visit the WSO2 website.


We welcome your feedback and would love to hear your thoughts on this release of WSO2 BAM.

The WSO2 BAM Development Team

 

WSO2 BAM 2.0.0-Alpha 2 released!

My team at WSO2 was able to release a 2nd alpha of our upcoming BAM 2.0. Do give it a spin.

The release note is below:

The WSO2 team is pleased to announce the release of version 2.0.0 – ALPHA 2 of WSO2 Business Activity Monitor.

WSO2 Business Activity Monitor (WSO2 BAM) is a comprehensive framework designed to solve the problems in the wide area of business activity monitoring. WSO2 BAM comprises of many modules to give the best of performance, scalability and customizability. These allow to achieve requirements of business users, dev ops, CxOs without spending countless months on customizing the solution without sacrificing performance or the ability to scale.

WSO2 BAM is powered by WSO2 Carbon, the SOA middleware component platform.

Downloads

The binary distribution can be downloaded at http://dist.wso2.org/products/bam/2.0.0-alpha2/wso2bam-2.0.0-ALPHA2.zip.

The documentation pack is available at http://dist.wso2.org/products/bam/2.0.0-alpha2/wso2bam-2.0.0-ALPHA2-docs.zip.

Samples
  1. Service Data Agent – Sample to install Service data agent, publish statistics and intercepted message activity from Service Hosting WSO2 Servers such as WSO2 AS, DSS, BPS, CEP, BRS and any other WSO2 Carbon server with the service hosting feature
  2. Mediation Data Agent – Sample to install Mediation data agent, publish mediation statistics and intercepted message activity using Message Activity Mediators from the WSO2 ESB
  3. Data center wide cluster monitoring – Sample to simulate two data centers each having two clusters sending statistics events, perform summarizations and visualize them in a dashboard
  4. End – End Message Tracing – Sample to simulate messages fired from a set of servers to WSO2 BAM and set up message tracing analytics and visualizations of respective messages
  5. KPI Definition – Sample to simulate receiving events from a server (ex: WSO2 AS), perform summarizations and visualize product and consumer data in a retail store
  6. Fault Detection & Alerting – Sample to simulate receiving events from a server (ex: WSO2 ESB), detect faults and fire email alerts

Features

  • Data Agents
    1. Pre built data agents – Service Data Agent for the WSO2 AS, DSS, BPS, CEP, BRS and any other WSO2 Carbon server with the service hosting feature and Mediation Data Agent for the WSO2 ESB
    2. A re-usable Agent API to publish events to the BAM server from any application (samples included)
    3. Apache Thrift based Agents to publish data at extremely high throughput rates
    4. Option to use Binary or HTTP protocols
  • Event Storage
    1. Apache Cassandra based scalable data architecture for high throughput of writes and reads
    2. Carbon based security mechanism on top of Cassandra
  • Analytics
    1. An Analyzer Framework with the capability of writing and plugging in any custom analysis tasks
    2. Built in Analyzers for common operations such as get, put aggregate, alert, fault detection, etc.
    3. Scheduling capability of analysis tasks
  • Visualization
    1. Drag and drop gadget IDE to visualize analyzed data with zero code
    2. Capability to plug in additional UI elements and Data sources to Gadget IDE
    3. Google gadgets based dashboard

Reporting Issues

WSO2 encourages you to report issues, enhancements and feature requests for WSO2 BAM. Use the issue tracker for reporting any of these.

WSO2 Business Activity Monitor 2.0.0 Alpha released!

After a lot of re-designing, re-architecting, re-writing, re-re-writing we have come with an alpha of the all new BAM 2. Although, this is still an alpha, it will provide a good taste of things to come in the major BAM release.

Here’s the (not so) official release note:

WSO2 Business Activity Monitor (BAM) 2.0.0-Alpha is now available for download at [1].
The 2.0.0 alpha version is a complete re-write of BAM concentrating on scalability, performance and customizability.
Samples

This release contains samples that can be run without setting up another server to send events to the BAM server.
  1. KPI Definition – Sample to simulate receiving events from a server (ex: WSO2 AS), perform summarizations and visualize product and consumer data in a retail store
  2. Fault Detection & Alerting – Sample to simulate receiving events from a server (ex: WSO2 ESB), detect faults and fire email alerts
Features

Data Agents
  1. A re-usable Agent API to publish events to the BAM server from any application (samples included)
  2. Apache Thrift based Agents to publish data at extremely high throughput rates
  3. Option to use Binary or HTTP protocols
Event Storage
  1. Apache Cassandra based scalable data architecture for high throughput of writes and reads
  2. Carbon based security mechanism on top of Cassandra
Analytics
  1. An Analyzer Framework with the capability of writing and plugging in any custom analysis tasks
  2. Built in Analyzers for common operations such as get, put aggregate, alert, fault detection, etc.
  3. Scheduling capability of analysis tasks
Visualization
  1. Drag and drop gadget IDE to visualize analyzed data with zero code
  2. Capability to plug in additional UI elements and Data sources to Gadget IDE
  3. Google gadgets based dashboard

We welcome to use this and provide feed back ahead of the major release in Q1/Q2 2012.
Keys available at [2], [3].