We have been blown away by the market success we had with our business process visibility suite nJAMS over the past years. It all began with the idea that you should not be required to build your own monitoring platform for your middleware and enterprise systems of choice over and over again, but much rather have a partner who could provide an out-of-the-box solution that would address all needs in no time.

The problems with very large data sets

Once the initial version of the product was built for TIBCO ActiveMatrix BusinessWorks and shipped to our very first customer, we started to fight the woes of performance degradation when dealing with very large data sets. While the majority of our customers do very well performance-wise, over time we met more and more prospects whose sheer data volumes nJAMS would not handle easily. While the core product itself (all the clients, plugins and the server we provide) could easily consume billions of events per day, the underlying data store was turning into a bottleneck.

Traditionally we stored all our data in a relational data model and database. And a lot of our customers loved the idea of having all transactional data available for further, custom-built reporting in any arbitrary technology. However, when the amount of business transactions to monitor are growing to numbers over and beyond 100 million per day, while retaining the data for a couple of months, that traditional way of storing and retrieving data did not work out any longer.

The solution: developed with customers, tested in real life

About 18 months back we hit the road and started visiting our largest customers. Jointly we checked their data volumes, validated long-term strategies on that data and developed a vision on how to overcome the challenges of our data stores of choice, while giving maximum flexibility to the downstream use of the data. Next, we shortlisted a number of technologies and discussed the options with these key customers. The checklist included ease of setup and operations, reliability, scalability and availability, as well as access to knowledge in the market place (in other words: professional services and a strong community) and long-term TCO for our customers.

Quickly the proposed solution hovered around open-source databases and NoSQL data stores. We picked two, three candidates of each domain and put them on test in our lab environments. Testing the aforementioned key aspects of the technologies at choice, the final candidate was identified and a prototype was built.

A few beta customers helped us finalize the software – amongst others, an US east-coast health insurance, an energy company in Europe and a world-class sports & lifestyle company in Germany installed, tested and approved the new technology.

Elasticsearch – new backend-technology for large nJAMS installations

And all of that makes me very proud as today I can announce the general availability of the nJAMS Plugin for Elasticsearch with immediate effect!

Allowing nJAMS to use Elasticsearch as the primary and sole data store unleashes the true power of our business process visibility solutions. Using the new back-end technology, your nJAMS installation scales to data volumes unmatched before, and also allows to run the entire solution on an easier to manage and cheaper to operate IT stack.

Reach out to us for a demo and to understand what that new plugin can do to your installation – we are sure you will be just equally blown away as we are by the results of our 18-month-long adventure.

About the Author: Hendrik Siegeln is co-founder and Managing Director of Integration Matters.