In this talk, we will describe a business need for data collection and streaming that led to implementation of a data-streaming solution for delivering messages from many clients to various end points including Kafka, HDFS, Mongo DB, and Splunk. The talk will walk attendees through the evolution of the solution from local use of Active MQ and relational databases to a real-time data-streaming solution: Pivotal’s Spring Extreme Data (XD).
We will cover scalability considerations, observed performance, and security considerations. We will describe why the tool was selected, the various solution patterns implemented and discarded, and the final solution topology. We will also discuss deployment considerations for the public cloud, including the secure transport of data between data centers.
