You are probably asking what is StreamInsight. Well, it is the platform for performing Complex Event Processing (CEP) from Microsoft. Actually to quote exactly what it is – it is a “platform for the continuous and incremental processing of unending sequences of events (event streams) from multiple sources with near-zero latency”.
There has been a need to deal with processing at near zero latency for some time. Many times existing products can perform at low latency and that has been acceptable, however, there are a number of situations where low latency wasn’t sufficient. That is where Complex Event Processing comes in.
According to Wikipedia CEP is defined as: primarily an event processing concept that deals with the task of processing multiple events with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes. CEP is to discover information contained in the events happening across all the layers in an organization and then analyze its impact from the macro level as “complex event” and then take subsequent action plan in real time.
Some of the scenarios that call for CEP are:
Manufacturing: Sensors on the plant floor capturing data through device controllers and aggregating data consisting of tens of thousands of events per second with requirements to act on patterns detected to ensure product quality
Web analytics: Instrument server to capture click-stream data and determine online customer behavior
Financial services: Listening to data feeds like news or stocks. Use that data to run queries looking for patterns that find opportunities to buy or sell stock
Some of the features of StreamInsight listed on the download site are:
- Derive meaningful and relevant information from data/events streams through complex patterns. These patterns can be defined using a declarative query paradigm based on well-defined streaming semantics with LINQ as query language.
- For the development of applications, adapters, and analytics, the user can rely on well-established and powerful development frameworks and tools such as .NET, LINQ, and Microsoft Visual Studio.
- The platform integrates with various data sources and sinks through input and output adapters. The framework to build domain-specific adapters utilizes a .NET API to make adoption of the platform easy. Independence between adapters and queries facilitates seamless integration of real-time and historical analysis.
- The platform architecture supports a variety of deployment options, from scenarios with a low-footprint embedded option to high-end server deployments.
- A rich set of manageability features such as a management interface, a diagnostic interface and a debugging tool are provided as part of the platform.
You can start working with the CTP by downloading it at: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=a3faa562-b6dc-4702-90c6-bf8e08df3b8b