Archive for September, 2010

Dynamic pattern matching

Thursday, September 2nd, 2010

As streaming systems become an integral part of business operations, there is an increasing need to allow automated data mining of collected historical data, and feed the results of such analysis back into the real-time monitoring component. We call this the monitor-manage-mine (M3) cycle.

Consider the problem of pattern matching over streams data. Traditionally, the pattern to be detected is provided a priori as a static automaton, and any change requires a new automaton to be deployed. This can be expensive in a dynamic and responsive environment, where the pattern is being constantly adjusted based on real-time historical analysis and mining. In a recent VLDB 2010 paper, one of our contributions is to propose an alternate model where the pattern definition itself is modeled as a second stream input to the pattern matching operator. This allows us to naturally integrate any changes to the pattern definition, into the continuous query processing. This can be very useful in a highly dynamic monitor-manage-mine business cycle. Interestingly, in our prototype implementation using StreamInsight, we found that dynamic patterns add insignificant overhead to normal processing. For more details, you are welcome to read the paper.