Publications

Oracle EDA Event Driven Architecture Suite


Advanced tools for Event Driven Architecture.


In the last issue of Optimize, we discussed concept concerning the latest paradigm in software development: Event Driven Architecture. In this issue, we take a closer look at the suite Oracle provides as a tool for developing Even Driven applications, the main component of which is the Complex Event Processor (CEP). At the moment of writing this article, in all likelihood CEP 11gR1 has just been released. Harold Gerritsen gives us a taste of what the likely improvements will be compared to the previous version.
EDA zichtbaar in actie

Fig. 1: EDA visibly in action. Dashboard of smart energy meters.


The EDA Suite

To be able to develop relatively simple EDA applications, Oracle has created the EDA Suite, which, as we saw earlier, contains four products, to wit:

• Business Activity Monitoring (BAM)
• Oracle Service Bus (OSB)
• Business Rules (BR)
• Complex Event Processor (CEP)

Each of these products provides a piece of the puzzle in the development of EDA applications. First of all, the OSB, which provides the gateway to the outside world. Events that are recorded elsewhere are submitted to the service bus electronically, meaning preferable in the form of an XML-message. The OSB will validate the messages and, based on their content, route them to the correct entry point (for instance JMS Queue) of the CEP. The CEP is ready to process the submitted events, which means that it gathers information from the events that have occurred.
On the basis of the information that has been gathered, the CEP will then use the available knowledge to generate a flow of business events and hand it back to, for instance, the OSB. Another possibility is that the CEP completes its output by triggering a piece of business logic in the form of a so-called Plain Old Java Object (POJO). Of course, in this POJO, anything is possible, for instance addressing the Business Rules Engine, the third component of the Suite, to test the information that has been gathered against the applicable business rules.
Oracle BAM has been added to the Suite to visualize the available information for the end-user (see example in Figure 1). To avoid confusion we emphasize that Oracle CEP itself also contains a component called the ‘Visualizer’ (see Figure 6).

However, this is not meant for end-users, but as a system monitor for the controller of the EDA application in question. We see that every product in the suite is making its contribution. However, the crucial product in any EDA application is, of course, the Complex Event Processor. Because Oracle CEP is an implementation of what is known in literature as a DSMS1, we will first take a closer look.

Developing Event Processing Network in Eclipse…

Fig. 2a: Developing Event Processing Network in Eclipse….

…via a right click on a Processor component you go to the Assembly Source…

Fig. 2b: …via a right click on a Processor component you go to the Assembly Source….

…taking you to an XML editor in which you can edit the EPL rule.

Fig. 2c: …taking you to an XML editor in which you can edit the EPL rule.

DSMS


As we have seen, the important thing with Complex Event Processing is the ability to translate an intangible quantity of unrelated events into something meaningful, in other words to turn it into a more limited number of events that are relevant to the business, in response to which action can be taken. And although traditional programmer may find it hard to understand, databases have there limitations in this respect. Databases are very useful in recording data (casting it in stone, so to speak) under a very professional regime of availability, recoverability, authorization, auditing, etc., after which the data can be queried on a regular basis. If we want to use a database to support CEP, this means that we start by recording all the events that occur and then proceed to eliminate all the redundant events or aggregate the events to create relevant business events.

That’s very nice, but if this has to be done for a massive million events per seconds, it is a little easier to imagine that this is not entirely the right approach. After all, one second later there are another million events that need to be processed. The remedy for this problem is what everybody who has ever moved house knows: you start by throwing away stuff that you do not need before moving. Applying this principle to EDA means that the information has to be filtered before it is further processed or stored in a database. People who are familiar with Oracle will, of course, respond immediately: “of course you store it in a database. After all, we have Oracle Times-Ten, the in-memory database”. And they aren’t that far from the truth. However, CEP requires more than just storing and filtering. In essence, we need to be able to monitor the information flows and connect seemingly unrelated events to create relevant information, and then it turns out that speed is not the only limiting factor, but also the language that is used in an RDBMS.

SQL is insufficient when it comes to filtering and correlating information flows. Instead, we need a language with a richer vocabulary. What we really want is to use an extended version of SQL to execute realtime statements of events flows before they are stored in a database. And what is also different is that the statements have to be carried out continuously, as the events keep coming in. We need to realize that this is the exact opposite of what a RDBMS does, where the information being stored is (virtually) stable and subjected to a bombardment of SQL statements. However, in EDA applications, an SQL-like rule is defined that is then applied to all the events that keep coming in, which means that it is the rule rather than the information that is being bombarded. As we indicated earlier, a product that makes this kind of approach possible is called a Data Stream Management System (DSMS). The requirements for a good DSMS (and in parentheses the element of the EDA suite that facilitates them) can be summarized in the following six items:

• It needs to be easy to connect to all kinds of sources and event destinations (Adaptors);
• It needs to be easy to merge and split up streams of events and to monitor the content of those streams (Profiles);
• It needs to be easy to define patterns (Processor Rules) in processing stages and use them to translate complex events into relevant business rules;
• Once we have determined that an event is relevant, it has to be easy to take action within an application (OutputBean) or to visualize (Adaptor and BAM Dashboard);
• We preferable want to be able to develop all this by connecting the various stages graphically (Event Processing Network);
• Finally, we want to able to use a standard language to express the statements that can be applied to the event stream (Continuous Query Language).

CEP


Oracle’s Complex Event Processor is such a DSMS. The takeover of BEA did not make life easier for Oracle in this respect: both companies had their own CEP product. The advantage of Oracle’s CEP was that the CEP statements could be written in Continuous Query Language (CQL), a language that at the moment is becoming the market standard for DSMS’. BEA still used its own Event Processing Language (ELP). In spite of the fact that ELP was not broadly supported by the market, Oracle decided to base the 10gR3 version of the EDA Suite on BEA’s CEP product. The most important reason for doing so was probable the considerably higher market share of the BEA WebLogic Event Server, as BEA’s CEP was called before the company was taken over by Oracle. As Oracle last year put it in deliberately vague terms, the company will present Oracle CEP 11gR1 ‘in the summer of 2009’. The most important change is that the EPL engine has been replaced by/expanded into a CQL engine. In addition, the look and feel of the development tool and management monitor (the Visualizer) has been streamlined. For an example, see Figure 3(10gR3) versus Figure 5 (11gR1). Also, the tools have been made more accessible by adding, for example, a Query Wizard to define CQL rules, which means that less knowledge and expertise is required to be able to use this language.


Managing EPL Rules in the 10gR3 version of the CEP Visualizer.

Fig. 3: Managing EPL Rules in the 10gR3 version of the CEP Visualizer.

Look and feel of manageing the Event Processing Network in the 11g version of the CEP Visualizer.

Fig. 4: Look and feel of manageing the Event Processing Network in the 11g version of the CEP Visualizer.


Development and management with CEP


Now that we know what Oracle CEP can do for us in functional terms, it is time to take a peek under the hood. Actually, Oracle CEP is a very lightweight Java application that is configured in XML and that, once it is deployed, can be managed in runtime. The way Oracle CEP operates is illustrated perfectly by the various illustrations in this article. In Figures 2a through 2c, the development process is illustrated. In 10gR3, the development still took place on the Eclipse platform, as was initially the case with BPEL after the takeover of Collaxa. In conceptual terms, there are great similarity between developing for Oracle CEP and modeling an ETL (Extract-Transform-Load) flow in a data warehouse environment like Oracle’s Warehouse Builder. In the case of CEP, the ETL flow is a so-called Event Processing Network (EPN). Figure 2a presents an EPN with Adaptors, Streams, Processors and an Output Bean. An sample EPN in this figure is used to monitor the financial markets of he Euro, the Dollar and the Yen and to take action in situations where action is required by using a Javabean to activate a Java application. Developers can right click (see Figure 2b) on a Processor step to go to the XML source of this processor and modify the so-called Processor Rule (see Figure 2c). To be quite frank, in 10gR3, this is a fairly Spartan affair, because developers are expected to work directly in the XML source code, rather than being able to use wizard-like dialogues, which isn’t that much more intelligent than using Notepad for the job. In the latest version of Oracle CEP, this can, of course, also be done from JDeveloper.


Manageing CQL Rules in the 11g version of the CEP Visualizer (similar to Figure 3).

Fig. 5: Manageing CQL Rules in the 11g version of the CEP Visualizer (similar to Figure 3).

System Management Dashboard in Visualizer 11g to monitor the correct operation of the Event Server.

Fig. 6: System Management Dashboard in Visualizer 11g to monitor the correct operation of the Event Server.

CEP applications can be managed using the Visualizer (Figures 3 through 6). This web application makes it possible, for instance, to visualize an EPN that has been deployed, to then select a Processor node and, through a right click (see Figure 4), access the CQL source of the Rule in question – similar to development with Eclipse. If necessary, the rule can be disabled, which makes it possible to manage deployment selectively. Another item of the Visualizer is the Dashboard function, which the manager of an CEP application can use to carry out system management tasks, like monitoring the throughput and the latency of the event stream within the chain, like a network manager who monitors the ‘health’ of his network on the basis of the same characteristics.

CQL


We have examined most of the aspects of Oracle CEP. To complete the picture, we will now take a brief look at the query language being used, the Continuous Query Language. The documentation on OTN2,3.4 contains various whitepapers that address the many possible constructions in CQL, which is why we limit ourselves to giving a small example. What is characteristic of CQL is the possibility to aggregate, filter, merge and split, and correlate the data. Furthermore, the period within which data has to be collected can be specified, as well as the interval on which measurements have to be made. The most advanced application of CQL is that of pattern-matching. The CQL syntax used to specify such patterns is as follows:

SELECT <select-list>
FROM <stream|relation-name>
MATCH_RECOGNIZE (PARTITION BY)
MEASURES (…)
ONE | ALL ROWS PER MATCH
PATTERN <pattern-expression>
DEFINE <define-alphabets>

We can apply this construction, for example, to pattern recognition in the development of a specific share or share index. In Figure 7, we see a frequently occurring pattern in the development of shares on the share market, the so-called W-pattern, which is characterized by the fact that there are five turning points in the development of the share value, which together take on a W-shaped form. In case of such a pattern, it is in particular the period between the initial descent and the ultimate ascent, that is important.

W-Pattern that commonly occurs in share-trading.

Fig. 7: W-Pattern that commonly occurs in share-trading.

This can easily be determined using a simple CQL statement:

SELECT FIRST(x.time)
, LAST(z.time)
FROM ticker
MATCH_RECOGNIZE (ONE ROW PER MATCH PARTITION BY name
PATTERN (X+ Y+ W+ Z+)
DEFINE X AS (price < PREV(price))
Y AS (price > PREV(price))
W AS (price < PREV(price))
Z AS (price > PREV(price)))

At the end of the statement in the define clause, we see that X stands for all the values that were lower than their predecessor (in other words, values that make up the left arm of the W). By analogy, Z stands for any values that were higher than their predecessor, making up the right arm of the W. This makes it easy for the select clause to return the starting point and finish of the occurring W-pattern. In the figure, these are day 1 and day 9 respectively, and later day 12 and day 19. If we apply this statement as a CQL rule in a Processor node of an EPN, for instance like we saw in Figure 4, we manage to reduce a myriad of occurring events to a minimal flow of starting points and end stages of any ‘W’s that may occur.

Summary


Given the context of Event Driven Architecture, as discussed in the previous issue of Optimize, and the discussion of the EDA Suite in this article, we get a pretty good idea of what the EDA Suite, and in particular Oracle CEP, can do for us. Surprisingly enough, the suite is extremely user-friendly, mainly because it is in fact a combination of concept and technology that has been around much longer in the Oracle domain, for instance ETL flows, a message bus, concept from the world of Java, like POJOs and Queues, and of course Business Rules and Business Activity Monitoring. In short: any Oracle developer who has made the transition from traditional design using Designer and Developer to developing in Java will encounter little by way of difficulties, which is why the number of applications of EDA will be limited above all by the creativity of business analysts. The challenge is to see the opportunities of applying EDA in regular business processes.

In short: anything is possible! Once again, the ball is in the business court.

References

1) ReCEPtor, publication about Complex Event Processing and HP's CEP engine:
https://www.hpl.hp.com/techreports/2007/HPL-2007-176.pdf

2) Oracle SOA on OTN:
https://www.oracle.com/soa

3) Oracle EDA on OTN:
https://www.oracle.com/goto/eda,
https://www.oracle.com/technology/products/event-driven-architecture/index.html

4) Oracle CEP on OTN:
https://www.oracle.com/goto/cep





For all publications, check https://www.anewlink.nl/ict/en/publications/



(c) Nov. 2009,  A New Link bv,  www.anewlink.nl.
Want to know more?
Do you want to know more about our services? Feel free to contact us without obligations via +31 (0)316 26 90 96 or e-mail us at info@anewlink.nl
A New Link bv
Burgemeester Kemmelaan 6
6922JD Duiven The Netherlands
T +31 (0)316 26 90 96
F +31 (0)316 28 09 89