When creating modules to participate in Marketcetera data flows
What is a Module?
In the context of this article, a module is a logical unit of compiled code that accomplishes some user-specified goal. This might be the implementation of a trading strategy, a market data adapter, or some other purpose. In order to run your own code in the Marketcetera platform, you need to create a Module. A Module can emit data and/or consume data. Its main purpose is to participate in data flows.
To create a Module, you need to create a class that extends Module:
This Module doesn't do anything interesting yet, we'll add more behavior shortly.
What is a Data Flow?
A data flow is an organization of Module objects where data flows from one to the next. The participants and order of Modules in a data flow is established at run-time by configuration or other means. Modules can emit data, receive data, or both. In order to designate a Module as an emitter, it should implement the DataEmitter interface, likewise, the DataReceiver interface to receive data.
Here is a sample data flow that uses an existing Marketcetera Module and the new Module created above.
This diagram implies that the Market Data Provider is a DataEmitter, and My Module is a DataReceiver. The first thing we need to do is change My Module so that it can receive data.
Now My Module implements DataReceiver, which makes it capable of participating in a data flow where this module receives data. In the code above, incoming data in the data flow will be received in receiveData. Each Module instance can participate in multiple data flows, and the incoming data includes the DataFlowID which indicates which data flow this particular datum belongs to.
At this point, My Module doesn't do anything with the incoming data. Let's start adding some layers to My Module that make it do something more interesting. Let's make it pay attention to market data.
Note that data flows are untyped, which means that each Module can emit any type of data it wants and must explicitly check the type of incoming data. Future modifications to the system will allow typed data flows. For now, data flows err on the side of maximum flexibility. This implementation of My Module sifts through the data flow to pick out TradeEvents. These events are produced by the Market Data Provider at the head of the data flow.
The behavior added to My Module isn't very interesting, but can be interpreted as a template for more sophisticated behavior. We create an order when we receive a trade at the trade price less one cent. At this point, we don't yet do anything with that order. We'll tackle that next.
This iteration of My Module adds a Client object which is initialized in preStart. The Client is used to send the order we create.
How Do I Create a Data Flow?
Data flows are created at run-time. The simplest configuration of the Marketcetera Platform uses a Strategy Engine (SE) component and a Deploy Anywhere Routing Engine (DARE) component. Your Modules will exist inside the SE.
There are two primary ways a data flow can be created. The first is by describing the dataflows in a configuration file.
This command file can be passed as a command line argument to the Strategy Engine. On Linux/OSX, it looks like this:
$ bin/strategyengine.sh src/commands.txt
On Windows, the Strategy Engine batch file automatically includes src/commands.txt, so, in either case, simply edit the existing commands.txt file.
But, wait, you say, executing the above command file doesn't work!
$ bin/strategyengine.sh src/commands.txt Oct 16, 2017 8:56:40 AM java.util.prefs.FileSystemPreferences$6 run WARNING: Prefs file removed in background /home/colin/.java/.userPrefs/prefs.xml Oct 16, 2017 8:56:40 AM java.util.prefs.FileSystemPreferences$6 run WARNING: Prefs file removed in background /etc/.java/.systemPrefs/prefs.xml 2017-10-16 08:56:41,098 INFO [main] ? (:) - Strategy Engine version '3.0.12' (build 736 17584 20171016T155640113Z) 2017-10-16 08:56:43,244 INFO [main] ? (:) - Running command 'createModule' with parameters 'metc:mycompany:mymodule;myinstance'... 2017-10-16 08:56:43,250 WARN [main] ? (:) - Unable to execute command 'createModule' at line number '16' with parameters metc:mycompany:mymodule;myinstance because of error: 'Unable to find a module provider with the URN 'metc:mycompany:mymodule'. Ensure that the URN is correct and retry operation'. Continuing... 2017-10-16 08:56:43,250 INFO [main] ? (:) - Running command 'startModule' with parameters 'metc:mdata:bogus:single'... 2017-10-16 08:56:43,267 INFO [main] ? (:) - Completed command 'startModule' with result 'true'. 2017-10-16 08:56:43,267 INFO [main] ? (:) - Running command 'createDataFlow' with parameters 'metc:mdata:bogus:single;type=marketdata:symbols=AAPL:content=LATEST_TICK^metc:mycompany:mymodule:myinstance'... 2017-10-16 08:56:43,269 WARN [main] ? (:) - Unable to execute command 'createDataFlow' at line number '38' with parameters metc:mdata:bogus:single;type=marketdata:symbols=AAPL:content=LATEST_TICK^metc:mycompany:mymodule:myinstance because of error: 'Unable to find a module with URN 'metc:mycompany:mymodule:myinstance'. Ensure that the module URN is correct and retry operation'. Continuing... ^C
It doesn't work, yet, because we haven't actually built My Module and deployed it to the Strategy Engine yet.
How Do I Build Custom Modules?
The easiest way to build custom modules is to convert your Strategy Engine installation to a development directory. We use Maven to manage our build environment. If you're not sure, just use Maven for now. You'll need to install Maven and the Java JDK. You can use Oracle, Open JDK, or another JDK implementation. Make sure these tools are properly installed before you proceed.
We're going to make some tweaks to our module that will come in handy later. First, we're going to make our module a DataEmitter as well as a receiver. This will allow our module to participate in data flows that don't just terminate with out module, in other words, our module can sit in the middle of a data flow, rather than just at the end. The second tweak we're going to make is to use an existing class called AbstractDataReemitterModule as our module's superclass. This class handles the basics of managing data flows and also automatically reemits data in the data flow. The updated class now looks like this.
To build a Module, you need a few other resources. Here's what my Strategy Engine directory looks like:
$ tree src
src
├── commands.txt
└── main
├── java
│ └── com
│ └── mycompany
│ └── modules
│ ├── Messages.java
│ ├── MyModuleFactory.java
│ └── MyModule.java
└── resources
├── META-INF
│ └── services
│ └── org.marketcetera.module.ModuleFactory
└── modules_messages.properties
8 directories, 6 files
First, you'll notice the main/java and main/resources under src. This directory structure is what Maven expects to see. Next, you'll notice there are a few other files that I haven't mentioned yet. There's a ModuleFactory implementation that constructs Module instances for your Module. There are also i18n resources, Messages.java and modules_messages.properties. Those are provided to help the system identify your Module. Last, the ModuleFactory file in the resources directory publishes your Module to the system when the JAR is loaded at run-time.
There is also a file called pom.xml in the Strategy Engine directory. This file tells Maven how to build your modules.
To build your module, execute a Maven command from the Strategy Engine directory
$ mvn install [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building My Company Strategies 1.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ strategy --- [INFO] [INFO] --- svn-revision-number-maven-plugin:1.13:revision (default) @ strategy --- [INFO] inspecting directory /opt/Marketcetera-3.0.12/strategyengine [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ strategy --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:compile (default-compile) @ strategy --- [INFO] Compiling 1 source file to /opt/Marketcetera-3.0.12/strategyengine/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ strategy --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /opt/Marketcetera-3.0.12/strategyengine/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ strategy --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ strategy --- [INFO] No tests to run. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ strategy --- [INFO] Building jar: /opt/Marketcetera-3.0.12/strategyengine/target/strategy-1.0.0-SNAPSHOT.jar [INFO] [INFO] >>> maven-source-plugin:2.1.2:jar (attach-sources) > generate-sources @ strategy >>> [INFO] [INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ strategy --- [INFO] [INFO] --- svn-revision-number-maven-plugin:1.13:revision (default) @ strategy --- [INFO] inspecting directory /opt/Marketcetera-3.0.12/strategyengine [INFO] [INFO] <<< maven-source-plugin:2.1.2:jar (attach-sources) < generate-sources @ strategy <<< [INFO] [INFO] [INFO] --- maven-source-plugin:2.1.2:jar (attach-sources) @ strategy --- [INFO] Building jar: /opt/Marketcetera-3.0.12/strategyengine/target/strategy-1.0.0-SNAPSHOT-sources.jar [INFO] [INFO] --- maven-jar-plugin:2.4:test-jar (build-test-jar) @ strategy --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ strategy --- [INFO] Installing /opt/Marketcetera-3.0.12/strategyengine/target/strategy-1.0.0-SNAPSHOT.jar to /home/colin/.m2/repository/com/mycompany/strategy/1.0.0-SNAPSHOT/strategy-1.0.0-SNAPSHOT.jar [INFO] Installing /opt/Marketcetera-3.0.12/strategyengine/pom.xml to /home/colin/.m2/repository/com/mycompany/strategy/1.0.0-SNAPSHOT/strategy-1.0.0-SNAPSHOT.pom [INFO] Installing /opt/Marketcetera-3.0.12/strategyengine/target/strategy-1.0.0-SNAPSHOT-sources.jar to /home/colin/.m2/repository/com/mycompany/strategy/1.0.0-SNAPSHOT/strategy-1.0.0-SNAPSHOT-sources.jar [INFO] Installing /opt/Marketcetera-3.0.12/strategyengine/target/strategy-1.0.0-SNAPSHOT-tests.jar to /home/colin/.m2/repository/com/mycompany/strategy/1.0.0-SNAPSHOT/strategy-1.0.0-SNAPSHOT-tests.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.532 s [INFO] Finished at: 2017-10-16T09:21:41-07:00 [INFO] Final Memory: 41M/305M [INFO] ------------------------------------------------------------------------
Copy the resulting JAR file to the Strategy Engine modules directory:
$ cp target/strategy-1.0.0-SNAPSHOT.jar modules/jars
Let's run the Strategy Engine again:
$ bin/strategyengine.sh src/commands.txt 2017-10-16 08:57:14,124 INFO [main] ? (:) - Strategy Engine version '3.0.12' (build 736 17584 20171016T155713156Z) 2017-10-16 08:57:17,163 INFO [main] ? (:) - Running command 'createModule' with parameters 'metc:mycompany:mymodule;myinstance'... 2017-10-16 08:57:17,169 INFO [main] ? (:) - Completed command 'createModule' with result 'metc:mycompany:mymodule:myinstance'. 2017-10-16 08:57:17,169 INFO [main] ? (:) - Running command 'startModule' with parameters 'metc:mdata:bogus:single'... 2017-10-16 08:57:17,183 INFO [main] ? (:) - Completed command 'startModule' with result 'true'. 2017-10-16 08:57:17,183 INFO [main] ? (:) - Running command 'createDataFlow' with parameters 'metc:mdata:bogus:single;type=marketdata:symbols=AAPL:content=LATEST_TICK^metc:mycompany:mymodule:myinstance'... 2017-10-16 08:57:17,210 INFO [main] ? (:) - Completed command 'createDataFlow' with result '1'.
Congratulations, your custom module is now running! If you happen to have a Photon instance running, you'll notice that your strategy is sending orders. As the example is written, the orders are IOC (Immediate or Cancel), so they may or may not fill, but they won't stay open. You can see them in the FIX Messages view.
Is There an Easier Way?
Developing your own Module implementations is the most versatile and reusable way to participate in Marketcetera data flows. There is an easier way than the one described above. There is an existing Module type called Strategy that handles the Module and data flow infrastructure for you.
Let's recreate the strategy implemented in My Module above as a Strategy.
This code does exactly the same thing as My Module above. It can participate in data flows, request market data, and send orders. Behind the scenes, your strategy is inserted into a Module and started for you. There are several advantages to deploying your code this way. The first is that you don't have to stop and start the Strategy Engine each time you modify your code. We'll start by restarting the Strategy Engine once with a new set of commands.
startModule;metc:mdata:bogus:single createModule;metc:strategy:system;myStrategy,MyStrategy,JAVA,src/main/java/com/mycompany/strategy/MyStrategy.java,,true,metc:sink:system startModule;metc:strategy:system:myStrategy
These commands create a new strategy Module and specifies the path to the source file for the strategy, src/main/java/com/mycompany/strategy/MyStrategy.java. The source is compiled on-demand when you start the module. That means that you can change the contents of your strategy without restarting the Strategy Engine.
After you run these commands, open the Strategy Engines view in Photon and connect to your local Strategy Engine.
This view shows all running strategies. From here, you can start and stop the strategy. Each time you restart the strategy, the contents will be recompiled automatically without having to start and stop the Strategy Engine.
When Should I Use a Strategy and When Should I Use a Module?
There's no hard-and-fast rule to indicate which method you should use to add your custom code to Marketcetera. An exploration of the strengths and weaknesses of each method should help you make the decision.
| Strategy | Module | |
|---|---|---|
| Data Flow Participation | Emitter and Receiver | Emitter And/Or Receiver |
| Complexity | Less Code Required | More Code Required |
| Code Canges | Can change without restarting Strategy Engine | Must restart Strategy Engine |
| Size/Class Number Limitations | Limited to one main class, with max codebase size (JDK limitations) | No limits |
| Available Services | Limited to Strategy API | Any available services |
In short, the limiting factor tends to be code complexity. A normal production strategy will have many classes. While a single deployed strategy can contain inner classes, this may be too limiting from an aesthetic structure, even if the max class size set by the JDK is not reached.
The best approach, then, is to use a mix. The bulk of your strategy can be deployed using the above build/copy/restart method described for My Module. The code doesn't have to be designed as a Module, you can simply place classes there for your strategy to use. From there, simply deploy a small master strategy that controls starting and stopping your strategy.
7 Comments
vinod verma
Is the above article for implementing new market data adapters too apart from strategy engines
Colin DuPlantis
It could be interpreted for creating new market data adapters, yes. Take a look at the Exsim market data adapter for a sample.
vinod verma
Do you mean exsimfeedmodule.java .Correct?
Lance Hinds
Just installed MarketCetera. Is work still being done on the web based client?
Colin DuPlantis
Yes, we're still working on it. It's targeted to go along with our 4.x release. We don't have a date yet on when it will be complete as we've been very busy on a number of projects lately.
Lance Hinds
Does the exchange software have a documented API in case we wish to write our own UI?
Colin DuPlantis
That's currently a work-in-progress coincident to the 4.x release. The current exchange is running on 3.x. We can provide the current exchange API, if necessary.