Velocity Component for Talend ODI
Talend ODI is my new favourite toy. I recently inherited a Spring Batch project for running a complex chain of XML processing, but decided to put it down after I became frustrated with the amount of work required to make changes. The project makes the best of a bad job, and using the same tools I doubt I could have done better, but here we are.
Spring Batch has a some really useful stuff in it, but it just a framework. It can do anything, but doesn’t do anything in particular, and all the minutiae are up to you. Like all Java frameworks, it offers a high level of abstraction which could be useful in certain circumstances. But most jobs for which a batch-oriented approach would useful are so messy that a proper abstractions are very difficult to extract without being tightly bound to the context of the business problem. If classes are not going to be useful outside the project, why bother with abstracting them out in the first place? Are you sure you wouldn’t be better of with a bash script?
Using a tool like Talend side-steps all this object-oriented hand-wringing because it dispenses with abstraction altogether. The interfaces between all components are defined in the tool, and nothing can be extended. You get what you get. The time you might have spent fussing with context configuration and writing base classes can be used to solve the business problem instead.
A job is defined as a set of components and connectors, and is compiled into a single Java class before being executed. It uses Java Emitter Templates (JET) to generate the executable. Components are not classes, but an XML descriptor and a JET template. The descriptor tells ODI what parameters the component has and how it may be wired into a job. The JET template generates the code that runs when the job runs.
You can export the job and all its dependencies from ODI and run it on the command line. This self-contained export is very handy because it could be run as a Jenkins job. I was not able to throw away the entire Spring Batch job, but had to proceed in stages. I removed Tasklets from the job, recreated them in Talend, and then ran the exported job as a Tasklet inside the original job. Eventually, everything will be in Talend, and run from within Jeknins, kicked off by advancing a ticket in JIRA.
A component is easy to write, once you get past the JET syntax and some quirks of ODI. The JET template does not generate a class, only chunks of code. It feels weird at first, but makes sense once you see how it all works. The easiest way to get started is to look at the templates and descriptors for all the components ODI comes with, buried down in the plugin directory (org.talend.designer.components.localprovider_*). There is also an excellent tutorial here.
I wrote a simple component to run Velocity templates, just to give it a try. You can find it here:
https://github.com/paulmcgovern/tVelocity
I guarantee this component works on my computer.