Scriptella ETL Blog

Running ETL files from Maven

Fyodor Kupolov — Mon, 16 Jan 2012 14:26:05 +0000

If you are using Maven, and your project includes dependency to scriptella-tools:

      
        com.javaforge.scriptella
        scriptella-tools
        1.0

You can always use the following command to execute various ETL files:

mvn exec:java -Dexec.mainClass="scriptella.tools.launcher.EtlLauncher" -Dexec.args="/path/to/etl/file.xml"

New feature: global variables

Fyodor Kupolov — Sun, 22 Aug 2010 22:27:29 +0000

Let’s start with a quick intro to Scriptella dataflow which is based on a concept of rows and columns(which can be treated as variables). When a query is executed it emits multiple rows making them available to nested elements. As a consequence a variable change is only visible to nested elements of the query. Here is an example to illustrate the above said:


userCount=0  



    
    SELECT COUNT(*) as userCount from Users

At times it is more convenient just to set a global variable, so that it’s value can be consumed in other places of ETL file. In Scriptella 1.0 this was possible only with a help of workarounds:

Use System.setProperty and System.getProperty to share a variable between scripts.
Another approach is similar to the technqiue utilized in anonymous inner classes to modify a single-element array declared as a final variable.The following example illustrates it:

    
    
        //Array with only one element modifiable by nested scripts
        globalVarArray = [0];
        query.next();
        
            SELECT COUNT(*) as userCount from Users

If you run the script the following output is printed on the console:



Inner script: globalVar=USER_COUNT

Outer script: globalVar=USER_COUNT

Out of scope: globalVar=0

As you can see it is possible to declare global variables in Scriptella, but this requires an additional query element and use of arrays. Scriptella 1.1 introduces support for etl.globals – a global map for variables available to ETL elements. Based on that, the example above can be rewritten to leverage the new mechanism:

    
        SELECT COUNT(*) as userCount from Users

The code became less verbose. Additionally the “out of scope” script was removed since global variables have no scope. The line globalVar=$globalVar (normal variable globalVar is not defined) was added to demonstrate that global variables do not affect normal variables, however it is not recommended for a global variable to share a name with normal variable to avoid possible misunderstandings.
And this time the output would be:

Inner script: etl.globals.globalVar=USER_COUNT

Outer script: etl.globals.globalVar=USER_COUNT

globalVar=$globalVar (normal variable globalVar is not defined)

Implementation note: As of now the etl.globals map is not shared between scripts when called by “scriptella” driver. Bug-12790 was logged to track this issue and will be resolved prior to 1.1 release.
Update 1:
Thanks to Anji for pointing out that the example with an array initialization globalVarArray = [0] will not work in Scriptella 1.0 due to lack of array instantiation support in JEXL1.1. JavaScript can be used as an alternative as explained in the FAQ entry, or you can use Janino driver to achieve the same effect:


	
	

	
	
		// Array with only one element modifiable by nested scripts
		set("globalVarArray", new int[1]);
		next();

Support for JDBC batching and query fetch has been added

Fyodor Kupolov — Sat, 14 Aug 2010 20:50:14 +0000

JDBC batching is a very important feature which allows sending multiple commands to the database in one call. Scriptella batching is controlled by statement.batchSize parameter. The value of this parameter specifies number of statements to be combined in a batch before sending it to the database.

Please note that behavior for batching depends on the type of statements processed, as the result non-prepared SQL statements (statements without ? parameters) are processed in a single batch group different from parametrized prepared SQL statements. The grouping rules are the following:

Normal(non-prepared) statements are always grouped in a single batch per ETL script element.
Parameterized prepared statement use SQL text as a group key, i.e. different statements go into different groups what may sometimes introduce an undesired behavior.

As the result mixing parameterized (prepared statements) with normal statements in a single ETL element is NOT RECOMMENDED for batch-mode since statements are processed in different batch groups and results may be unexpected.

The following 2 examples show RECOMMENDED ways of batching.
Example 1. Extract->Load using PreparedStatement.setParameters/addBatch:


   #enables batching and set batch size to 100
   statement.batchSize=100


    SELECT * from Bug

Example 2. Bulk load of non-prepared statements (without ?, but $ are allowed)

Please pay attention to the following items when use batching:

Queries are not supported in batch mode. Typically batches are intended for DB modifications, consider using a separate connection if you need querying.
Batch buffers(pending SQL commands) are flushed immediately before ETL script commits, rather than after ETL element completes. You may face this problem only while querying tables being updated, which is not recommended (see item 1).
Optimal size of batch varies between DBs and available JVM heap size, but in most cases it should not be less than 10.

Another important parameter statement.fetchSize is added to give the JDBC driver a hint about the number of rows that should be fetched from the database when more rows are needed for the result set. Examples:


   #MySQL-specific  hack
   #set fetchSize to Integer.MIN_VALUE to avoid loading all data into memory
   statement.fetchSize = -2147483648


   #For performance reasons increase fetchSize
   statement.fetchSize = 1000

New feature: dynamic include elements

Fyodor Kupolov — Sat, 05 May 2007 14:25:32 +0000

It seems obvious and natural, but it was challenging for me to add dynamic includes implementation requested by one of Scriptella users. Here is an example of dynamic includes.


    SELECT version FROM schema_info

This feature is especially useful for building Rails-like migrations. Later me or Mike will post a short how-to on a simplified migrations in Scriptella ETL.

New feature: ?{textfile ‘url/file_name’} to upload CLOBs

Fyodor Kupolov — Fri, 20 Apr 2007 19:15:50 +0000

Yesterday I’ve implemented a requested feature which allows inserting CLOB content from files. It works the same way as ?{file } for BLOBs.

A simple example:

INSERT INTOEmail (ID, Text) VALUES (1 , ?{textfile ‘email1.txt’})

Using Scriptella for In-Memory Testing

Fyodor Kupolov — Fri, 23 Mar 2007 13:08:13 +0000

I’ve started writing a small article on how-to populate/refresh a database with JUnit 4 and Spring.

In fact it’s very simple – just lookup and run a Runnable ETLExecutor registered in the application context.

Working with dates in JEXL expressions

Fyodor Kupolov — Mon, 19 Mar 2007 20:44:32 +0000

I’ve just added a useful feature to the Scriptella 0.8 code base. It allows formatting/parsing dates in JEXL expressions. JEXL lacks a possibility to instantiate Java object or directly invoke static methods. The solution is to use a helper variable named etl which provides several utility classes.

etl.date is a date format/parse helper. Here is a simple ETL to script utilizing the new feature:

In future I can add a possibility to work files, numbers etc. using the same approach, e.g. ${etl.io.listFiles(...)}