Performance Tips & Reusability in Talend Data Integration

 Performance:  

          Usually to increase the speed of execution process or improving the Performance of a job, the developers should follow some performance techniques. This Performance technique is a key matrix of a Jobs’ success or failure.

We can speed up the job execution process in below following ways: 

• Global Context: Defining the context group and access context details based on the context group and appropriate context variable throughout the job and we can pass the same context to sub jobs also.



• If we have more complex job, then we can divide the job into multiple sub jobs to increase the processing time and speed.



• For the job parallel execution purpose, we can use Multithread option to increase the processing time and speed.

      (Job –> Extra –> Multithread Execution) 

• By increasing the JVM parameters is also one of the performance techniquesin talend.  

             (Allocating more memory to the jobs). 

• Use Database Bulk Components (While loading huge datasets to database from Talend Job, it is recommended to use Bulk components provided by Talend for almost all databases.)

Ex: – tMSSqlBulkExec, tMSSqlOutputBulk, tMSSqlOutputBulkExec etc.

• Use SELECT Query to retrieve data from the database.

Ex: – tOracleinput, tMySqlinput…..etc. 

• Removing   the    unnecessary    data/rows     using   tFilterColumns
• Removing the unnecessary fields/columns using tFilterColumns
• We can also increase the performance by changing the commit size and batch size.
• Avoiding the tMapcomponent for the single piece of work. It will also increase the performance and time. 

 Reusability: 

Talend provides many ways to make our code reusable. 

• Custom Code– Talend provides few custom components like tJava, tJavaRow, tJavaFlex These components are used to integrate our custom java code into a talend program. By using these Custom Code components, we can create codes for specific needs and for accessing the codes very quickly. 
• Job: With the help of tRunJob, we can call the another Job. 
• Metadata-Data that gives the information about the data is called as Meta Data. In terms of Talend open Studio, metadata refers to reusable configurations that describe the data, its attributes. 

 Once we defined the metadata, it can be used across multiple jobs. For example, if the Username or Password to an DB account changes and this DB connection is used in 5 different jobs, the details would have to be updated 5 times. However, if we store this configuration in a single metadata component, then it only needs to be updated once, no need to change in differentjobs. 

• Context Groups– We can define the contexts are User-Defined parameters. Context Variables are the values that may change from one environment to another environment like Development to Testing or Testing to Production.  
• In these cases, Talend provides the Context Group facility. Here, a group of context variables that can be added to our job or multiple jobs as a context group from the metadata section. Suppose if we change any value from context group it will reflect to all other jobs wherever we used the same context group.
• Job Lets:We can use job lets for reusable purpose. In job any piece of code we get repeatedly that piece of code put into job let.   
• SQL Templates: In Talend, we have by default some SQL Templates. If we are using Databases in our jobs, then we can create our own SQL Templates, or we can use default SQL Templates. 
• Use Repository mode for storing the information that you want to use repeatedly in multiple components or Jobs, such as a database connection. Avoid the built-in mode.

About GIRIKON:

Girikon is a reputed provider of end-to-end IT services including but not limited to Salesforce consulting, Salesforce implementation and Salesforce support.

About Author

Sreenivasulu Pujala

Sreenivas has 3 years of experience working as a Software Developer with strong knowledge in Informatica Power Center, Talend DI, Talend ESB, Alteryx, MuleSoft, Unix, Oracle & MySQL. Apart from this,he has also handled Data Intergration& ETL Devlopement projects. Sreenivas loves to spend his leisure time in reading books and blogging.