Welcome! Log In Create A New Profile

Advanced

ETL - Extract, Transform, Load, best practice?

Posted by ArtBonds 
ETL - Extract, Transform, Load, best practice?
December 04, 2018 02:31AM
ETL... the redheaded forgotten stepchild in the project. Usually the last to be done. ETL is a one shot convert and done, bringing the old data forward, then forgotten.

What is the best practice these days? Can we just create a new WD project, copy over the analysis of the new project, then use file import to pull in one of the old files at a time (creating all fields for import, then write the code to transform the old data and save the data to new file(s)?

Or is there a good tool out there for reading the old data and putting it into new file(s)? One where we do not need to use Wx tools at all?

Thanks in advance, Art
Re: ETL - Extract, Transform, Load, best practice?
December 04, 2018 05:48AM
Hi Art,

this is a frequent problem here. No, not a real problem, it's just work. A challenging work if data originates from outside the WX-world where we try to transfer data to a csv file first and then read / insert into our files. A bit more comfortable if data has to be transferred from a WX 7.x+ file, not so easy if data is sitting in a Windev 5.5 file. We make transfer-programs, in most cases they're part of the application and called "Data import from ...". We know our competition and their file structures and we are prepared for having to transfer their data to ours not as a single event. Transfer is challenging if one can expect the source data to be broken in some way, if a no-index item becomes a unique key item, some data fields have to be "invented", some data has to be kept for future use and so on. All those eventualities have to be put into the transfer program. In most cases even the sequence of reading source data files is important. So, yes, each of those programs has to be written separately, there is no fast way ...

Kind regards,
Guenter Predl
office@windev.at
Re: ETL - Extract, Transform, Load, best practice?
December 06, 2018 11:57AM
Hi Guenter, thanks for the input. I figured there would be no easy answer to this. I did ETL conversions for our clients back turn of the century and for another company a few years later, and from what I can dig up it has not gotten any easier nor any better tools. But I had to cross my fingers and check. Thanks for your insight, much appreciated.

Best regards, Art
Author:

Your Email:


Subject:


Spam prevention:
Please, enter the code that you see below in the input field. This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right. If you enter the wrong code, a new image is created and you get another chance to enter it right.
Message: