Home
>
WinDev Forum
>
Topic
ETL - Extract, Transform, Load, best practice?
Posted by ArtBonds
|
ETL - Extract, Transform, Load, best practice? December 04, 2018 03:31AM |
Registered: 5 years ago Posts: 11 |
ETL... the redheaded forgotten stepchild in the project. Usually the last to be done. ETL is a one shot convert and done, bringing the old data forward, then forgotten.
What is the best practice these days? Can we just create a new WD project, copy over the analysis of the new project, then use file import to pull in one of the old files at a time (creating all fields for import, then write the code to transform the old data and save the data to new file(s)?
Or is there a good tool out there for reading the old data and putting it into new file(s)? One where we do not need to use Wx tools at all?
Thanks in advance, Art
What is the best practice these days? Can we just create a new WD project, copy over the analysis of the new project, then use file import to pull in one of the old files at a time (creating all fields for import, then write the code to transform the old data and save the data to new file(s)?
Or is there a good tool out there for reading the old data and putting it into new file(s)? One where we do not need to use Wx tools at all?
Thanks in advance, Art
|
Re: ETL - Extract, Transform, Load, best practice? December 04, 2018 06:48AM |
Admin Registered: 5 years ago Posts: 501 |
Hi Art,
this is a frequent problem here. No, not a real problem, it's just work. A challenging work if data originates from outside the WX-world where we try to transfer data to a csv file first and then read / insert into our files. A bit more comfortable if data has to be transferred from a WX 7.x+ file, not so easy if data is sitting in a Windev 5.5 file. We make transfer-programs, in most cases they're part of the application and called "Data import from ...". We know our competition and their file structures and we are prepared for having to transfer their data to ours not as a single event. Transfer is challenging if one can expect the source data to be broken in some way, if a no-index item becomes a unique key item, some data fields have to be "invented", some data has to be kept for future use and so on. All those eventualities have to be put into the transfer program. In most cases even the sequence of reading source data files is important. So, yes, each of those programs has to be written separately, there is no fast way ...
Kind regards,
Guenter Predl
office@windev.at
this is a frequent problem here. No, not a real problem, it's just work. A challenging work if data originates from outside the WX-world where we try to transfer data to a csv file first and then read / insert into our files. A bit more comfortable if data has to be transferred from a WX 7.x+ file, not so easy if data is sitting in a Windev 5.5 file. We make transfer-programs, in most cases they're part of the application and called "Data import from ...". We know our competition and their file structures and we are prepared for having to transfer their data to ours not as a single event. Transfer is challenging if one can expect the source data to be broken in some way, if a no-index item becomes a unique key item, some data fields have to be "invented", some data has to be kept for future use and so on. All those eventualities have to be put into the transfer program. In most cases even the sequence of reading source data files is important. So, yes, each of those programs has to be written separately, there is no fast way ...
Kind regards,
Guenter Predl
office@windev.at
|
Re: ETL - Extract, Transform, Load, best practice? December 06, 2018 12:57PM |
Registered: 5 years ago Posts: 11 |
Hi Guenter, thanks for the input. I figured there would be no easy answer to this. I did ETL conversions for our clients back turn of the century and for another company a few years later, and from what I can dig up it has not gotten any easier nor any better tools. But I had to cross my fingers and check. Thanks for your insight, much appreciated.
Best regards, Art
Best regards, Art