WD14 - Loop on String vs. XML ?

I am working with files which I need to extract records from a text file (txt & csv). the files usually have 5 pieces of data (fields) per record and 1 record per line.

Recently the files have been getting bigger. the file I am currently working on has 720,000 lines in a text file and the fields are separated by TAB

I have been using a LOOP on the Line Item (separated by CR) and then using ExtractString to get the field data.

In the current file of 720,000 records I notice the process is getting slower and slower as the program gets to the higher numbered records (it is 2 minutes slower every 10,000 records). I am saving off the records every 10,000 records to a file. It will take 72 files to save all the records. The first fill took 3minutes ... the 15th file took 28 minutes & the 17th file just took 33 minutes.

QUESTION : Is this because for every time the loop finds the next line item it has to count from the beginning of the file to find the correct position?

Here is the code

nIndexlLineItem = 1
LOOP
// ExtractLineItem
sLineItem = ExtractString(sFileData,nIndexLineItem,CR)

// Extract all fields from the line item

// increment lineitem position by one
nIndexLineItem++
END

QUESTION : Is there a better way to cycle through the sFileData without starting at the beginning everytime to re-position?

QUESTION: Would an XML File allow me to cycle through the sFileData faster by using XMLNext? (I have never worked with XML so I do not know how it would work?)

Any suggestions are appreciated.

Dan,

Sorry you posted in the wrong section. But here is something which might help.

FOR EACH STRING sLineItem OF sFileData SEPARATED BY CR
	// Extract all fields from the line item
END

Hi. I think XML will be slower, because the file must me loaded in memory and parsed.

You can try the Carlo's method

FOR EACH STRING sLineItem OF sFileData SEPARATED BY CR
// Extract all fields from the line item
FOR EACH STRING sFieldItem OF sLineItem SEPARATED BY TAB
// Assign each field
....
END
END



Rubén