I am working with files which I need to extract records from a text file (txt & csv). the files usually have 5 pieces of data (fields) per record and 1 record per line.
Recently the files have been getting bigger. the file I am currently working on has 720,000 lines in a text file and the fields are separated by TAB
I have been using a LOOP on the Line Item (separated by CR) and then using ExtractString to get the field data.
In the current file of 720,000 records I notice the process is getting slower and slower as the program gets to the higher numbered records (it is 2 minutes slower every 10,000 records). I am saving off the records every 10,000 records to a file. It will take 72 files to save all the records. The first fill took 3minutes ... the 15th file took 28 minutes & the 17th file just took 33 minutes.
QUESTION : Is this because for every time the loop finds the next line item it has to count from the beginning of the file to find the correct position?
Here is the code
nIndexlLineItem = 1
LOOP
// ExtractLineItem
sLineItem = ExtractString(sFileData,nIndexLineItem,CR)
// Extract all fields from the line item
// increment lineitem position by one
nIndexLineItem++
END
QUESTION : Is there a better way to cycle through the sFileData without starting at the beginning everytime to re-position?
QUESTION: Would an XML File allow me to cycle through the sFileData faster by using XMLNext? (I have never worked with XML so I do not know how it would work?)
Any suggestions are appreciated.