Looping question with CSV data

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
matthewjtrotter
Posts: 1
Joined: Fri Dec 28, 2018 7:20 pm

Looping question with CSV data

Post by matthewjtrotter » Fri Dec 28, 2018 7:36 pm

I have looked around to see what the *correct* way to do this is, however I haven't found a definitive answer so I figured I would poll the community

I am trying to (during each iteration of a looped macro execution), read the contents of a CSV file (over 20,000 lines and will likely exceed 100,000 at some point) and check against the dataset for a match of on-screen information. I will also need to, depending on conditions write the on-screen information into a CSV once I have read through the CSV to check for a match to ensure the record doesn't already exist.

This is for specific targeting and reporting of the macro.

Any ideas on the *right* way to put something like this together? Can it be done exclusively in IIM or do I need JS, From what I can tell, IIM and CSV can only use the current loop value to read the contents of a file, no way to read the whole contents in to perform conditional actions based on the dataset

Thanks in advance!

-M
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Looping question with CSV data

Post by chivracq » Fri Dec 28, 2018 9:31 pm

matthewjtrotter wrote:
Fri Dec 28, 2018 7:36 pm
I have looked around to see what the *correct* way to do this is, however I haven't found a definitive answer so I figured I would poll the community

I am trying to (during each iteration of a looped macro execution), read the contents of a CSV file (over 20,000 lines and will likely exceed 100,000 at some point) and check against the dataset for a match of on-screen information. I will also need to, depending on conditions write the on-screen information into a CSV once I have read through the CSV to check for a match to ensure the record doesn't already exist.

This is for specific targeting and reporting of the macro.

Any ideas on the *right* way to put something like this together? Can it be done exclusively in IIM or do I need JS, From what I can tell, IIM and CSV can only use the current loop value to read the contents of a file, no way to read the whole contents in to perform conditional actions based on the dataset

Thanks in advance!

-M
CIM...! :mrgreen: (Read my Sig... and/or the Forum Rules for Required Info to include in your OP when you open a Thread to ask a Qt...)

Interesting Qt/Scenario but that has already been asked several times and that I have already answered nearly as many times..., I'm a bit surprised you didn't manage to find relevant/similar Threads... :o

It can be done in a '.js' Script indeed (but only supported if you are on FF, => using FF v55.0.3 + iMacros for FF v8.9.7 for example, would be the "recommended" FCI if you want to use a '.js' Script), but it can also be done in pure '.iim' and is actually much more efficient (=> about 0.5 Sec for every single Check on the whole '.CSV'/DataSource (for 20,000 Rows, and maybe 0.7-1.0 Sec for 100,000 Rows)) while the '.js' Solution already might need several Minutes for one same single Check on 20,000 Rows (=> and Time x5 for 100,000 Rows)...

=> "no way to read the whole contents in to perform conditional actions based on the dataset" is not true, it can be done (in pure '.iim')...! :D

Dunno what Keywords you've used to search the Forum for your a bit vague "I have looked around to see what the *correct* way to do this is", or maybe your own Thread Title is a bit vague also if you tried to use the Keywords contained in it..., but mention your FCI and I'll point you to a few similar/relevant Threads where I had explained "my Solution" for the Functionality that you would like to implement... :wink:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply