Data extraction from .CSV -> export/submission on webpage.

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
jc9419
Posts: 5
Joined: Mon May 15, 2017 3:26 pm

Data extraction from .CSV -> export/submission on webpage.

Post by jc9419 » Thu Aug 10, 2017 3:23 pm

System info:
1. VERSION BUILD=9.030808 RECORDER=FX
2. Windows 10 - English
3. Firefox 54.0.1
4. The macro works okay, it just doesn't have all the features I need.
5. No problems
6. No problems
7. Just need help with the macro.

I am attempting to submit a large amount of zip codes (for each state in the U.S.), and would like to export data from my .CSV file and loop the function to run through the single column of data.

----------------------------------------------------------------------------------------------------------------------------------------------

Here is my example:
SET !DATASOURCE zips.csv
SET !DATASOURCE_COLUMNS 1
SET !ERRORIGNORE YES
'
'
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:agentProfileForm ATTR=ID:serviceAreaZip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON FORM=ID:agentProfileForm ATTR=ID:zips

----------------------------------------------------------------------------------------------------------------------------------------------
The function works partially (I am open to any, and all suggestions). Furthermore, after I type in the zip code, the website displays a "X reps found in this area." I was curious to see if it is possible to basically export a number from the answer "X" and import it into the 2nd column of my .CSV file.

To summarize it, I am looking to critique my macro to make sure I have set it up correctly, and to also seek assistance on how to export an answer from the submission on the website to the 2nd column in my .CSV file. I appreciate any assistance that I may receive.

(Pic of the result whenever it finds reps)
https://drive.google.com/file/d/0B-7PKC ... 0txMF92bmM

(Pic of the result whenever it doesn't find reps)
https://drive.google.com/open?id=0B-7PK ... WUtUzRESGM

EDIT/UPDATE:
After some consideration, I have gotten to the point where I am able to extract the text into the .CSV file but I am having difficulty trying to choose which column to output the data into. It seems to be that the input isn't the problem, just the export from the text extraction to the input of the extracted data into the .CSV.
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Data extraction from .CSV -> export/submission on webpag

Post by chivracq » Thu Aug 10, 2017 6:01 pm

jc9419 wrote:System info:

Code: Select all

1. VERSION BUILD=9.030808 RECORDER=FX
2. Windows 10 - English
3. Firefox 54.0.1
4. The macro works okay, it just doesn't have all the features I need.
5. No problems
6. No problems
7. Just need help with the macro.

I am attempting to submit a large amount of zip codes (for each state in the U.S.), and would like to export data from my .CSV file and loop the function to run through the single column of data.

----------------------------------------------------------------------------------------------------------------------------------------------

Here is my example:

Code: Select all

SET !DATASOURCE zips.csv
SET !DATASOURCE_COLUMNS 1
SET !ERRORIGNORE YES
'
'
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:agentProfileForm ATTR=ID:serviceAreaZip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON FORM=ID:agentProfileForm ATTR=ID:zips
----------------------------------------------------------------------------------------------------------------------------------------------
The function works partially (I am open to any, and all suggestions). Furthermore, after I type in the zip code, the website displays a "X reps found in this area." I was curious to see if it is possible to basically export a number from the answer "X" and import it into the 2nd column of my .CSV file.

To summarize it, I am looking to critique my macro to make sure I have set it up correctly, and to also seek assistance on how to export an answer from the submission on the website to the 2nd column in my .CSV file. I appreciate any assistance that I may receive.

(Pic of the result whenever it finds reps)
https://drive.google.com/file/d/0B-7PKC ... 0txMF92bmM

(Pic of the result whenever it doesn't find reps)
https://drive.google.com/open?id=0B-7PK ... WUtUzRESGM

EDIT/UPDATE:
After some consideration, I have gotten to the point where I am able to extract the text into the .CSV file but I am having difficulty trying to choose which column to output the data into. It seems to be that the input isn't the problem, just the export from the text extraction to the input of the extracted data into the .CSV.
Hum..., you didn't really follow up anymore / finish your previous Thread completely... (You are expected to share your final Script to finish a Thread "neatly" and useful for other Users...)

For this one, hum..., some other User had opened a very similar Thread recently (maybe 2 or 3 weeks ago, I think...), by Design, you can only append new Data when saving to a '.CSV' File (which could be your original DataSource), so you cannot directly save the new Data you just extracted from the Site in a 2nd Column in your File..., but you can still import the original Data from Col_1, use it on the Site for your Input, extract the Data you need, and reconstruct the Content of the '!EXTRACT' Var to rebuild your '.CSV' File (in a new File) Row by Row with the 2 Cols.

Another Technique is to use a GoogleSheet that you can access in a 2nd Tab from your Script and for which you can modify the Data directly or add an extra Col.
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
jc9419
Posts: 5
Joined: Mon May 15, 2017 3:26 pm

Re: Data extraction from .CSV -> export/submission on webpag

Post by jc9419 » Fri Aug 11, 2017 7:25 pm

I apologize for not posting this sooner. After some testing and troubleshooting, I finally got it to work exactly how I wanted. For those of you who are wanting an answer, you may find the final macro down below:

VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT
'ADD !EXTRACT {{!COL2}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1

Thanks for the help anyways chivracq :D
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Data extraction from .CSV -> export/submission on webpag

Post by chivracq » Fri Aug 11, 2017 7:47 pm

jc9419 wrote:I apologize for not posting this sooner. After some testing and troubleshooting, I finally got it to work exactly how I wanted. For those of you who are wanting an answer, you may find the final macro down below:

Code: Select all

VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT
'ADD !EXTRACT {{!COL2}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1
Thanks for the help anyways chivracq :D
Yep, very good, more or less what I meant... Except that some "SET !EXTRACT {{!COL1}}" is missing before your Extract if you want to include the Data from 'Col_1' in your final File:

Code: Select all

VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2

SET !EXTRACT {{!COL1}}
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
jc9419
Posts: 5
Joined: Mon May 15, 2017 3:26 pm

Re: Data extraction from .CSV -> export/submission on webpag

Post by jc9419 » Fri Aug 11, 2017 8:25 pm

Yes, that would be nice. However, after going through the process of using this Macro, I have found that using two separate documents to differentiate the data seems much more convenient so that is why I set it up that way. Thanks again!
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Data extraction from .CSV -> export/submission on webpag

Post by chivracq » Fri Aug 11, 2017 10:50 pm

jc9419 wrote:Yes, that would be nice. However, after going through the process of using this Macro, I have found that using two separate documents to differentiate the data seems much more convenient so that is why I set it up that way. Thanks again!
Ah OK..., "my" Solution was to answer your original Qt/Requirement where you wanted to be able to add the extracted Data in a 2nd Col (on the same Row) next to the original Input Data...

If you are "happy" with 2 Files now, that's perfect, ah-ah...! And that removes the Need for deleting the original Input File for which there was no straightforward Solution in pure '.iim'... 8)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply