Data extraction from .CSV -> export/submission on webpage.

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

Data extraction from .CSV -> export/submission on webpage.

by jc9419 on Thu Aug 10, 2017 8:23 am

System info:
1. VERSION BUILD=9.030808 RECORDER=FX
2. Windows 10 - English
3. Firefox 54.0.1
4. The macro works okay, it just doesn't have all the features I need.
5. No problems
6. No problems
7. Just need help with the macro.

I am attempting to submit a large amount of zip codes (for each state in the U.S.), and would like to export data from my .CSV file and loop the function to run through the single column of data.

----------------------------------------------------------------------------------------------------------------------------------------------

Here is my example:
SET !DATASOURCE zips.csv
SET !DATASOURCE_COLUMNS 1
SET !ERRORIGNORE YES
'
'
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:agentProfileForm ATTR=ID:serviceAreaZip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON FORM=ID:agentProfileForm ATTR=ID:zips

----------------------------------------------------------------------------------------------------------------------------------------------
The function works partially (I am open to any, and all suggestions). Furthermore, after I type in the zip code, the website displays a "X reps found in this area." I was curious to see if it is possible to basically export a number from the answer "X" and import it into the 2nd column of my .CSV file.

To summarize it, I am looking to critique my macro to make sure I have set it up correctly, and to also seek assistance on how to export an answer from the submission on the website to the 2nd column in my .CSV file. I appreciate any assistance that I may receive.

(Pic of the result whenever it finds reps)
https://drive.google.com/file/d/0B-7PKC ... 0txMF92bmM

(Pic of the result whenever it doesn't find reps)
https://drive.google.com/open?id=0B-7PK ... WUtUzRESGM

EDIT/UPDATE:
After some consideration, I have gotten to the point where I am able to extract the text into the .CSV file but I am having difficulty trying to choose which column to output the data into. It seems to be that the input isn't the problem, just the export from the text extraction to the input of the extracted data into the .CSV.
jc9419
 
Posts: 5
Joined: Mon May 15, 2017 8:26 am

Re: Data extraction from .CSV -> export/submission on webpag

by chivracq on Thu Aug 10, 2017 11:01 am

jc9419 wrote:System info:
Code: Select all
1. VERSION BUILD=9.030808 RECORDER=FX
2. Windows 10 - English
3. Firefox 54.0.1

4. The macro works okay, it just doesn't have all the features I need.
5. No problems
6. No problems
7. Just need help with the macro.

I am attempting to submit a large amount of zip codes (for each state in the U.S.), and would like to export data from my .CSV file and loop the function to run through the single column of data.

----------------------------------------------------------------------------------------------------------------------------------------------

Here is my example:
Code: Select all
SET !DATASOURCE zips.csv
SET !DATASOURCE_COLUMNS 1
SET !ERRORIGNORE YES
'
'
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:agentProfileForm ATTR=ID:serviceAreaZip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON FORM=ID:agentProfileForm ATTR=ID:zips


----------------------------------------------------------------------------------------------------------------------------------------------
The function works partially (I am open to any, and all suggestions). Furthermore, after I type in the zip code, the website displays a "X reps found in this area." I was curious to see if it is possible to basically export a number from the answer "X" and import it into the 2nd column of my .CSV file.

To summarize it, I am looking to critique my macro to make sure I have set it up correctly, and to also seek assistance on how to export an answer from the submission on the website to the 2nd column in my .CSV file. I appreciate any assistance that I may receive.

(Pic of the result whenever it finds reps)
https://drive.google.com/file/d/0B-7PKC ... 0txMF92bmM

(Pic of the result whenever it doesn't find reps)
https://drive.google.com/open?id=0B-7PK ... WUtUzRESGM

EDIT/UPDATE:
After some consideration, I have gotten to the point where I am able to extract the text into the .CSV file but I am having difficulty trying to choose which column to output the data into. It seems to be that the input isn't the problem, just the export from the text extraction to the input of the extracted data into the .CSV.

Hum..., you didn't really follow up anymore / finish your previous Thread completely... (You are expected to share your final Script to finish a Thread "neatly" and useful for other Users...)

For this one, hum..., some other User had opened a very similar Thread recently (maybe 2 or 3 weeks ago, I think...), by Design, you can only append new Data when saving to a '.CSV' File (which could be your original DataSource), so you cannot directly save the new Data you just extracted from the Site in a 2nd Column in your File..., but you can still import the original Data from Col_1, use it on the Site for your Input, extract the Data you need, and reconstruct the Content of the '!EXTRACT' Var to rebuild your '.CSV' File (in a new File) Row by Row with the 2 Cols.

Another Technique is to use a GoogleSheet that you can access in a 2nd Tab from your Script and for which you can modify the Data directly or add an extra Col.
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6481
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Data extraction from .CSV -> export/submission on webpag

by jc9419 on Fri Aug 11, 2017 12:25 pm

I apologize for not posting this sooner. After some testing and troubleshooting, I finally got it to work exactly how I wanted. For those of you who are wanting an answer, you may find the final macro down below:

VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT
'ADD !EXTRACT {{!COL2}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1

Thanks for the help anyways chivracq :D
jc9419
 
Posts: 5
Joined: Mon May 15, 2017 8:26 am

Re: Data extraction from .CSV -> export/submission on webpag

by chivracq on Fri Aug 11, 2017 12:47 pm

jc9419 wrote:I apologize for not posting this sooner. After some testing and troubleshooting, I finally got it to work exactly how I wanted. For those of you who are wanting an answer, you may find the final macro down below:

Code: Select all
VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT
'ADD !EXTRACT {{!COL2}}
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1


Thanks for the help anyways chivracq :D

Yep, very good, more or less what I meant... Except that some "SET !EXTRACT {{!COL1}}" is missing before your Extract if you want to include the Data from 'Col_1' in your final File:
Code: Select all
VERSION BUILD=9030808 RECORDER=FX
SET !DATASOURCE zipsforreading.csv.csv
SET !DATASOURCE_COLUMNS 1
SET !TIMEOUT_STEP 10
SET !ERRORIGNORE YES
'
'
SET !LOOP 1
SET !DATASOURCE_LINE {{!LOOP}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL1}}
WAIT SECONDS=1
TAG POS=1 TYPE=INPUT:BUTTON ATTR=ONCLICK:searchLocations()
WAIT SECONDS=2

SET !EXTRACT {{!COL1}}
TAG POS=6 TYPE=SPAN ATTR=* EXTRACT=TXT

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\User\Documents\iMacros\Datasources FILE=zipsforoutput.csv
WAIT SECONDS=1
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6481
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Data extraction from .CSV -> export/submission on webpag

by jc9419 on Fri Aug 11, 2017 1:25 pm

Yes, that would be nice. However, after going through the process of using this Macro, I have found that using two separate documents to differentiate the data seems much more convenient so that is why I set it up that way. Thanks again!
jc9419
 
Posts: 5
Joined: Mon May 15, 2017 8:26 am

Re: Data extraction from .CSV -> export/submission on webpag

by chivracq on Fri Aug 11, 2017 3:50 pm

jc9419 wrote:Yes, that would be nice. However, after going through the process of using this Macro, I have found that using two separate documents to differentiate the data seems much more convenient so that is why I set it up that way. Thanks again!

Ah OK..., "my" Solution was to answer your original Qt/Requirement where you wanted to be able to add the extracted Data in a 2nd Col (on the same Row) next to the original Input Data...

If you are "happy" with 2 Files now, that's perfect, ah-ah...! And that removes the Need for deleting the original Input File for which there was no straightforward Solution in pure '.iim'... 8)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6481
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: No registered users and 4 guests

-->