Create URL from CSV and loop

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

Create URL from CSV and loop

by rdone on Tue Apr 11, 2017 8:54 am

I'm hoping someone can point me to a model to solve my issue:
As part of a scrape macro, I need to create and navigate to a series of URLs based on values in a csv file. The template URL could be:
http://www.friendlyfriends.com/friendse ... name=LNAME
where FNAME and LNAME are columns in the csv file.
While at each URL I would scrape text but for now I just want to be able to loop thru each URL.
Thanks in advance for any direction.
rdone
 
Posts: 7
Joined: Wed May 26, 2010 2:09 pm

Re: Create URL from CSV and loop

by chivracq on Tue Apr 11, 2017 9:56 am

rdone wrote:I'm hoping someone can point me to a model to solve my issue:
As part of a scrape macro, I need to create and navigate to a series of URLs based on values in a csv file. The template URL could be:
Code: Select all
http://www.friendlyfriends.com/friendsearch.aspx?firstname=FNAME&lastname=LNAME

where FNAME and LNAME are columns in the csv file.
While at each URL I would scrape text but for now I just want to be able to loop thru each URL.
Thanks in advance for any direction.

Hum, funny to see that you registered on the Forum already in 2010 but only 7 years later reach out to the Forum for the first time, ah-ah...! :D

But hum, I would think you had plenty of time to read the Forum Rules for Required Info to include when you open a Thread (or post for the first time in some existing Thread), I would think..., oops...! :shock:
=> CIM...! :mrgreen: (Read my Sig..., many Commands are not implemented for all Browsers/Versions...)

And hum, your Qt is pretty basic Use of iMacros to be honest, there is no Difficulty at all... :o
But OK, mention your FCI and I'll point you in the right Direction...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 5962
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Create URL from CSV and loop

by rdone on Tue Apr 11, 2017 10:34 am

I did register in 2010 and have been able to learn from more experienced members without needing to post until now. I did search before posting, but didn't see an immediate solution, probably because the solution is so basic that it goes without saying. But I appreciate any direction you can provide. My environment is:
1. iMacros for Firefox 9.0.3 version build 9030808
2. OS = Windows 7 (64-bit) on ThinkPad L460
3. Browser = Firefox 52.0.2 (32-bit)
4. The Loop-Csv-2-Web.iim macro appears to work
5-7. N/A
Let me know if you need more info. Thanks.
rdone
 
Posts: 7
Joined: Wed May 26, 2010 2:09 pm

Re: Create URL from CSV and loop

by chivracq on Tue Apr 11, 2017 10:54 am

rdone wrote:I did register in 2010 and have been able to learn from more experienced members without needing to post until now. I did search before posting, but didn't see an immediate solution, probably because the solution is so basic that it goes without saying. But I appreciate any direction you can provide. My environment is:
Code: Select all
1. iMacros for Firefox 9.0.3 version build 9030808
2. OS = Windows 7 (64-bit) on ThinkPad L460
3. Browser = Firefox 52.0.2 (32-bit)

4. The Loop-Csv-2-Web.iim macro appears to work
5-7. N/A
Let me know if you need more info. Thanks.

"I did register in 2010 and have been able to learn from more experienced members without needing to post until now."
=> For me actually a bit the other way around, I started using iMacros about 10 years ago approx I would think, and was always able to solve my "Scenarios" by "experimenting" and from the Wiki, and I only came to the Forum in 2013 to report a very blocking Bug (in this Thread, ah-ah...!, my first Thread on the Forum, how nostalgic...!) for which I couldn't find a Workaround, and that still has not been solved btw, but I finally/luckily found a Workaround for it a few months ago when I took the time to dig again a bit into it... (even if hum, there is still one Situation I noticed last week that my Workaround doesn't cover, grr...!, I'll have to dig into it again, but I think I know how to solve it...)

OK for your FCI, perfect...! :D I'm usually "happy" as well with this "simplified" Format:
Code: Select all
iMacros for FF v9.0.3, FF52-x32, Win7-x64
... unless some exact/specific FF Version is breaking some Functionality, sometimes Screen Resolution or Character Set and Language can be needed, but that's maybe in not even 5% of the Threads...

But, OK, you actually give yourself the Answer to your own Qt, the 'Loop-Csv-2-Web.iim' Demo-Macro is all you need, with minimal Adaptation/Editing, '!COL1' and '!COL2' will be your 'FNAME' and 'LNAME' Vars and your 'URL GOTO' Statement will become:
Code: Select all
URL GOTO=http://www.friendlyfriends.com/friendsearch.aspx?firstname={{!COL1}}&lastname={{!COL2}}
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 5962
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Create URL from CSV and loop

by rdone on Tue Apr 11, 2017 12:42 pm

The demo macro worked with revisions, and it started at the right place (row 2 to skip the headers) but only worked for the first two rows of data (rows 2-3) in the csv file. I wanted it to loop thru all rows in the file (25 in this case but it will vary). Is there a way for the macro to determine the number of rows and loop thru all of them?
rdone
 
Posts: 7
Joined: Wed May 26, 2010 2:09 pm

Re: Create URL from CSV and loop

by chivracq on Tue Apr 11, 2017 1:28 pm

rdone wrote:The demo macro worked with revisions, and it started at the right place (row 2 to skip the headers) but only worked for the first two rows of data (rows 2-3) in the csv file. I wanted it to loop thru all rows in the file (25 in this case but it will vary). Is there a way for the macro to determine the number of rows and loop thru all of them?

Well, the Default Value in the Side Panel for the Loop_Max Field is 3 indeed, so if you start at '!LOOP=2' to avoid the Header, then yep, the Macro will only run for Loops 2 and 3..., you need to specify dunno 25 or 100 depending on how many Rows you know you have in your DataSource...
And even if you don't know, you can always specify 99999 and if you don't use '!ERRORIGNORE', the Macro will automatically stop with some RuntimeError once it doesn't fund any new Row...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 5962
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Create URL from CSV and loop

by rdone on Tue Apr 11, 2017 5:18 pm

Good thing that side panel wasn't a snake or it would have bit me. Once the loop was configured for the size of the data file, everything worked as you said it would. That was Part I. Part II is to evaluate the results as one of three possible outcomes, and Part III is to update the record in the csv file with the results. I appreciate your help and will give the next a try but I would be pleasantly surprised to figure it all out on my own.
rdone
 
Posts: 7
Joined: Wed May 26, 2010 2:09 pm

Re: Create URL from CSV and loop

by chivracq on Wed Apr 12, 2017 4:06 am

rdone wrote:Good thing that side panel wasn't a snake or it would have bit me. Once the loop was configured for the size of the data file, everything worked as you said it would. That was Part I. Part II is to evaluate the results as one of three possible outcomes, and Part III is to update the record in the csv file with the results. I appreciate your help and will give the next a try but I would be pleasantly surprised to figure it all out on my own.

Ah-ah...!, don't worry, iMacros doesn't bite...! :D
But, OK, glad I could help and good luck for your further Implementations... 8)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 5962
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: No registered users and 4 guests

-->