Data that is extracted does not change

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

Data that is extracted does not change

by Reflex18 on Thu Apr 28, 2016 7:04 pm

Imacros VERSION BUILD=8350307
Windows 10
Google chrome Version 50.0.2661.87 m (64-bit)

I have setup a rather long macro that extracts data from a website based on the output from a dropdown box and data input section. I require to input information that changes the output however the area that it extracts is the same.

My problem is that the information that is extracted does not change and only appears to extract the first set of inputs out of a total of five inputs.

If i have not explained anything well enough or you need more explanation please ask and i will attempt to add more information to help you answer the problem. I'm a rather new user of this forum with only my second post so any help with the explanations as well would be great, i have pasted the code below for troubleshooting.

Thanks for any help that you can provide.

Code: Select all
VERSION BUILD=8340723 RECORDER=CR
'Setting up input data and looping
SET !DATASOURCE Shippinginput.csv
SET !DATASOURCE_COLUMNS 11
'SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_PAGE 10
SET !ERRORIGNORE YES
'Varible url based on Sku name
'First area code
URL GOTO=http://www.australiandirect.com.au/?rf=kw&kw={{!COL1}}
'Outputting Sku Name
' Reset webpage bypass values to defaults
SET !ERRORIGNORE NO
TAG POS=1 TYPE=P ATTR=TXT:{{!COL1}} Extract=txt
'searching for product using Sku data
'  bypass webpage loading time
SET !TIMEOUT_PAGE 10
TAG POS=1 TYPE=IMG ATTR=SRC:http://www.australiandirect.com.au/assets/thumb/{{!COL1}}*
'Inputing postcode and suburb for first time
WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL2}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL3}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb second time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL4}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL5}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for third time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL6}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL7}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for forth time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL8}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL9}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for last time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL10}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL11}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
'Header added manually in the File...
' Reset webpage bypass values to defaults
SET !ERRORIGNORE NO
SET !EXTRACT {{!COL1}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy}}.csv
Reflex18
 
Posts: 5
Joined: Thu Mar 10, 2016 4:29 pm

Re: Data that is extracted does not change

by chivracq on Thu Apr 28, 2016 8:14 pm

Reflex18 wrote:Imacros VERSION BUILD=8350307
Windows 10
Google chrome Version 50.0.2661.87 m (64-bit)

I have setup a rather long macro that extracts data from a website based on the output from a dropdown box and data input section. I require to input information that changes the output however the area that it extracts is the same.

My problem is that the information that is extracted does not change and only appears to extract the first set of inputs out of a total of five inputs.

If i have not explained anything well enough or you need more explanation please ask and i will attempt to add more information to help you answer the problem. I'm a rather new user of this forum with only my second post so any help with the explanations as well would be great, i have pasted the code below for troubleshooting.

Thanks for any help that you can provide.

Code: Select all
VERSION BUILD=8340723 RECORDER=CR
'Setting up input data and looping
SET !DATASOURCE Shippinginput.csv
SET !DATASOURCE_COLUMNS 11
'SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_PAGE 10
SET !ERRORIGNORE YES
'Varible url based on Sku name
'First area code
URL GOTO=http://www.australiandirect.com.au/?rf=kw&kw={{!COL1}}
'Outputting Sku Name
' Reset webpage bypass values to defaults
SET !ERRORIGNORE NO
TAG POS=1 TYPE=P ATTR=TXT:{{!COL1}} Extract=txt
'searching for product using Sku data
'  bypass webpage loading time
SET !TIMEOUT_PAGE 10
TAG POS=1 TYPE=IMG ATTR=SRC:http://www.australiandirect.com.au/assets/thumb/{{!COL1}}*
'Inputing postcode and suburb for first time
WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL2}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL3}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb second time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL4}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL5}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for third time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL6}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL7}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for forth time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL8}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL9}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !ERRORIGNORE NO
'Inputing postcode and suburb for last time
'WAIT SECONDS=8
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL10}}
WAIT SECONDS=8
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%{{!COL11}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
WAIT SECONDS=4
'Outputting Standard shipping
' Ignore error incase of extract error
SET !ERRORIGNORE YES
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
' Outputting Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
'Header added manually in the File...
' Reset webpage bypass values to defaults
SET !ERRORIGNORE NO
SET !EXTRACT {{!COL1}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy}}.csv

Hum..., nice to see you back on the Forum...!

1- A few real Values are missing for your '{{!COL1}}'-'{{!COL11}}' Values to be able to test your Script even if I guess if I could still catch at least one from your previous Thread...

2- But from a quick look, we had come up with a Script in your previous Thread that was for 1 Block of Extraction, now you have 5 Blocks but you keep reusing the same 2 Vars for each Section, meaning they each time get replaced by the new Data... Either you need to give them different Names ('Std_Shipping_1' + 'Express_Shipping_1' + 'Std_Shipping_2' + etc...) and reuse those Names at the end of your Macro when you "reconstruct" the Content of '!EXTRACT' before doing the 'SAVEAS'..., or even better because if later you want to add 5 or 10 extra Sections, you can simply do it with an easy Copy&Paste, you need to use some extra Temp Var to store the Data you will want to save in the end to your '.CSV'.

3- And you need to reset '1EXTRACT' to "NULL" between each Section, i would think.

4- If you had kept the 'PROMPT' like I had told you in your previous Thread, you would be able to follow and debug your Vars...! :idea:

5- Your Use of '!ERRORIGNORE' that you toggle between ON and OFF is not needed, because a 'TAG' + 'EXTRACT' never goes wrong, if the HTML Element you are trying to tag is not found, "#EANF#" is returned by the 'EXTRACT'.

6- And you should put some empty Line between each Section, that makes your Script much easier to read.

I let you implement all the "Recommendations" I just gave you, to comply with your "... so any help with the explanations as well would be great...", it would have been quicker for me to adapt your Script directly rather than typing all this Reply, but you'll learn more indeed if you get a bit dirty yourself, ah-ah...! Good luck and I'll help you further if you get stuck...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6473
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: Majestic-12 [Bot] and 6 guests

-->