How to extract a variable number

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

How to extract a variable number

by Reflex18 on Wed Mar 16, 2016 11:37 pm

Imacros VERSION BUILD=8350307
Windows 10
Google chrome Version 49.0.2623.87 m (64-bit)

So far i have a simple macro set-up to extract the price of the standard shipping from the webpage, however when i use the record setting it highlights the whole column and displays alot of unnecessary information. After the information is removed and only the price of the shipping remains i want to be able to use the macro to extract the shipping regardless of the two inputs.

If i have not explained anything well enough or you need more explanation please ask and i will attempt to add more information to help you answer the problem. I'm a new user of this forum and of marco so any help with the explanations as well would be great, i have pasted a small portion of the code below for troubleshooting.

Thanks for any help that you can provide.

VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* Extract=txt
Reflex18
 
Posts: 5
Joined: Thu Mar 10, 2016 4:29 pm

Re: How to extract a variable number

by chivracq on Thu Mar 17, 2016 5:36 am

Reflex18 wrote:
Code: Select all
Imacros VERSION BUILD=8350307
Windows 10
Google chrome Version 49.0.2623.87 m (64-bit)


So far i have a simple macro set-up to extract the price of the standard shipping from the webpage, however when i use the record setting it highlights the whole column and displays alot of unnecessary information. After the information is removed and only the price of the shipping remains i want to be able to use the macro to extract the shipping regardless of the two inputs.

If i have not explained anything well enough or you need more explanation please ask and i will attempt to add more information to help you answer the problem. I'm a new user of this forum and of marco so any help with the explanations as well would be great, i have pasted a small portion of the code below for troubleshooting.

Thanks for any help that you can provide.

Code: Select all
VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* Extract=txt

Woaw...!, compliments on the way you presented your Case, with all Required Info, FCI and Script included... It is so rare now (for new Users) that I had to mention it... :D

Which means that I was able to have a look at your Site and to do some Testing to come up with this Script:
Code: Select all
VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
'>
' Extract 'Standard Shipping' from 'DIV':
'TAG POS=1 TYPE=STRONG ATTR=TXT:Standard<SP>Shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
'PROMPT _{{Std_Shipping_Tmp}}_<BR>_{{Std_Shipping}}_
PROMPT Standard<SP>Shipping:<SP>_{{Std_Shipping}}_
(Tested on iMacros for FF v8.8.2, Pale Moon v26.1.1 (=FF44), Win10-x64.)

Hum, I will mention a bit which Functions I used outside the '[CODE]' Tags for this Thread to be found by the Forum Search Engine as the Search Engine doesn't look inside '[CODE]' Tags...! :roll:
=> 'EVAL()' + 'split()' + 'substr()' + 'trim()'.

And I tried to isolate some "smaller" HTML Element containing the Data you are interested in, using Relative Positioning as well on the 'Standard Shipping' 'STRONG' Element, but it's only contained in the containing 'DIV'.
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6479
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: How to extract a variable number

by Reflex18 on Thu Mar 17, 2016 3:39 pm

Thank you for responding to my topic and thanks for the compliments on the information provided, however in the end all i did was read the FAQ in the forums and use the information provided from there to structure my post.

I took a look at the code provided and then attempted to adjust the code save the extract to a csv file instead of showing it as a prompt however it seems to save the information before the adjustments made. I assumed it was a wrong placement of the EXTRACT=TXT but if i move that to the end of
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;") it only shows a syntax error so i'm guessing i need to look at it a different way.

VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
'>
' Extract 'Standard Shipping' from 'DIV':
'TAG POS=1 TYPE=STRONG ATTR=TXT:Standard<SP>Shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
Reflex18
 
Posts: 5
Joined: Thu Mar 10, 2016 4:29 pm

Re: How to extract a variable number

by chivracq on Thu Mar 17, 2016 4:12 pm

Reflex18 wrote:Thank you for responding to my topic and thanks for the compliments on the information provided, however in the end all i did was read the FAQ in the forums and use the information provided from there to structure my post.

I took a look at the code provided and then attempted to adjust the code save the extract to a csv file instead of showing it as a prompt however it seems to save the information before the adjustments made. I assumed it was a wrong placement of the EXTRACT=TXT but if i move that to the end of
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;") it only shows a syntax error so i'm guessing i need to look at it a different way.

Code: Select all
VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
'>
' Extract 'Standard Shipping' from 'DIV':
'TAG POS=1 TYPE=STRONG ATTR=TXT:Standard<SP>Shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv

Yep, if you want to save the "clean" Data and not the "original" "dirty' Extract like you are doing, you need to add the following Statement just before the 'SAVEAS':
Code: Select all
SET !EXTRACT {{Std_Shipping}}

So your whole Script becomes:
Code: Select all
VERSION BUILD=8350307 RECORDER=CR
URL GOTO=http://www.australiandirect.com.au/buy/kickass-12v-230ah-deep-cycle-agm-battery/KA12230
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT=6215
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=%PRESTON<SP>BEACH;WA
TAG POS=1 TYPE=BUTTON ATTR=ID:btn-calculator
'>
' Extract 'Standard Shipping' from 'DIV':
'TAG POS=1 TYPE=STRONG ATTR=TXT:Standard<SP>Shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
'>
'Save 'Standard Shipping' to a '.CSV' File:
SET !EXTRACT {{Std_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
(Not tested...)

And it's a good Practice to keep the 'PROMPT' in your Script (commented out) if you ever want to modify/reuse the 'EVAL()' Statement(s) to debug your Code...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6479
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: How to extract a variable number

by Reflex18 on Thu Mar 17, 2016 6:05 pm

chivracq wrote:Yep, if you want to save the "clean" Data and not the "original" "dirty' Extract like you are doing, you need to add the following Statement just before the 'SAVEAS':
Code: Select all
SET !EXTRACT {{Std_Shipping}}



Thankyou for the help on extracting the clean data as i was about to extract the data to a csv. I also managed to adjust the code provided from you to also include the express shipping as well and it fits into the same csv when it saves. Now my next main problem is ensuring the csv output is edited to have some sort of heading so i know the place and if it is express or standard shipping. I also have an input csv created so that it swaps to a new set of places every loop but that also appears to create a new csv each loop so i will need to figure out how to fix that as well.

However i believe that is beyond this thread so i thank you again for all the time and effort put into helping a newcomer into the world of macros. I have also pasted my ending code with the input csv part of it for anyone else that is looking at it.

Code: Select all
VERSION BUILD=8340723 RECORDER=CR
'TAB OPEN
'Tab T=2
'URL GOTO=chrome://newtab/
URL GOTO=http://www.australiandirect.com.au/buy/kickass-120ah-deep-cycle-agm-battery/KA12120
SET !DATASOURCE Shippinginput.csv
SET !DATASOURCE_COLUMNS 3
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=INPUT:TEXT ATTR=ID:zip CONTENT={{!COL2}}
WAIT SECONDS=5
TAG POS=1 TYPE=SELECT ATTR=ID:shloc_selector_cysel CONTENT=$*{{!COL3}}*
TAG POS=1 TYPE=BUTTON FORM=NAME:NoFormName ATTR=TXT:Calculate
' Standard shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Std_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Standard Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Std_Shipping EVAL("var s='{{Std_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !EXTRACT {{Std_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
' Express shipping
TAG POS=1 TYPE=DIV ATTR=TXT:Add<SP>To<SP>Cart<SP>Add<SP>To<SP>Wishlist<SP>Standard<SP>Sh* EXTRACT=TXT
SET Express_Shipping_Tmp EVAL("var s='{{!EXTRACT}}'; var x,y,z; x=s.split('Express Shipping'); y=x[1].split('- '); z=y[1].substr(0,7); z;")
SET Express_Shipping EVAL("var s='{{Express_Shipping_Tmp}}'; var z=s.trim(); z;")
SET !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
'PROMPT _{{Std_Shipping_Tmp}}_<BR>_{{Std_Shipping}}_
'PROMPT Standard<SP>Shipping:<SP>_{{Std_Shipping}}_
'PROMPT Express<SP>Shipping:<SP>_{{Express_Shipping}}_
Reflex18
 
Posts: 5
Joined: Thu Mar 10, 2016 4:29 pm

Re: How to extract a variable number

by chivracq on Thu Mar 17, 2016 6:55 pm

hum..., very good...!, you managed indeed to adapt the Code for the 'Express Shipping'...

About a different '.CSV' being saved at each Loop, yep, normal, it depends on the Format you use for '{{!NOW}}', you are currently using 'nn' (= Min) and 'ss' (= Sec) and when iMacros is executing the next Loop, you are already a few seconds further in time, so iMacros creates a new File...! If you want to reuse the same File, you need to remove the 'nnss' part in the Filename, then you will get 1 File per Hour, or you can remove the 'hh' part as well and you will get 1 File per Day, logical I would think...!

About including some Header into your '.CSV' File, there are 4 ways:
-1 You "prepare" your File by adding first the Header manually, and you save both Shippings together next to each other in one Row (in 2 Columns):
Code: Select all
'Header added manually in the File...
SET !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hh}}.csv


-2 For each Loop, one 'SAVEAS' for the Header and one 'SAVEAS' for the Shippings and the Header will be repeated for each Loop (=> Data saved in 2 Columns, 2 Rows per Loop):
Code: Select all
SET !EXTRACT "Standard Shipping"
ADD !EXTRACT "Express Shipping"
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hh}}.csv
'>
SET !EXTRACT {{Std_Shipping}}
ADD !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hh}}.csv


-3 One 'SAVEAS' per Loop, with 4 Columns:
Code: Select all
SET !EXTRACT "Standard Shipping"
ADD !EXTRACT {{Std_Shipping}}
ADD !EXTRACT "Express Shipping"
ADD !EXTRACT {{Express_Shipping}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hh}}.csv


-4 It is possible just like in Option 1 to save the Data in only 2 Columns and to let iMacros save only once (for the first Loop) the Header, instead of putting it manually into the '.CSV', but that might be a bit "advanced" for you, search my Posts on "Conditional 'SAVEAS'" if you are interested...
But I would think that Option 2 or 3 already meets your Needs...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6479
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: How to extract a variable number

by Reflex18 on Thu Mar 17, 2016 11:35 pm

chivracq wrote:About a different '.CSV' being saved at each Loop, yep, normal, it depends on the Format you use for '{{!NOW}}', you are currently using 'nn' (= Min) and 'ss' (= Sec) and when iMacros is executing the next Loop, you are already a few seconds further in time, so iMacros creates a new File...! If you want to reuse the same File, you need to remove the 'nnss' part in the Filename, then you will get 1 File per Hour, or you can remove the 'hh' part as well and you will get 1 File per Day, logical I would think...!


Thanks for information about changing the format to only have one csv file saved when i'm looping the macro. I will have to go through all the options to provide formatting to the csv file however since i'm only going to be running this macro around every week i can provide some manual formatting beforehand to the csv file and just use that as a template for each week.

Thanks for all the help you have gone about and beyond the original question and has helped me out immensely.
Reflex18
 
Posts: 5
Joined: Thu Mar 10, 2016 4:29 pm

Re: How to extract a variable number

by chivracq on Fri Mar 18, 2016 3:53 am

Reflex18 wrote:Thanks for information about changing the format to only have one csv file saved when i'm looping the macro. I will have to go through all the options to provide formatting to the csv file however since i'm only going to be running this macro around every week i can provide some manual formatting beforehand to the csv file and just use that as a template for each week.

Thanks for all the help you have gone about and beyond the original question and has helped me out immensely.

Yep, glad I could help and I'm always glad to help Users who use the Forum a bit correctly like you did.... You should read a few Pages of Threads on the Forum, best way to learn a lot quickly, even if many Threads have a very low "Quality"...

About your '.CSV' 'SAVEAS' File, if you are only going to use your Macro about once a week, reusing the same File would be indeed an Option, just make sure to remove any part of the Filename based on the Date (which will change every week obviously), unless you want to keep different Files every week for historical Data, but you could include the Timestamp as a 3rd Column in your '.CSV' as well with some Statement like:
Code: Select all
ADD !EXTRACT {{!NOW:yyyy-mm-dd_hhhnn}}
which will provide the Timestamp Info in a more easily readable Format: 2016-03-18_11h51.
Well there are many Possibilities, your Creativity is the Limit...! :wink:
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6479
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: No registered users and 3 guests

-->