xpath and variables???

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
itsbrendan
Posts: 6
Joined: Thu Dec 23, 2010 8:14 pm

xpath and variables???

Post by itsbrendan » Thu Dec 23, 2010 9:16 pm

I have determined the xpath of some data I need to capture on a webpage, the data is displayed in columns. Specifically column 5 & column 10, there is not a graphic I can use as an anchor so I'm using xpath and I'm using the value in the first column to get my xpath.

The first column is a record number and it starts at 1 and goes to some 8000 records. So 2 questions here: I can use the code I have below to read data from row 1, column 5, and row 1, column 10 and put the data into extraction.csv, but I really don't want to create 16,000 lines of code in my .iim file. Is there a way to increment the number in the "id('___#') parameter? I have tried ="id('___"+{{LOOP!}}+"')/td[5]", I'm not sure how to pull this off.

I'll just do this one question at a time. Thanks in advance.

VERSION BUILD=7031111 RECORDER=FX
TAG XPATH="id('___1')/td[5]" EXTRACT=TXT
TAG XPATH="id('___1')/td[10]" EXTRACT=TXT
' SET !EXTRACT ^^^ [was attempting to insert some sort of break between records]
TAG XPATH="id('___2')/td[5]" EXTRACT=TXT
TAG XPATH="id('___2')/td[10]" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv
siniy
Posts: 118
Joined: Sat Nov 07, 2009 7:44 pm

Re: xpath and variables???

Post by siniy » Thu Dec 23, 2010 9:42 pm

You are on the right track thinking about !LOOP

Code: Select all

!SET !LOOP 1
TAG XPATH="id('___{{!LOOP}}')/td[5]" EXTRACT=TXT
TAG XPATH="id('___{{!LOOP}}')/td[10]" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv
Play this script will loop MAX: 8000
I believe there is limit in 2000 lines for iim script, so 16000 wouldn't help
itsbrendan
Posts: 6
Joined: Thu Dec 23, 2010 8:14 pm

Re: xpath and variables???

Post by itsbrendan » Sun Dec 26, 2010 12:09 am

Thank you, thank you, thank you!! That is exactly what I needed. Woohoo!

I have one final problem. There are only 50 records displayed on each page, and after listing 50 records, I can click "next" to get to the next page of 50 records.

My Script:

VERSION BUILD=7031111 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !LOOP 1
TAG XPATH="id('___{{!LOOP}}')/td[5]" EXTRACT=TXT
TAG XPATH="id('___{{!LOOP}}')/td[10]" EXTRACT=TXT
TAG POS=1 TYPE=A ATTR=TXT:Next
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv

I can get the TAG POS=1.... ATTR=TXT:Next to get to the next page, I know that with subsequent iterations, my extraction.csv file is appended to - that's cool. So I can run a loop for 10 records, the first 10 records are inserted into my .csv, I can run it again for 10 records (because my loop counter resets), and those same 10 records are appended to the .csv file. But I really want it to go "Next" after 50 records, to get the next 50 records, and so on and so forth until I have all 8000. Automagically.

Can you think of a way for me to do this? So far I can only think to click NEXT manually and run my loop for the next 50 records; this would involve me manually setting the LOOP counter every 50 records and re-running the macro. I don't really want to update the script and click "play loop" 160 times. But I can if I have to.

Thanks for your help, it is most appreciated.
siniy
Posts: 118
Joined: Sat Nov 07, 2009 7:44 pm

Re: xpath and variables???

Post by siniy » Sun Dec 26, 2010 12:25 am

I would suggest for you looking at javascript interface for imacros. I myself was putting it off for too long. Now I'm writing a lot of scripts with javascript.

Code: Select all

SET !EXTRACT_TEST_POPUP NO
'1
SET !VAR1 1
TAG XPATH="id('___{{!VAR1}}')/td[5]" EXTRACT=TXT
TAG XPATH="id('___{{!VAR1}}')/td[10]" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv
'2
ADD !VAR1 1
TAG XPATH="id('___{{!VAR1}}')/td[5]" EXTRACT=TXT
TAG XPATH="id('___{{!VAR1}}')/td[10]" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv
'3,4,5...48,49
'50
ADD !VAR1 1
TAG XPATH="id('___{{!VAR1}}')/td[5]" EXTRACT=TXT
TAG XPATH="id('___{{!VAR1}}')/td[10]" EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=c:\temp\xpath\ FILE=extraction.csv

TAG POS=1 TYPE=A ATTR=TXT:Next
You should make 50 blocks of such code. ADD !VAR1 1 is simple counter. So with each block code you move to the next line in your table. I didn't do it myself because it would be too long to post here. Set the loop MAX: equal to the pages number.
itsbrendan
Posts: 6
Joined: Thu Dec 23, 2010 8:14 pm

Re: xpath and variables???

Post by itsbrendan » Sun Dec 26, 2010 12:48 am

This is great. Thanks a ton, again. :-)

Is the javascript version the "full" version of iMacros? I need to just buy it, I'm not sure what I'm in for though. I know enough javascript to be dangerous, but you're probably right. I'll just have to figure it out. I'm kind of crippled with the firefox add-on, and I imagine the scripting version is much more powerful.

Thanks for your help!
itsbrendan
Posts: 6
Joined: Thu Dec 23, 2010 8:14 pm

Re: xpath and variables???

Post by itsbrendan » Sun Dec 26, 2010 3:01 am

turns out that after my 50 iterations, the next page starts with an xpath id of 51, so the script breaks.

I could make a script that hits "NEXT" after 50 iterations, and I could copy that 170 times and change the !VAR1 to 51, 100, etc...

So the best i could come up with is having it do all 50, then click NEXT, then update the .iim file to update !VAR1 to use the next number in the list. Its is much better than I started out with, and it is relatively quick. I just gotta stay on top of what page I'm on, I'm sure this will take a few days to complete.

But thanks for the help, you really got me out of a bind.
siniy
Posts: 118
Joined: Sat Nov 07, 2009 7:44 pm

Re: xpath and variables???

Post by siniy » Sun Dec 26, 2010 8:01 am

There is addon for firefox called autopagerlite. It allows to autoexpand paginated pages. It may help you get all your pages on one page.
Javascript interface works in firefox and its free. Its not that hard. If you know in general how to assign variables, use loops and functions that it will be easy. Look at imacros documentation and examples.
Post Reply