extracting data same web but different pages

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.
Post Reply
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

extracting data same web but different pages

Post by Maxscrap » Thu Aug 02, 2018 8:21 am

Dear all,
I am evaluating the iMacros product, in general I do find it as an useful tool and before to buy it I wanted to check how powerful tool it is and here is the point, since I am trying to scrap data from a web (made with wordpress) which has (apparently) tables in hundreds of pages which are similar but not the same.

I am posting here an example of a command I did create in order to extract only a row which reports some data from one webpage:

1- TAG POS=1 TYPE=DIV ATTR=VALIGN:top EXTRACT=TXT
2- SAVEAS TYPE=EXTRACT FOLDER=* FILE=*
3- TAG POS=5 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
4- TAG POS=6 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
5- TAG POS=7 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
6- TAG POS=8 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
7- TAG POS=9 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
8- TAG POS=10 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT


When I create a LOOP, the page 2 and following ones have same header (here as above at line 1) but different positions. It makes impossible to scrap hundreds of pages, because the output comes with a .csv file with an irregular structure, not possible to use for further job.

I might be wrong using the script but in your experience, is there any solution that would help me?
Hoping to have been clear in my explanation, otherwise please post me further questions, I'll be more than glad to reply you.

p.s. I am using windows 8.1 / iMacros browser
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: extracting data same web but different pages

Post by chivracq » Thu Aug 02, 2018 2:46 pm

Maxscrap wrote:

Code: Select all

windows 8.1 / iMacros browser
Dear all,
I am evaluating the iMacros product, in general I do find it as an useful tool and before to buy it I wanted to check how powerful tool it is and here is the point, since I am trying to scrap data from a web (made with wordpress) which has (apparently) tables in hundreds of pages which are similar but not the same.

I am posting here an example of a command I did create in order to extract only a row which reports some data from one webpage:

Code: Select all

1- TAG POS=1 TYPE=DIV ATTR=VALIGN:top EXTRACT=TXT 
2- SAVEAS TYPE=EXTRACT FOLDER=* FILE=*
3- TAG POS=5 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
4- TAG POS=6 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
5- TAG POS=7 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
6- TAG POS=8 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
7- TAG POS=9 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT
8- TAG POS=10 TYPE=TD ATTR=VALIGN:top EXTRACT=TXT

When I create a LOOP, the page 2 and following ones have same header (here as above at line 1) but different positions. It makes impossible to scrap hundreds of pages, because the output comes with a .csv file with an irregular structure, not possible to use for further job.

I might be wrong using the script but in your experience, is there any solution that would help me?
Hoping to have been clear in my explanation, otherwise please post me further questions, I'll be more than glad to reply you.

p.s. I am using windows 8.1 / iMacros browser
"evaluating" + "iMacros browser" => iMB v12.0
(Always mention the exact Version...)

Yep indeed, always try to use/get the 'POS=n' as low as possible, and I'm not surprised if your 'POS=8', 'POS=9' etc will shift Fields/Cells on some later Pages...

The "Solution" would be to use 'Relative Positioning' based on the Header, with for example 'POS=R5' if the Tables contain 5 Cols, and all Pages always contain the same Cols and the same Headers of course..., and the same Nb of Cols...

If the Nb of Cols might be variable across the different Pages, you would first need to extract and "count" the Nb of Cols to reuse for the 'POS=Rn' instead of a fixed 'R5'.

If you can post a few URL's, then I can have a look..., if you don't come out by yourself...

Hum, and what about a

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=* EXTRACT=TXT
... => that will extract your whole Table with just one single Extract...? :idea:
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

Re: extracting data same web but different pages

Post by Maxscrap » Thu Aug 02, 2018 8:02 pm

Dear chivracq,
thank you so much for you attention.
Yes numbers of columns are the same for all pages fortunately. I did try your advice but it doesn't work and I've tried also to work it out with Relative Positioning but it doesn'tr extract data. Honestly I didn't use Rel.Pos. on the header but directly to each POS=R5, POS=R6, .... and soo on but no way, so my question comes up with your second advice, why to place Rel.POS on header?

The only thing that might change is the numbers of rows which might differ on some webpages.

Here we have the URL - http://www.bda-ieo.it/wordpress/?page_id=87, unlukily it is only in Italian language but you could use the following code numbers into the search box:

(these codes are linked to specifics food nutritional values tables)

- 1300
- 1502 (as you can see at the header ACIDI GRASSI, the first row below doesn't match with the same row at food code 1300)
- 1501


And voilà, I hope I have been helpful, thanks a lot for any advice.


p.s. yes, sorry I am evaluating iMacros iMB v12.0
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: extracting data same web but different pages

Post by chivracq » Fri Aug 03, 2018 2:41 am

Maxscrap wrote:Dear chivracq,
thank you so much for you attention.
Yes numbers of columns are the same for all pages fortunately. I did try your advice but it doesn't work and I've tried also to work it out with Relative Positioning but it doesn'tr extract data. Honestly I didn't use Rel.Pos. on the header but directly to each POS=R5, POS=R6, .... and soo on but no way, so my question comes up with your second advice, why to place Rel.POS on header?

The only thing that might change is the numbers of rows which might differ on some webpages.

Here we have the URL - http://www.bda-ieo.it/wordpress/?page_id=87, unlukily it is only in Italian language but you could use the following code numbers into the search box:

(these codes are linked to specifics food nutritional values tables)

- 1300
- 1502 (as you can see at the header ACIDI GRASSI, the first row below doesn't match with the same row at food code 1300)
- 1501

And voilà, I hope I have been helpful, thanks a lot for any advice.

p.s. yes, sorry I am evaluating iMacros iMB v12.0
OK, I managed to have a look at your Site, which is available in English btw from this URL, and even "better", all Pages/Tables are also available (for both Languages) directly with some direct (separate) URL by using the direct URL of the Frame, which uses for each URL the 'Food_ID'. If you have 500 Tables to extract, that will be much quicker for you as you won't have to go back to the Main Page for each new Table, and it means also less Data to load. 8)

One thing you didn't mention is what Data exactly are you going to extract...?
You didn't react to my Sugg about 'TYPE=TABLE', but if you want to extract the complete Table each time, that would also be the easiest and the quickest, with for example:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=3 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
The "best" Table seems to be on 'POS=3', there are several Tables "embedded", you can try on 'POS=1' and 'POS=2' as well. The 'PROMPT' gets truncated in my Test Env (iMacros for FF v8.8.2, Pale Moon v26.3.3 (=FF47), Win10_x64) but a 'SAVEAS' should save the whole Data I expect...
But the Cells from the 3rd Col ('Source') contain some embedded Table that you maybe don't want to extract/save...

It would be possible to save only one specific Section for example by truncating yourself the Extract on the 'TABLE' using 'EVAL()'. I only "hope" there is no Char Size Limit in 'EVAL()' as the '!EXTRACT' Var then contains a lot of Data...

And if you want to keep extracting the Data Row by Row, well Cell by Cell, to select yourself only specific Rows, then 'Relative Positioning' is indeed the way to go, for example:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
It is also possible to extract a complete Row with just one 'EXTRACT', but I guess the Data is a bit "dirty"...:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TR ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TR ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

Re: extracting data same web but different pages

Post by Maxscrap » Fri Aug 03, 2018 7:53 am

well, I have no words ...

With one string you gave me solution against hours and hours of trials, I have been using many combination but no way at all, now I see. Thank you chivracq.

Indeed this solution gave me the best output I would expect and in the fastest way.

Code: Select all

TAG POS=3 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT
But I wanted try also the other option which are not really giving me what I am looking for, the concat tables are reporting data which are not useful for my needs.
The following solution, still doesn't give output correctly because it refer it's own extraction to the first row only, if I well understand (and correct me if I am wrong), all POS=R1 will always point to the POS=1 (header) ... but my consideration might be wrong.

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_

The last option it is working but as you said the output it is a litlle "dirty"....

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TR ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_

Indeed, iMacros it is a great tool and I am going to buy it, you convinced me, thank you chivracq.
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

Re: extracting data same web but different pages

Post by Maxscrap » Fri Aug 03, 2018 8:00 am

only one question more chivracq.
How did you get the page http://www.bda-ieo.it/test/ComponentiAl ... did=1300_2 directly?
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

Re: extracting data same web but different pages

Post by Maxscrap » Fri Aug 03, 2018 8:07 am

and one more ... sorry, but how to get the following webpages tables?

In my original thought I was using a !LOOP which point to a .csv column and one by one did check in search box to open the corrispondent page. But now?
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: extracting data same web but different pages

Post by chivracq » Fri Aug 03, 2018 8:47 pm

Maxscrap wrote:well, I have no words ...

With one string you gave me solution against hours and hours of trials, I have been using many combination but no way at all, now I see. Thank you chivracq.

Indeed this solution gave me the best output I would expect and in the fastest way.

Code: Select all

TAG POS=3 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT
Yeah, well, I had already mentioned it in my first Reply actually... 8)
Maxscrap wrote:But I wanted try also the other option which are not really giving me what I am looking for, the concat tables are reporting data which are not useful for my needs.
The following solution, still doesn't give output correctly because it refer it's own extraction to the first row only, if I well understand (and correct me if I am wrong), all POS=R1 will always point to the POS=1 (header) ... but my consideration might be wrong.

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
"... because it refer it's own extraction to the first row only, if I well understand (and correct me if I am wrong), all POS=R1 will always point to the POS=1 (header)..."
=> Nope, not exactly correct, 'R1' always refers to the previous (successful) 'TAG', and if you use multiple 'R-POS' Statements, then yep, you need some 'POS=n' as 'Anchor' at some point to start the "Chain", and if any 'TAG' is not found, then the Chain is broken...! Which wouldn't happen on this Site because it is a Table, and all Rows always have the 6 Cells, except the Section Titles which only have 1 Cell per Row..., hum, and the 'Retinol equivalent' Row as well, I notice, first Row of the 'FAT SOLUBLE VITAMINS' Section, with only 1 Cell/Col as well.

But if you repeat the Anchor, 'R1' will be for the 1st Cell in the 1st Row within a Section, than 'R7' would be for the 1st Cell in the 2nd Row, etc..., by incrementing the first 'R-POS' by Modulo[6] for each Row.

And if you want to skip the embedded (fake) Tables in the Popups for Cols 3 & 5 ('Source' & 'Class'), you can simply skip them:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R2 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R2 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R2 TYPE=TD ATTR=TXT:* EXTRACT=TXT
TAG POS=R2 TYPE=TD ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
(Not tested...)

It would be possible to extract only the "AA"/"86"/"G"/"AF"/etc for those 2 Cols, but hum..., it will become a little bit more complex for Col5 which is not always populated, then you would need to build some extra-Check to extract that Col only when it is populated, or 'POS=R1' will "stupidly" simply catch the next one it will be able to catch, which might be already 10 Rows below, ah-ah...!
Maxscrap wrote:The last option it is working but as you said the output it is a litlle "dirty"....

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 1
TAB T=1
URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid=1300_2

TAG POS=1 TYPE=TD ATTR=TXT:FATTY<SP>ACIDS
'>
SET !EXTRACT NULL
TAG POS=R1 TYPE=TR ATTR=TXT:* EXTRACT=TXT
PROMPT _{{!EXTRACT}}_
Yep indeed...
Maxscrap wrote:Indeed, iMacros it is a great tool and I am going to buy it, you convinced me, thank you chivracq.
Simply glad to help on the Forum, I only get 0.005% from the Sales they (=Ipswitch) make thanks to the Forum...! 8)
(Is a Joke of course, I'm a pure "Volunteer", not affiliated with "the Company", ah-ah...! (And I bash them regularly, ah-ah...! :shock: ))

But, hum, from using iMB, it contains some Extraction Wizard that might have some more powerful Options, and on iMB you might want to have a Look at the '!ENDOFPAGE' Command which could be useful for you...
(I've never used iMB, so I never had a Chance to play with that Command which is only supported on iMB (+ IE maybe I think), and I have some "Workaround" for myself, but '!ENDOFPAGE' is probably easier to use...)
Maxscrap wrote:only one question more chivracq.
How did you get the page http://www.bda-ieo.it/test/ComponentiAl ... did=1300_2 directly?
I explained already a bit in my previous Reply, looking at the Source Code, hum, Data in a Frame, => 'Open Frame in New Tab', with direct URL.
Maxscrap wrote:and one more ... sorry, but how to get the following webpages tables?

In my original thought I was using a !LOOP which point to a .csv column and one by one did check in search box to open the corrispondent page. But now?
Yep, that's exactly the "Aim of the Game", instead of using '{{!COL1}}' to fill in the 'Food_ID' on the Main Search Page, you will use it in the URL directly: 8)

Code: Select all

URL GOTO=http://www.bda-ieo.it/test/ComponentiAlimento.aspx?Lan=Eng&foodid={{!COL1}}_2
... => Provided your Col_1 contains the [1300,1502,etc] Food ID's...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Maxscrap
Posts: 12
Joined: Thu Aug 02, 2018 7:57 am

Re: extracting data same web but different pages

Post by Maxscrap » Fri Aug 03, 2018 9:04 pm

Simply glad to help on the Forum, I only get 0.005% from the Sales they (=Ipswitch) make thanks to the Forum...! 8)
I will push Ipswitch to rise your % :)))

Jokes aparts, thanks again for your clear explanation, ...
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: extracting data same web but different pages

Post by chivracq » Sat Aug 04, 2018 1:20 am

Maxscrap wrote:
Simply glad to help on the Forum, I only get 0.005% from the Sales they (=Ipswitch) make thanks to the Forum...! 8)
I will push Ipswitch to rise your % :)))

Jokes aparts, thanks again for your clear explanation, ...
:wink:
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Post Reply