How to extract dynamic table rows with loop & stop when reach a specific condition

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Wed Sep 14, 2022 11:43 am

Google Chrome: Version 105.0.5195.102 (Official Build) (64-bit)
iMacros: 2021
System: Windows 10 Pro 21H2 19044.1949

With Guru chivracq kind guidance from my earlier post viewtopic.php?f=7&t=32028, i'm able to extract required data with unique element style="background-color: orange;" by using 'Double Relative Positioning' instead of using Xpath.

Since the data table is dynamic with 15mins refresh interval, the total of rows will change from each refresh. I've tried to search for solution & study eval() from the forum posts (especially Guru chivracq solutions) but i'm still not able to grasp the technic somehow :(

Here again, I hope to have more guidance to find the solution.

Questions:-
1. Is there a way to make a loop instead of the way i'm using as per code below eg. 'Extract 1st rows, 'Extract 2nd rows and so on?
2. The extraction should be stopped when reached <tr class="montage_atm_bottom even", how can i achieve this?

My current code:-

Code: Select all

SET !EXTRACT NULL
TAG POS=1 TYPE=TD ATTR=STYLE:"background-color: orange;"
TAG POS=R-1 TYPE=* ATTR=* 


'Extract 1st rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 2nd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 3rd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 4th rows...
'Extract 5th rows...

Last edited by birdchew on Fri Sep 16, 2022 5:41 am, edited 1 time in total.
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 3:28 am

get Enterprise Edition trial
and use VBS

or

if website works in older browser
Use javascript with
Palemoon 28.1
iMacros 8.9.7
Available for custom iim, javascript iMacros scripts
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 3:45 am

techimac wrote:
Fri Sep 16, 2022 3:28 am
get Enterprise Edition trial
and use VBS

or

if website works in older browser
Use javascript with
Palemoon 28.1
iMacros 8.9.7
Hi techimac, firstly thanks for your advise above. Is there any possibilities solving my 2 questions above by using iMacros Browser without VBS or JS(with Palemoon)? the reason is i'm just a beginner in this and if by using VBS or JS is the only way, i'll have to study those all over again :(
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 4:34 am

with iim you have 2 options
since your 1st row has different HTML
so you can't do loop

1st option to run in loop from Row 2 onwards

TAG POS={{!loop}} TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

try this in a loop,specify any number
But you will miss your 1st row data.

other option is to add more repetitive code for many rows without Errorignore to Yes and see if it stops when element is not found which I doubt.

macro can be forced to stop in iim but you have to add code at every extraction
and I have to find that code as well.
Available for custom iim, javascript iMacros scripts
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 4:38 am

Tell website
I will check if it opens in Palemoon

if yes then I will convert the code to js.
Available for custom iim, javascript iMacros scripts
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 4:39 am

Code: Select all

TAG POS=1 TYPE=TD ATTR=CLASS:descriptorcell
Is this code working for 2nd Row?
Can you check?
Available for custom iim, javascript iMacros scripts
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 4:50 am

try these 2 codes
You have to add more rows

Code: Select all

SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !TIMEOUT_STEP 0

TAG POS=1 TYPE=TD ATTR=STYLE:"background-color: orange;"
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
SET !EXTRACT NULL

'Extract 1st rows
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 2nd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
SET !VAR1 EVAL("if('{{!extract}}' == '#EANF#') MacroError('No More Rows');")
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 3rd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
SET !VAR1 EVAL("if('{{!extract}}' == '#EANF#') MacroError('No More Rows');")
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv
'Extract 4th rows...
'Extract 5th rows...

Code: Select all

SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !TIMEOUT_STEP 0

TAG POS=1 TYPE=TD ATTR=STYLE:"background-color: orange;"
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
SET !EXTRACT NULL

'Extract 1st rows
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 2nd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 3rd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 4th rows...
'Extract 5th rows...
Available for custom iim, javascript iMacros scripts
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 5:32 am

techimac wrote:
Fri Sep 16, 2022 4:34 am
with iim you have 2 options
since your 1st row has different HTML
so you can't do loop

1st option to run in loop from Row 2 onwards

TAG POS={{!loop}} TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

try this in a loop,specify any number
But you will miss your 1st row data.

other option is to add more repetitive code for many rows without Errorignore to Yes and see if it stops when element is not found which I doubt.

macro can be forced to stop in iim but you have to add code at every extraction
and I have to find that code as well.
try this in a loop,specify any number
But you will miss your 1st row data.
Good to hear from you again techimac :) . Basically, all the paragraphs will be the same (1st row will be the same too, i'll make the code amendment for easier reference in a while). My earlier code with this line TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT is supposed to function as TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell for the 'Extract 1st rows paragragh. Therefore 'Extract 1st rows paragragh is left without TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell.
other option is to add more repetitive code for many rows without Errorignore to Yes and see if it stops when element is not found which I doubt.
i'm partially practicing this method for the time being BUT i need the code to stop when extraction reached <tr class="montage_atm_bottom even". Otherwise, the code will extract until the end of table rows which the data is not required.
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 5:36 am

techimac wrote:
Fri Sep 16, 2022 4:38 am
Tell website
I will check if it opens in Palemoon

if yes then I will convert the code to js.
the URL is a paid subscription service. Do let me know if you need to have access to it.
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 5:38 am

techimac wrote:
Fri Sep 16, 2022 4:39 am

Code: Select all

TAG POS=1 TYPE=TD ATTR=CLASS:descriptorcell
Is this code working for 2nd Row?
Can you check?
just realized that you have just provided 2 other solutions a moment ago. Will give them a try and revert ASAP. TQ TQ techimac
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 5:47 am

i'm partially practicing this method for the time being BUT i need the code to stop when extraction reached <tr class="montage_atm_bottom even". Otherwise, the code will extract until the end of table rows which the data is not required.
You will never reach automatically.

You have to do the extraction at every step.
But since you are not using TR
you have to find some other logic
use the ones I shared
but since I dont know table structure, I can't be sure what will work for you.
I just followed your sample code and gave 2 options.


if all your rows have same code then
you can just run in a loop
and use macroerror example to stop
Available for custom iim, javascript iMacros scripts
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 5:54 am

techimac wrote:
Fri Sep 16, 2022 4:50 am
try these 2 codes
You have to add more rows

Code: Select all

SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !TIMEOUT_STEP 0

TAG POS=1 TYPE=TD ATTR=STYLE:"background-color: orange;"
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
SET !EXTRACT NULL

'Extract 1st rows
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 2nd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
SET !VAR1 EVAL("if('{{!extract}}' == '#EANF#') MacroError('No More Rows');")
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 3rd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
SET !VAR1 EVAL("if('{{!extract}}' == '#EANF#') MacroError('No More Rows');")
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv
'Extract 4th rows...
'Extract 5th rows...

Code: Select all

SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !TIMEOUT_STEP 0

TAG POS=1 TYPE=TD ATTR=STYLE:"background-color: orange;"
TAG POS=R-1 TYPE=* ATTR=* EXTRACT=TXT
SET !EXTRACT NULL

'Extract 1st rows
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 2nd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 3rd rows
TAG POS=R1 TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv

'Extract 4th rows...
'Extract 5th rows...

I believed the above proposed 2 solutions will not be ideal for my current requirement since I need the code to stop when extraction reached <tr class="montage_atm_bottom even">.
birdchew
Posts: 10
Joined: Sun Aug 28, 2022 2:37 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by birdchew » Fri Sep 16, 2022 6:01 am

techimac wrote:
Fri Sep 16, 2022 5:47 am
i'm partially practicing this method for the time being BUT i need the code to stop when extraction reached <tr class="montage_atm_bottom even". Otherwise, the code will extract until the end of table rows which the data is not required.
You will never reach automatically.

You have to do the extraction at every step.
But since you are not using TR
you have to find some other logic
use the ones I shared
but since I dont know table structure, I can't be sure what will work for you.
I just followed your sample code and gave 2 options.


if all your rows have same code then
you can just run in a loop
and use macroerror example to stop
your kind advise fully noted. I shall try using your EVAL() with MacroError() method to test further whether will the extraction stop when reached <tr class="montage_atm_bottom even">. Your effort and precious time given greatly appreciated. Thanks again Guru techimac :wink:
techimac
Posts: 466
Joined: Fri Feb 20, 2015 9:27 pm

Re: How to extract dynamic table rows with loop & stop when reach a specific condition

Post by techimac » Fri Sep 16, 2022 6:32 am

your kind advise fully noted. I shall try using your EVAL() with MacroError() method to test further whether will the extraction stop when reached <tr class="montage_atm_bottom even">. Your effort and precious time given greatly appreciated. Thanks again Guru techimac :wink:
you will not reach as you are not doing TR extraction in shared code.
if using TD for all rows

just check 1st TD extraction
if its eanf then force stop

run in loop

Code: Select all

SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !TIMEOUT_STEP 0

TAG POS={{!loop}} TYPE=TD ATTR=CLASS:descriptorcell
TAG POS=R1 TYPE=A ATTR=CLASS:montagelink EXTRACT=TXT
SET !VAR1 EVAL("if('{{!extract}}' == '#EANF#') MacroError('No More Rows');")

TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
TAG POS=R1 TYPE=TD ATTR=CLASS:rightcelltd<SP>volcell EXTRACT=TXT
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\Desktop FILE=test_{{!NOW:yymmdd_hhnn}}.csv
You have to choose static file name as its dynamic and will have many output files.
Available for custom iim, javascript iMacros scripts
Post Reply