Extract Filtered Table

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
Tamilselvan
Posts: 116
Joined: Mon Mar 07, 2016 10:49 am

Extract Filtered Table

Post by Tamilselvan » Wed Oct 13, 2021 10:35 am

Firefox 52.9.0 (32-bit)
iMacros 8.9.7
Win-10 (64-bit)

Hi,
I am extracting Table data after filtering. It does not get filtered data. its extract Table normally.

How to select item one by one in dropdown list... ">" this symbol does not work...

Code: Select all

TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%>
Please guide me...

Code: Select all

VERSION BUILD=9030808 RECORDER=FX
TAB T=1
SET !TIMEOUT_STEP 0
'URL GOTO=https://bhuvan-app2.nrsc.gov.in/mgnrega/nrega_dashboard_phase2/
TAG POS=1 TYPE=SELECT ATTR=ID:select_central CONTENT=%7
TAG POS=1 TYPE=SELECT ATTR=ID:stage CONTENT=%1
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%>
wait seconds=.5
TAG POS=1 TYPE=BUTTON ATTR=TXT:Load<SP>The<SP>Report
TAG POS=40 TYPE=SELECT ATTR=TXT:102550100 CONTENT=$100
SET Extract NULL
wait seconds=1

'Filtering Pending
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
TAG POS=2 TYPE=TD ATTR=TXT:2906015* extract = txt

SET Work_Code {{!EXTRACT}}
SET !EXTRACT NULL
SET !ERRORIGNORE YES
TAG POS=R2 TYPE=TD ATTR=TXT:* extract = txt
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=BhuvanPh2.csv
Filter-Table.jpg
chivracq
Posts: 9953
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Filtered Table

Post by chivracq » Wed Oct 13, 2021 12:05 pm

Tamilselvan wrote:
Wed Oct 13, 2021 10:35 am
Firefox 52.9.0 (32-bit)
iMacros 8.9.7
Win-10 (64-bit)

Hi,
I am extracting Table data after filtering. It does not get filtered data. its extract Table normally.

How to select item one by one in dropdown list... ">" this symbol does not work...

Code: Select all

TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%>
Please guide me...

Code: Select all

VERSION BUILD=9030808 RECORDER=FX
TAB T=1
SET !TIMEOUT_STEP 0
'URL GOTO=https://bhuvan-app2.nrsc.gov.in/mgnrega/nrega_dashboard_phase2/
TAG POS=1 TYPE=SELECT ATTR=ID:select_central CONTENT=%7
TAG POS=1 TYPE=SELECT ATTR=ID:stage CONTENT=%1
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%>
wait seconds=.5
TAG POS=1 TYPE=BUTTON ATTR=TXT:Load<SP>The<SP>Report
TAG POS=40 TYPE=SELECT ATTR=TXT:102550100 CONTENT=$100
SET Extract NULL
wait seconds=1

'Filtering Pending
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
TAG POS=2 TYPE=TD ATTR=TXT:2906015* extract = txt

SET Work_Code {{!EXTRACT}}
SET !EXTRACT NULL
SET !ERRORIGNORE YES
TAG POS=R2 TYPE=TD ATTR=TXT:* extract = txt
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=BhuvanPh2.csv
Filter-Table.jpg

This one works for me...:

Code: Select all

VERSION BUILD=8820413 RECORDER=FX
TAB T=1
URL GOTO=https://bhuvan-app2.nrsc.gov.in/mgnrega/nrega_dashboard_phase2/

SET Delay_DDLBs 0.8

TAG POS=1 TYPE=LABEL ATTR=TXT:Report<SP>:
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Stage<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:stage CONTENT=%1
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Financial<SP>Year<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:select_fy CONTENT=%2021-2022
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:State<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:state CONTENT=%29
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:District<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:district CONTENT=%2906
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Block<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:block CONTENT=%2906015
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Panchayat<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%2906015003
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Asset<SP>Category<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:asset_category CONTENT=%11
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Asset<SP>Sub<SP>Category<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:asset_sub_category CONTENT=%11008
(Tested with iMacros for FF v8.8.2, PM v26.3.3, Win10_x64.)

The DDLB's from "State" + "District" + ... and beyond need some mini-'WAIT' to have the time to get populated, => hence my "Delay_DDLBs" Var for a 'WAIT' Statement between the DDLB's, and you can "tune" that Delay yourself, it works (for me) with 1.0 Sec, but 0.5 Sec was too short, and it works also with 0.8 Sec.

>>>

EDIT:
And some mini-Trick/Advice, as you say "select item one by one in dropdown list", would be to select the Entries in the "Panchayat" DDLB by Index (instead of by Value)... :idea:
=> Starting at #2 for "Achamangalam", => #3 for "Agarakoratottai", => #4 for "Arungunam", etc..., ... until #62 for "Vedal" which is the last Entry in the DDLB.
The #1 corresponds to the "All" Entry/Label, that's why you need to start at #2 for the 1st "real" Entry in the DDLB... :!:

Code: Select all

TAG POS=1 TYPE=LABEL ATTR=TXT:Panchayat<SP>:
'TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%2906015003
'Select by Index: (Start at #2 to bypass the "All" Label/Entry. => Range=[2-62])
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=#2
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Tamilselvan
Posts: 116
Joined: Mon Mar 07, 2016 10:49 am

Re: Extract Filtered Table

Post by Tamilselvan » Wed Oct 13, 2021 12:53 pm

Thank You for valuable reply .....
I am going to extract filtered table data.... How to do....? is this correct...?

Code: Select all

'Filtering Pending
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
TAG POS=2 TYPE=TD ATTR=TXT:2906015* extract = txt

SET Work_Code {{!EXTRACT}}
SET !EXTRACT NULL
SET !ERRORIGNORE YES
TAG POS=R2 TYPE=TD ATTR=TXT:* extract = txt
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=BhuvanPh2.csv
Filter-Table.jpg
chivracq
Posts: 9953
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Filtered Table

Post by chivracq » Wed Oct 13, 2021 1:15 pm

Tamilselvan wrote:
Wed Oct 13, 2021 12:53 pm
Thank You for valuable reply .....
I am going to extract filtered table data.... How to do....? is this correct...?

Code: Select all

'Filtering Pending
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
TAG POS=2 TYPE=TD ATTR=TXT:2906015* extract = txt

SET Work_Code {{!EXTRACT}}
SET !EXTRACT NULL
SET !ERRORIGNORE YES
TAG POS=R2 TYPE=TD ATTR=TXT:* extract = txt
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=BhuvanPh2.csv
Filter-Table.jpg

"Thank You for valuable reply ....."
=> Hum, OK, but you don't really react to the Content of my "valuable" Reply...!? :?

Your Issue, as I understood it, was to manage to select an Entry in the "Panchayat" DDLB, is it working now...? :?
=> Even if I see some "Kundaiyankuppam" Entry now selected in your last Screenshot, so I "reckon" it is now working, but you could be "a bit more specific", ah-ah...! :P

>>>

"I am going to extract filtered table data.... How to do....? is this correct...?"
=> Then, hum..., well..., I don't really know, I get a different Screen with different Data than your Screenshot, from the Script I posted... :?

But, yep..., from the Script you posted in your last Reply, it looks correct to me, and I would expect it will extract the "Work Code" + the "Work Name" from the 1st Row, corresponding to the "NADEP Vermi Compost Pit for...[...]" Row, which I guess is what you want, I don't know... :|
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Tamilselvan
Posts: 116
Joined: Mon Mar 07, 2016 10:49 am

Re: Extract Filtered Table

Post by Tamilselvan » Wed Oct 13, 2021 3:03 pm

that screen shot is logged page... you can see top of right side... you have tested log out page..

Code: Select all

TAG POS=1 TYPE=LABEL ATTR=TXT:Panchayat<SP>:
'Select by Index: (Start at #2 to bypass the "All" Label/Entry. => Range=[2-62])
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=#2
The above line executes 2nd value of list....only ..if you play in loop..

I can extract data without filtered data.... Though i have filtered pending content.. its extracted/fetch the normal data... which is table of 1st row....

Again I have changed code.... and tested... the result is....

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET Delay_DDLBs 0.8

TAG POS=1 TYPE=LABEL ATTR=TXT:Report<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:select_central CONTENT=%7
WAIT SECONDS={{Delay_DDLBs}}

'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Stage<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:stage CONTENT=%1
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Financial<SP>Year<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:select_fy CONTENT=%2021-2022
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:State<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:state CONTENT=%29
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:District<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:district CONTENT=%2906
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Block<SP>:
TAG POS=1 TYPE=SELECT ATTR=ID:block CONTENT=%2906015
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=LABEL ATTR=TXT:Panchayat<SP>:
'TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=%2906015003
'Select by Index: (Start at #2 to bypass the "All" Label/Entry. => Range=[2-62])
TAG POS=1 TYPE=SELECT ATTR=ID:panchayat CONTENT=#2
WAIT SECONDS={{Delay_DDLBs}}
'>
TAG POS=1 TYPE=BUTTON ATTR=TXT:Load<SP>The<SP>Report

WAIT SECONDS={{Delay_DDLBs}}
TAG POS=40 TYPE=SELECT ATTR=TXT:102550100 CONTENT=$100

'Filtering Pending Geo Tag
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
TAG POS=2 TYPE=TD ATTR=TXT:2906015* extract = txt

SET Work_Code {{!EXTRACT}}
SET !EXTRACT NULL
SET !ERRORIGNORE YES
TAG POS=R2 TYPE=TD ATTR=TXT:* extract = txt
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
SAVEAS TYPE=EXTRACT FOLDER=* FILE=BhuvanPh2.csv
Before the Table....
Before-Table-min.jpg
After filtered data in Table..
After-Filtered Table-min.jpg
Extracted Data is....
Extracted-Data-min.jpg
chivracq
Posts: 9953
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Filtered Table

Post by chivracq » Wed Oct 13, 2021 3:29 pm

Tamilselvan wrote:
Wed Oct 13, 2021 3:03 pm
that screen shot is logged page... you can see top of right side... you have tested log out page..

I can extract data without filtered data.... Though i have filtered pending content.. its extracted/fetch the normal data... which is table of 1st row....

Alright, I'm not "completely sure" I understand what you mean / what you want..., but if I'm correct you only want to extract the Rows with "Geotag Status" = "Pending for Geotagging", which the Filter Func from the Page correctly displays ... => by hiding the "other" Rows, ah-ah...! :P , but that Data/those Rows are still "present" in the HTML Source of the Page, and iMacros therefore can still "see" them also, ah-ah...! And that's "normal (and expected!)Behaviour" for iMacros...! :!:

But then, OK, no Pb, once you understand how the Page "behaves", and iMacros based on the HTML Structure of the Page, then it's fairly easy to implement your "own" Filtering, => by simply applying an extra Level of 'R-POS', and you start "the 'R-POS' Chain" with the "Pending for Geotagging" Cell as "first Anchor", => like in...:

Code: Select all

'Filtering Pending
TAG POS=239 TYPE=INPUT:TEXT ATTR=* CONTENT=Pending<SP>for<SP>Geotagging

TAG POS=1 TYPE=TD ATTR=TXT:Pending<SP>for<SP>Geotagging
'TAG POS={{!LOOP}} TYPE=TD ATTR=TXT:Pending<SP>for<SP>Geotagging

SET !EXTRACT NULL
'TAG POS=2 TYPE=TD ATTR=TXT:2906015* EXTRACT=TXT
TAG POS=R-4 TYPE=TD ATTR=TXT:2906015* EXTRACT=TXT
SET Work_Code {{!EXTRACT}}

'SET !ERRORIGNORE YES // No Use...!
SET !EXTRACT NULL
TAG POS=R2 TYPE=TD ATTR=TXT:* EXTRACT=TXT
SET Work_Name {{!EXTRACT}}

SET !EXTRACT {{Work_Code}}[EXTRACT]{{Work_Name}}
PROMPT _{{!EXTRACT}}_
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Tamilselvan
Posts: 116
Joined: Mon Mar 07, 2016 10:49 am

Re: Extract Filtered Table

Post by Tamilselvan » Wed Oct 13, 2021 4:02 pm

Ok i give demo user id & psw.... you may understood.....
'Logged page link...
Last edited by Tamilselvan on Thu Oct 14, 2021 2:41 am, edited 1 time in total.
chivracq
Posts: 9953
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Filtered Table

Post by chivracq » Thu Oct 14, 2021 2:37 am

Tamilselvan wrote:
Wed Oct 13, 2021 4:02 pm
Ok i give demo user id & psw.... you may understood.....
'Logged page link...
https://bhuvan-app2.nrsc.gov.in/mgnrega ... quest=true#

User id : xxx
Psw : yyy

Alright, managed to log in, Login and Password recorded in my Browser Credentials, no Time "now", just back from a "late" Evening, I'm moving between 2 Aparts, I'll try to have a look tomorrow..., or when I can... :|

But yep-yep, I do get the same Screen like on your Screenshot with the 7 DDLB's after selecting "Collection Report" from the 1st DDLB => "Report" one... :D

>

I've removed the L&P from my Quote, in case you don't want to leave them "Public"... => You can remove them from your own Post if you want... :| :|

+ For iMacros Threads, you can always post some L&P or some "secret" Info "not publicly" by "reporting" any Post in the Thread, then only me (= Moderator) + Forum Admin (= @TechSup) can see your Report..., but hum, that was "the Part" you didn't understand 1 or 2 years ago when I got "a bit fed up" that you kept opening Threads in the "wrong" Sub-Forum each time, ah-ah...! :wink:
(Would be less "problematic" now, as I now can move Threads from one Sub-Forum to another one without needing to "harass" the Forum Admin every time..., which is more "relaxed" for everybody... 8) )

>

But hum..., again, you didn't react at all to anything I posted before, I'm already "pretty confident" the Script I posted in my last Reply should already work, without any Testing from my Side, some "slight" Adjustments always possible of course if I wasn't able to test myself, but pff..., should be fairly easy for you, come on...!, after 5 years using iMacros, you are not a Newbie anymore...! :o
(I "only" had 2 or 3 years "Experience" with iMacros when I joined the Forum, (and it was just "an/one Add-on" I was using at that time among 70 other Add-ons I was also using), and started directly helping other Users..., as I had for most Scenarios already tried to implement them for myself, ... and apparently succeeded, without ever needing any Help, ah-ah...! :P )
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Post Reply