Extract Table & Append

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
Tamilselvan
Posts: 129
Joined: Mon Mar 07, 2016 10:49 am

Extract Table & Append

Post by Tamilselvan » Sun Nov 29, 2020 3:15 pm

Firefox 52.9.0 (32-bit)
iMacros 8.9.7
Win-10 (64-bit)
Hi,
I have voucher number using this, I extract the table from website and I need to append table in loop but i could not extract as table format. Where i have to change in my code, please guide me...

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !TIMEOUT_STEP 0
SET !DATASOURCE Vr_No.csv
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
'URL GOTO=http://mnregaweb4.nic.in/netnrega/homesearch.htm
FRAME F=1
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:form1 ATTR=ID:txt_keyword2 CONTENT={{!COL1}}
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:form1 ATTR=ID:btn_go
TAB T=2

'Extract text

TAG POS=2 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT

'Remove quotes from extracted text
SET TAB EVAL("var extr2=\"{{!EXTRACT}}\"; extr2.replace(/'/g,'');")
'Save extracted data
SAVEAS TYPE=EXTRACT FOLDER=* FILE=VrTable.csv
Material List to Vr.No.-1.jpg
Material List to Vr.No.-2.jpg
Attachments
Vr_No.rar
(595 Bytes) Downloaded 175 times
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Table & Append

Post by chivracq » Sun Nov 29, 2020 7:51 pm

Tamilselvan wrote:
Sun Nov 29, 2020 3:15 pm

Code: Select all

Firefox 52.9.0 (32-bit)
iMacros 8.9.7
Win-10 (64-bit)
Hi,
I have voucher number using this, I extract the table from website and I need to append table in loop but i could not extract as table format. Where i have to change in my code, please guide me...

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !TIMEOUT_STEP 0
SET !DATASOURCE Vr_No.csv
SET !LOOP 2
SET !DATASOURCE_LINE {{!LOOP}}
'URL GOTO=http://mnregaweb4.nic.in/netnrega/homesearch.htm
FRAME F=1
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:form1 ATTR=ID:txt_keyword2 CONTENT={{!COL1}}
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:form1 ATTR=ID:btn_go
TAB T=2

'Extract text

TAG POS=2 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT

'Remove quotes from extracted text
SET TAB EVAL("var extr2=\"{{!EXTRACT}}\"; extr2.replace(/'/g,'');")
'Save extracted data
SAVEAS TYPE=EXTRACT FOLDER=* FILE=VrTable.csv
Material List to Vr.No.-1.jpg
Material List to Vr.No.-2.jpg

Hum, Screenshots made in CR while you mention FF52 ESR as your Browser in your FCI... :?

>>>

Hum, yep-yep, the 'TYPE=TABLE' + 'EXTRACT=TXT' Mechanism is not "really" compatible with looping a Script, especially on FF, as every Cell gets enclosed between Double Quotes, and the whole Table gets again/also enclosed between Double Quotes..., which will give when saving with iMacros, that each Loop = each separate Table will give one single Cell per Loop/Table when opened in 'Excel' or any Equivalent from 'OO'/'LO'...

=> 4 Approaches...:
1- Still with the "Standard" 'TYPE=TABLE' + 'EXTRACT=TXT' Mechanism, there "might be a way" to still manage to separate all Cells/Tables into separate Cells/Rows in 'Excel' by playing with all the Settings/Separators when opening/importing the '.CSV' File in(to) 'Excel'... (But I've never managed... :oops: )

2- Before opening your '.CSV' File in 'Excel', you could open it first in 'Notepad' and in just one 'Search & Replace' Action, => replace all _""_ Occurrences (=> 2 Double Quotes next to each other) with just one Double Quote (=> _"_).

3- The 'TYPE=TABLE' + 'EXTRACT=TXT' Mechanism is meant as a "readily usable" Mechanism in iMacros, it looks "simple and easy" 8) , but there is a lot of Coding behind it for that Func to work with just one easy and short Command..., and that Func has only been coded at the 'TYPE=TABLE' Level...

You could try to extract your Table Row by Row, with a 'TYPE=TR' (+ 'EXTRACT=TXT'), but you will quickly understand that this is not "really usable", ah-ah...! (Do the Test yourself, and you'll understand... :wink: )

Still on 'TYPE=TR', you could use 'EXTRACT=HTM' instead of 'EXTRACT=TXT', but you will need to implement the 'EXTRACT' Mechanism at the 'TR' Level yourself, by removing all HTML Formatting... Can be done, but will probably be a bit cumbersome and not "directly straightforward"... But that might be 'an Option" to consider, if you need to extract several Rows, and the HTML Formatting for all Cells is "more or less" the same...
But hum, for "several Rows" (with a changing Nb of Rows), I think I would rather start from the 'TYPE=TABLE' then... (+ 'EXTRACT='HTM').

4- And the last but probably the "best"/easiest Option, is to simply extract all Cells in one Row one by one, => Cell by Cell with 'TYPE=TD' + 'EXTRACT=TXT', => x8 times as you seem to have 8 Cells per Row on your Screenshot... :idea:
(And you can use some 'POS=R1' Mechanism to simply copy&paste 8 times the same mini-Block of Code...)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Tamilselvan
Posts: 129
Joined: Mon Mar 07, 2016 10:49 am

Re: Extract Table & Append

Post by Tamilselvan » Tue Dec 01, 2020 3:38 am

Code: Select all

TAG POS=3 TYPE=TR ATTR=TXT:* EXTRACT=TXT
I could extract row but i could not remove blanks between data
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract Table & Append

Post by chivracq » Tue Dec 01, 2020 10:13 am

Tamilselvan wrote:
Tue Dec 01, 2020 3:38 am
chivracq wrote:
Sun Nov 29, 2020 7:51 pm
You could try to extract your Table Row by Row, with a 'TYPE=TR' (+ 'EXTRACT=TXT'), but you will quickly understand that this is not "really usable", ah-ah...! (Do the Test yourself, and you'll understand... :wink: )

Code: Select all

TAG POS=3 TYPE=TR ATTR=TXT:* EXTRACT=TXT
I could extract row but i could not remove blanks between data

Alright, good, now you've done your Test, and you understand what I meant with "not really usable", ah-ah...! :D

Even if, hum..., you are "nearly lucky" to get some Blanks between the Cols in your Case, each time I tried that Approach again on different Sites, the Data from all Cols would be actually directly concatenated with no Space/Blank at all between the Cols, which would make re-separating the Data corresponding to each Col nearly impossible... :(

=> If those Blanks are a fixed Number/Width of Spaces or 'Tab' Chars, then it might still be possible in your Case to use "that" as a Separator to re-split the Data... You can try..., but you'll be very lucky if that works... :twisted:
My "Advice", nice if you "quickly" find a Solution, but don't spend too much time or you'll end up pulling all your Hair, ah-ah...!! :(

=> ... Like I told you, the Approach where you extract all 8 Cols/Cells one by one is probably the most straightforward one... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Tamilselvan
Posts: 129
Joined: Mon Mar 07, 2016 10:49 am

Re: Extract Table & Append

Post by Tamilselvan » Wed Dec 02, 2020 2:48 pm

Code: Select all

TAG POS=3 TYPE=TR ATTR=TXT:* EXTRACT=TXT
SET TABLE EVAL("'{{!EXTRACT}}'.trim()")
'Remove quotes from extracted text
SET TAB EVAL("var extr2=\"{{!EXTRACT}}\"; extr2.replace(/'/g,'');")
'Save extracted data
SAVEAS TYPE=EXTRACT FOLDER=* FILE=VrTable.csv
It works but little bit more spaces between extracted data .

As you told extract all 8 Cols/Cells one by one i have tried but some data has single or multiple rows like below. i skipped it.
Vr_No.jpg
In another method I can download file but it stored as single/separate file in table format.
What I need extract table by loop that table should be stored as single file.

Generally I request you, technical / admin to provide code extract table as it format it will helpful to users. :D
or is there any new version available for extract table as it format :?:
Post Reply