Data extraction problem in tabular format imacros

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
Lekha Mohanty
Posts: 3
Joined: Mon Oct 25, 2021 7:46 am

Data extraction problem in tabular format imacros

Post by Lekha Mohanty » Mon Oct 25, 2021 7:54 am

Hi.
I am using firefox 48
imacros 8.9.7
Windows 10 64 bit OS

I am using the below code to extract data

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
URL GOTO=https://www.nseindia.com/option-chain
TAG POS=1 TYPE=TH ATTR=TXT:CALLS
TAG POS=2 TYPE=DIV ATTR=TXT:CALLSPUTS<SP>OI<SP>Chng<SP>in<SP>OI<SP>Volume<SP>IV<SP>LTP* EXTRACT=TXT
But after extraction I am not getting the data in table format. is there any solution ?
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Data extraction problem in tabular format imacros

Post by chivracq » Mon Oct 25, 2021 1:21 pm

Lekha Mohanty wrote:
Mon Oct 25, 2021 7:54 am
Hi.
I am using firefox 48
imacros 8.9.7
Windows 10 64 bit OS

I am using the below code to extract data

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
URL GOTO=https://www.nseindia.com/option-chain
TAG POS=1 TYPE=TH ATTR=TXT:CALLS
TAG POS=2 TYPE=DIV ATTR=TXT:CALLSPUTS<SP>OI<SP>Chng<SP>in<SP>OI<SP>Volume<SP>IV<SP>LTP* EXTRACT=TXT
But after extraction I am not getting the data in table format. is there any solution ?

Yep, the Result you are getting is "normal", because you are extracting the Data from this Table at some (Containing) 'DIV' Level, but you need/want to extract it at the 'TABLE' Level: :idea:

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=ID:"optionChainTable-indices" EXTRACT=TXT
+ Another Solution if you prefer, which doesn't even require looking at the HTML Source of the Page..., => by simply using the first few 'TXT' Terms from the Table Header...: :P

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=TXT:CALLS*PUTS* EXTRACT=TXT
(Tested in iMacros for FF v8.8.2, PM v26.3.3, Win10_x64.)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Lekha Mohanty
Posts: 3
Joined: Mon Oct 25, 2021 7:46 am

Re: Data extraction problem in tabular format imacros

Post by Lekha Mohanty » Mon Oct 25, 2021 4:43 pm

Hello sir,
Thanks for your response.

I have tried code

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=TABLE ATTR=ID:"optionChainTable-indices" EXTRACT=TXT
 SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=TABLE ATTR=TXT:CALLS*PUTS* EXTRACT=TXT
 SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
With these codes I tried to save the data in CSV format but all the data placed in a single cell, I want it in Excel format or in Google sheet. is there any solution available ?

file extract table format
file extract table format
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Data extraction problem in tabular format imacros

Post by chivracq » Tue Oct 26, 2021 3:33 am

Lekha Mohanty wrote:
Mon Oct 25, 2021 4:43 pm
Hello sir,
Thanks for your response.

I have tried code

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=TABLE ATTR=ID:"optionChainTable-indices" EXTRACT=TXT
 SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv

Code: Select all

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=TABLE ATTR=TXT:CALLS*PUTS* EXTRACT=TXT
 SAVEAS TYPE=EXTRACT FOLDER=* FILE=Extract_{{!NOW:ddmmyy_hhnnss}}.csv
With these codes I tried to save the data in CSV format but all the data placed in a single cell, I want it in Excel format or in Google sheet. is there any solution available ?


nse.png

Hum, you don't need to call me "Sir", we have Ladies working for @TechSup and sometimes answering Threads on the Forum, they won't be "very happy" if you call them "Sir", ah-ah...! Keep it Gender-Neutral... :idea:

>>>

Hum, "Excel format" doesn't mean anything btw, 'CSV' is a "Format", `Excel` is just one Software Program/Application able to open that kind of File "trying" to interpret that Format, because you/most Users have associated the '.csv' File Extension with that Program (at the OS Level), to display like "it" thinks Users want to have the Content of that file displayed, with some limited Control from the "CSV Import Options"... :!: :idea:

>>>

Hum..., whole Data (= whole Table) "saved in a single Cell", => hum-hum, yep indeed, that's FF and iMacros for FF that saves the Data with Double Quotes around each Cell, + also some extra Double Quotes at the complete beginning + complete end of the Table, => the 'SAVEAS=EXTRACT' in your '.csv' File if the '!EXTRACT' only contains your Table...

=> Before opening your '.csv' in `Excel`, you'll need to open that File in `Notepad` (as you are on OS = Win10_x64) (and I reckon/suppose you've associated the '.csv' Windows File Extension to be associated (to be automatically opened) with `Excel` (...?, like most People do...) ... at the OS Level...), => to remove those 2 Double Quotes... (the very first + the very last ones...!). :!: :idea:
(Could be (automatically) removed also from some '.bat' File... And as you are using v8.9.7 for FF, that "Post-Processing" '.bat' File can even be launched from iMacros, at the end of your "normal"/current Script...) :idea:

=> Other Option: => Is from `Excel`, you need to "play" with the "Excel CSV Import Options", especially with the "Delimiter" Setting, => not selecting 'Double Quotes', => 'Single Quotes' for example should do "the Trick", I would think... :idea:

"... I would think..." => I'm not completely sure, you'll need to "experiment" what works (best)..., I haven't used `Excel` in many-many years :P , (8 or 10 maybe, I think...), and didn't need to extract Full Table(s) for myself for "a very long time", ah-ah...! :twisted:

(All my "Knowledge and Expertise" come from "Situations" I have personally "encountered" and had "to solve for myself", sometimes from Cases/Threads on the Forum (very good btw that you posted the URL to your Site, or I wouldn't have been able to test..., ... and you would only have gotten some "Generic" Advice from me then, ah-ah...! :D ), but I don't do any Data Extraction at the 'TABLE' Level for myself, so I only use (and know and test) that Func for the Forum, ah-ah...! :wink: )

>>>

Maybe "useful" to know, if you noticed my "yep indeed, that's FF and iMacros for FF that saves the Data with Double Quotes around each Cell, + also some extra Double Quotes at the complete beginning + complete end of the Table,", => then yep, this is bit "typical" FF + iMacros for FF Behaviour..., other Browsers (and their corresponding iMacros Version(s)) behave(d) differently and don't/didn't save the extra Double Quotes causing the "Table EXTRACT in One Cell" Behaviour... :idea:

But v8.9.7 for FF that you are using is "now" the last "fully functional" Version left which is 'FREE', there is no more any other Version to use with as much Func supported coming "close" to v8.9.7 for FF, ah-ah...! then, tja...!, you'll have to "live" with the "little" Limitations/Bugs from that Version, ah-ah...! :wink:

>>>

=> Not sure what "Bug" means...!?
=> Simply run this mini-Script... (on the Page you posted): :idea:

Code: Select all

TAG POS=2 TYPE=DIV ATTR=TXT:CALLSPUTS<SP>OI<SP>Chng<SP>in<SP>OI<SP>Volume<SP>IV<SP>LTP* EXTRACT=HTM
... Let it run..., your Browser will eventually "come back", ... after 5 minutes, ah-ah...! :twisted:

Workaround:
Use '!EXTRACT_TEST_POPUP' (=> "=NO"...!), + 'PROMPT' to display the '!EXTRACT' Content... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Lekha Mohanty
Posts: 3
Joined: Mon Oct 25, 2021 7:46 am

Re: Data extraction problem in tabular format imacros

Post by Lekha Mohanty » Wed Oct 27, 2021 10:29 am

Thanks for your answer.

Sorry I will not call you sir further....

Yes I found a solution in excel by deleting the quotations and additional comma's within the value. Now I have no issues with that.
Also I don't know how to use the .bat file to delete the comma's and quotations.
But I just want to do one thing that if I have saved a csv file in my system D:\NSE.csv, then I want that whenever I will run the code it will replace whatever existing data in the sheet and will put the new data extracted from the webpage, I have tried that but what I found that if already there is an existing file in that location its not replacing the old data. Please tell me is it possible with iMacros ?

Second thing when I clicked on the table I found the code

Code: Select all

TAG POS=2 TYPE=DIV ATTR=TXT:CALLSPUTS<SP>OI<SP>Chng<SP>in<SP>OI<SP>Volume<SP>IV<SP>LTP*
but from where You have used the code

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=TXT:CALLS*PUTS* EXTRACT=TXT
It means if the Attribute code Calls and puts are there in the table it will work ? but why TAG POS=1 ?

Also from where did you find the code

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=ID:"optionChainTable-indices" EXTRACT=TXT
From where you have found this ATTR ID ? I mean how to find out these things ? By looking source code ?
Tamilselvan
Posts: 129
Joined: Mon Mar 07, 2016 10:49 am

Re: Data extraction problem in tabular format imacros

Post by Tamilselvan » Wed Oct 27, 2021 4:50 pm

Hi,
How do you get in Table Format which is extracted data....?

Code: Select all

TAG POS=1 TYPE=TABLE ATTR=ID:"optionChainTable-indices" EXTRACT=TXT
This is found in Inspect Element....(Right Click on page)
ID:"optionChainTable-indices"
Post Reply