Macro perfomance slow when extracting table from javascript

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.
Post Reply
sdpat
Posts: 2
Joined: Tue Apr 03, 2018 8:34 pm

Macro perfomance slow when extracting table from javascript

Post by sdpat » Mon Apr 16, 2018 2:08 pm

VERSION BUILD=11.5.498.2403, Windows 7 , IE11

The following macro extracts a HTML table to csv file. There are approximately 14,000 rows in the table. When I run the macro from the IMacros Browser in IE the macro extracts the file in less than one minute. When I attempt to execute the macro from javascript using iimPlay the macro runs for more than 30 minutes and never returns any results. If I execute the same macro using iimPlay on a smaller table(12 rows) the macro returns the csv file.

Is there a setting or something I can check while executing the macro from javascript to get the same performance I get when executing the macro in IMacros browser?

Macro

Code: Select all

VERSION BUILD=11.5.498.2403
TAB T=1
TAB CLOSEALLOTHERS
SET !PLAYBACKDELAY 0.2
SET !EXTRACT_TEST_POPUP NO
URL GOTO=https://www.exchange.com/Start/
TAG POS=1 TYPE=A ATTR=TXT:Pending<SP>Inspections
TAG POS=1 TYPE=SELECT ATTR=NAME:vendor_id CONTENT=%863885910
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:action
TAG POS=7 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT 
SAVEAS TYPE=EXTRACT FOLDER=\xx\nas\Downloads FILE=cis_{{!NOW:yymmdd_hhnnss}}.csv
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Macro perfomance slow when extracting table from javascr

Post by chivracq » Mon Apr 16, 2018 3:49 pm

sdpat wrote:

Code: Select all

VERSION BUILD=11.5.498.2403,  Windows 7 , IE11
The following macro extracts a HTML table to csv file. There are approximately 14,000 rows in the table. When I run the macro from the IMacros Browser in IE the macro extracts the file in less than one minute. When I attempt to execute the macro from javascript using iimPlay the macro runs for more than 30 minutes and never returns any results. If I execute the same macro using iimPlay on a smaller table(12 rows) the macro returns the csv file.

Is there a setting or something I can check while executing the macro from javascript to get the same performance I get when executing the macro in IMacros browser?

Macro

Code: Select all

VERSION BUILD=11.5.498.2403
TAB T=1
TAB CLOSEALLOTHERS
SET !PLAYBACKDELAY 0.2
SET !EXTRACT_TEST_POPUP NO
URL GOTO=https://www.exchange.com/Start/
TAG POS=1 TYPE=A ATTR=TXT:Pending<SP>Inspections
TAG POS=1 TYPE=SELECT ATTR=NAME:vendor_id CONTENT=%863885910
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:action
TAG POS=7 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT 
SAVEAS TYPE=EXTRACT FOLDER=\xx\nas\Downloads FILE=cis_{{!NOW:yymmdd_hhnnss}}.csv
If running that Macro "from javascript", then you must be using FF and iMacros for FF but you don't mention which Versions you use for/on FF...

'!PLAYBACKDELAY' is not supported on FF btw..., but hum, you wouldn't be able to extract the smaller Table with only 12 Rows either... :?

But hum, you could be right, I had indeed noticed (maybe 2 years ago), that when doing an 'EXTRACT=HTM' on a large/heavy Page at the 'TYPE=BODY' Level, (was on v8.8.2 for FF I would think, or maybe v8.9.7 for FF), that the Extract was taking a very long time indeed on FF compared to IE (I guess I was using v11.0 or v11.5 for IE) if '!EXTRACT_TEST_POPUP' was not disabled.
But you've disabled '!EXTRACT_TEST_POPUP', then it should be OK...

You could add a "PROMPT {{!EXTRACT}}" between the 'EXTRACT' and the 'SAVEAS' to find out if the 'EXTRACT' already works or if the Pb comes from the 'SAVEAS'... (On FF, all Cell Entries get enclosed between Double Quotes, which can take a while I would think, for 14,000 Rows x n Cols...)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
sdpat
Posts: 2
Joined: Tue Apr 03, 2018 8:34 pm

Re: Macro perfomance slow when extracting table from javascr

Post by sdpat » Fri Apr 20, 2018 12:00 am

Thanks for the response. I am definitely using an Internet Explorer Browser and I am definitely launching the macro using javascript within that IE Browser. I decided to write a .NET C# application that launches the macro and I do not experience the performance hit when launching the same macro from C#.
chivracq
Posts: 7722
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Macro perfomance slow when extracting table from javascr

Post by chivracq » Fri Apr 20, 2018 3:47 pm

sdpat wrote:Thanks for the response. I am definitely using an Internet Explorer Browser and I am definitely launching the macro using javascript within that IE Browser. I decided to write a .NET C# application that launches the macro and I do not experience the performance hit when launching the same macro from C#.
Hum, OK..., running a '.js' Script from IE..., I thought that was not possible..., Users in this (kind of) Thread(s) for example might be interested to know how you do that... :?

But hum, if you do everything in IE, then I don't know what could be causing the Pb... :oops:
But hum again..., wondering why you need a Script to save 14,000 Rows, I don't think you get 14,000 new Rows everyday, and wondering what kind of "further Processing" you do on 14,000 Rows of Data...!? Well, never mind, it's a "rhetorical" Qt, ah-ah...! :shock:

I had suggested to add a 'PROMPT' in the middle of your Script to try to identify if the Pb came from the 'EXTRACT' or 'SAVEAS' Mechanism, and I would have had a few possible Workarounds in mind for each, but I guess you are now a bit "settled" if your '.NET C#' Solution works fine already... 8)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
Post Reply