Macro perfomance slow when extracting table from javascript

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
sdpat
Posts: 2
Joined: Tue Apr 03, 2018 8:34 pm

Macro perfomance slow when extracting table from javascript

Post by sdpat » Mon Apr 16, 2018 2:08 pm

VERSION BUILD=11.5.498.2403, Windows 7 , IE11

The following macro extracts a HTML table to csv file. There are approximately 14,000 rows in the table. When I run the macro from the IMacros Browser in IE the macro extracts the file in less than one minute. When I attempt to execute the macro from javascript using iimPlay the macro runs for more than 30 minutes and never returns any results. If I execute the same macro using iimPlay on a smaller table(12 rows) the macro returns the csv file.

Is there a setting or something I can check while executing the macro from javascript to get the same performance I get when executing the macro in IMacros browser?

Macro

Code: Select all

VERSION BUILD=11.5.498.2403
TAB T=1
TAB CLOSEALLOTHERS
SET !PLAYBACKDELAY 0.2
SET !EXTRACT_TEST_POPUP NO
URL GOTO=https://www.exchange.com/Start/
TAG POS=1 TYPE=A ATTR=TXT:Pending<SP>Inspections
TAG POS=1 TYPE=SELECT ATTR=NAME:vendor_id CONTENT=%863885910
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:action
TAG POS=7 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT 
SAVEAS TYPE=EXTRACT FOLDER=\xx\nas\Downloads FILE=cis_{{!NOW:yymmdd_hhnnss}}.csv
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Macro perfomance slow when extracting table from javascr

Post by chivracq » Mon Apr 16, 2018 3:49 pm

sdpat wrote:

Code: Select all

VERSION BUILD=11.5.498.2403,  Windows 7 , IE11
The following macro extracts a HTML table to csv file. There are approximately 14,000 rows in the table. When I run the macro from the IMacros Browser in IE the macro extracts the file in less than one minute. When I attempt to execute the macro from javascript using iimPlay the macro runs for more than 30 minutes and never returns any results. If I execute the same macro using iimPlay on a smaller table(12 rows) the macro returns the csv file.

Is there a setting or something I can check while executing the macro from javascript to get the same performance I get when executing the macro in IMacros browser?

Macro

Code: Select all

VERSION BUILD=11.5.498.2403
TAB T=1
TAB CLOSEALLOTHERS
SET !PLAYBACKDELAY 0.2
SET !EXTRACT_TEST_POPUP NO
URL GOTO=https://www.exchange.com/Start/
TAG POS=1 TYPE=A ATTR=TXT:Pending<SP>Inspections
TAG POS=1 TYPE=SELECT ATTR=NAME:vendor_id CONTENT=%863885910
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:action
TAG POS=7 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT 
SAVEAS TYPE=EXTRACT FOLDER=\xx\nas\Downloads FILE=cis_{{!NOW:yymmdd_hhnnss}}.csv
If running that Macro "from javascript", then you must be using FF and iMacros for FF but you don't mention which Versions you use for/on FF...

'!PLAYBACKDELAY' is not supported on FF btw..., but hum, you wouldn't be able to extract the smaller Table with only 12 Rows either... :?

But hum, you could be right, I had indeed noticed (maybe 2 years ago), that when doing an 'EXTRACT=HTM' on a large/heavy Page at the 'TYPE=BODY' Level, (was on v8.8.2 for FF I would think, or maybe v8.9.7 for FF), that the Extract was taking a very long time indeed on FF compared to IE (I guess I was using v11.0 or v11.5 for IE) if '!EXTRACT_TEST_POPUP' was not disabled.
But you've disabled '!EXTRACT_TEST_POPUP', then it should be OK...

You could add a "PROMPT {{!EXTRACT}}" between the 'EXTRACT' and the 'SAVEAS' to find out if the 'EXTRACT' already works or if the Pb comes from the 'SAVEAS'... (On FF, all Cell Entries get enclosed between Double Quotes, which can take a while I would think, for 14,000 Rows x n Cols...)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
sdpat
Posts: 2
Joined: Tue Apr 03, 2018 8:34 pm

Re: Macro perfomance slow when extracting table from javascr

Post by sdpat » Fri Apr 20, 2018 12:00 am

Thanks for the response. I am definitely using an Internet Explorer Browser and I am definitely launching the macro using javascript within that IE Browser. I decided to write a .NET C# application that launches the macro and I do not experience the performance hit when launching the same macro from C#.
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Macro perfomance slow when extracting table from javascr

Post by chivracq » Fri Apr 20, 2018 3:47 pm

sdpat wrote:Thanks for the response. I am definitely using an Internet Explorer Browser and I am definitely launching the macro using javascript within that IE Browser. I decided to write a .NET C# application that launches the macro and I do not experience the performance hit when launching the same macro from C#.
Hum, OK..., running a '.js' Script from IE..., I thought that was not possible..., Users in this (kind of) Thread(s) for example might be interested to know how you do that... :?

But hum, if you do everything in IE, then I don't know what could be causing the Pb... :oops:
But hum again..., wondering why you need a Script to save 14,000 Rows, I don't think you get 14,000 new Rows everyday, and wondering what kind of "further Processing" you do on 14,000 Rows of Data...!? Well, never mind, it's a "rhetorical" Qt, ah-ah...! :shock:

I had suggested to add a 'PROMPT' in the middle of your Script to try to identify if the Pb came from the 'EXTRACT' or 'SAVEAS' Mechanism, and I would have had a few possible Workarounds in mind for each, but I guess you are now a bit "settled" if your '.NET C#' Solution works fine already... 8)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply