macro slows down

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
shaves
Posts: 77
Joined: Tue Apr 29, 2008 6:50 pm

macro slows down

Post by shaves » Thu Oct 26, 2017 6:12 pm

I'm using iMacros to loop through a list of purchase orders on a company owned web site. It seems like the macro slows down the longer it goes. I'm not sure if it is the web site or something with my code. Below is some of the code to show how the macro works. The rest of the code is very similar to the examples below. It's similar to code I've used in the past. Is there any options or settings that I could change to speed up the macro or is the result of the underlying application?

We are using Office 2010 / Windows 7 Enterprise / iMacros 10.3

Code samples:

' EXTRACTING THE PO DATE
IM = ""
IM = "CODE:"
IM = IM + "TAB T=1" + vbNewLine
IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
IM = IM + "SET !TIMEOUT 60" + vbNewLine
IM = IM + "TAG POS=" & VarA & " TYPE=TD ATTR=CLASS:required<SP>essential EXTRACT=TXT" + vbNewLine
iret = iim1.iimPlay(IM)

MyMsg = ""
MyMsg = iim1.iimGetLastExtract()
MyMsg = Replace(MyMsg, "[EXTRACT]", "")
MyMsg = Trim(MyMsg)

Cells(ResultsRow, vCol).NumberFormat = "mm/dd/yy;@"
Cells(ResultsRow, vCol) = MyMsg
vCol = vCol + 1
VarA = VarA + 4

' EXTRACTING THE PO NUMBER
IM = ""
IM = "CODE:"
IM = IM + "TAB T=1" + vbNewLine
IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
IM = IM + "SET !TIMEOUT 60" + vbNewLine
IM = IM + "TAG POS=" & VarB & " TYPE=TD ATTR=CLASS:required<SP>essential EXTRACT=TXT" + vbNewLine
iret = iim1.iimPlay(IM)

MyMsg = ""
MyMsg = iim1.iimGetLastExtract()
MyMsg = Replace(MyMsg, "[EXTRACT]", "")
MyMsg = Trim(MyMsg)

Cells(ResultsRow, vCol).HorizontalAlignment = xlCenter
Cells(ResultsRow, vCol).NumberFormat = "@"
Cells(ResultsRow, vCol) = MyMsg
vCol = vCol + 1
VarB = VarB + 4

' EXTRACTING THE BRANCH
IM = ""
IM = "CODE:"
IM = IM + "TAB T=1" + vbNewLine
IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
IM = IM + "SET !TIMEOUT 60" + vbNewLine
IM = IM + "TAG POS=1 TYPE=SELECT ATTR=NAME:poDisplay.po.rentalBranch EXTRACT=TXT" + vbNewLine
iret = iim1.iimPlay(IM)

MyBranch = ""
MyBranch = iim1.iimGetLastExtract()
MyBranch = Replace(MyBranch, "[EXTRACT]", "")
MyBranch = Trim(MyBranch)


Thanks for the help.........
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: macro slows down

Post by chivracq » Thu Oct 26, 2017 7:37 pm

shaves wrote:I'm using iMacros to loop through a list of purchase orders on a company owned web site. It seems like the macro slows down the longer it goes. I'm not sure if it is the web site or something with my code. Below is some of the code to show how the macro works. The rest of the code is very similar to the examples below. It's similar to code I've used in the past. Is there any options or settings that I could change to speed up the macro or is the result of the underlying application?

We are using

Code: Select all

Office 2010 / Windows 7 Enterprise / iMacros 10.3
Code samples:

' EXTRACTING THE PO DATE

Code: Select all

            IM = ""
            IM = "CODE:"
            IM = IM + "TAB T=1" + vbNewLine
            IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
            IM = IM + "SET !TIMEOUT 60" + vbNewLine
            IM = IM + "TAG POS=" & VarA & " TYPE=TD ATTR=CLASS:required<SP>essential EXTRACT=TXT" + vbNewLine
            iret = iim1.iimPlay(IM)
    
            MyMsg = ""
            MyMsg = iim1.iimGetLastExtract()
            MyMsg = Replace(MyMsg, "[EXTRACT]", "")
            MyMsg = Trim(MyMsg)
            
            Cells(ResultsRow, vCol).NumberFormat = "mm/dd/yy;@"
            Cells(ResultsRow, vCol) = MyMsg
            vCol = vCol + 1
            VarA = VarA + 4

' EXTRACTING THE PO NUMBER

Code: Select all

            IM = ""
            IM = "CODE:"
            IM = IM + "TAB T=1" + vbNewLine
            IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
            IM = IM + "SET !TIMEOUT 60" + vbNewLine
            IM = IM + "TAG POS=" & VarB & " TYPE=TD ATTR=CLASS:required<SP>essential EXTRACT=TXT" + vbNewLine
            iret = iim1.iimPlay(IM)

            MyMsg = ""
            MyMsg = iim1.iimGetLastExtract()
            MyMsg = Replace(MyMsg, "[EXTRACT]", "")
            MyMsg = Trim(MyMsg)
            
            Cells(ResultsRow, vCol).HorizontalAlignment = xlCenter
            Cells(ResultsRow, vCol).NumberFormat = "@"
            Cells(ResultsRow, vCol) = MyMsg
            vCol = vCol + 1
            VarB = VarB + 4
' EXTRACTING THE BRANCH

Code: Select all

            IM = ""
            IM = "CODE:"
            IM = IM + "TAB T=1" + vbNewLine
            IM = IM + "TAB CLOSEALLOTHERS" + vbNewLine
            IM = IM + "SET !TIMEOUT 60" + vbNewLine
            IM = IM + "TAG POS=1 TYPE=SELECT ATTR=NAME:poDisplay.po.rentalBranch EXTRACT=TXT" + vbNewLine
            iret = iim1.iimPlay(IM)
            
            MyBranch = ""
            MyBranch = iim1.iimGetLastExtract()
            MyBranch = Replace(MyBranch, "[EXTRACT]", "")
            MyBranch = Trim(MyBranch)
Thanks for the help.........
OK, hum-hum...!, several "things" could play a Role...
1- Memory Leak:
All Browsers and Applications can have some Memory Leaks, especially after running for a long time... Firefox is a bit "famous" for Memory Leaks with iMacros, especially with '.js' Scripts, but I reckon all Browsers supported by iMacros have Memory Leaks and FF is the "notorious" one because it is the only one which can run '.js' Scripts natively from the Free FF Add-on.
You seem to be running your Script in iMB v10.3, you could try running it in FF or CR to see if that changes stg, as you don't seem to be using any iMB/IE specific Commands.

You didn't post your whole Script and I don't know how many PO-Records you are extracting and for how long your Script will have to run, but if you suspect Memory Leak, you could always include in your Script a Stop/Exit/Kill of the Browser every 1000 Loops or every 2 hours for example, then restart the Browser Clean and keep looping from where it stopped.

2- If some PO-Records don't exist, they will return "#EANF#" for the Extracts. You don't seem to be using '!TIMEOUT_STEP' for your Script, which means that each failed (= not found) 'EXTRACT' will take 6 Sec, as your '!TIMEOUT_PAGE' Setting ('!TIMEOUT' is deprecated btw...) is 60 Sec, which is the Default Value btw, so that Statement is a bit useless.

3- Your main '.vbs' Script does a lot of back of forth between the Scripting Interface/'.vbs' Compiler and the Browser/iMacros Engine.
For each single mini-Extraction, you each time recreate a mini-'.iim' Macro on the fly with the same '!TIMEOUT_xxx' Settings and where you each time tell the Browser to close all other Tabs, ... which don't exist..., or maybe only the first time. Not needed...!
All 'Replace('[EXTRACT]')' Statements are not needed either, you only get '[EXTRACT]' in the Full '!EXTRACT' (with 'iimGetExtract()' or 'iimGetExtract(0)') as a Separator if you do several Extractions.

I think it would be more efficient if you created only one '.iim' Macro for all Extracts in a row, and populate your Excel Sheet from their corresponding 'iimGetExtract([1,n])' 'EXTRACT' ('iimGetLastExtract()' is deprecated as well btw...).
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
shaves
Posts: 77
Joined: Tue Apr 29, 2008 6:50 pm

Re: macro slows down

Post by shaves » Thu Oct 26, 2017 8:49 pm

Thanks for the feedback. I have (1) taken out all of the CLOSEALLOTHERS command (2) changed TIMEOUT to TIMEOUT_PAGE, and (3) changed all of the GETLASTEXTRACT to GETEXTRACT. I also removed all of the TIMEOUT_PAGE commands from any section that was just extracting data. Since the page was already loaded, this didn't seem to be of much benefit. I'm only using the TIMEOUT_PAGE any time a screen changes.

I also noticed that some of the TIMEOUT_PAGE had 600 seconds. I reduced them as well to 60.

While researching this issue I came across a suggestion to uncheck (1) show javascript during replay, (2) scroll to object when found, and (3) highlight object when found as a way to speed up iMacros. If I remember correctly, this suggestion was from a few years ago and an earlier version of iMacros. I couldn't find these options anywhere. Are they still available?

I noticed that iMacros shifts the page left and right and up and down as it is selecting / extracting different fields. Does this have any impact? Is there a way to "turn this off"?

Thanks for your help and suggestions...........
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: macro slows down

Post by chivracq » Fri Oct 27, 2017 4:44 pm

shaves wrote:Thanks for the feedback. I have (1) taken out all of the CLOSEALLOTHERS command (2) changed TIMEOUT to TIMEOUT_PAGE, and (3) changed all of the GETLASTEXTRACT to GETEXTRACT. I also removed all of the TIMEOUT_PAGE commands from any section that was just extracting data. Since the page was already loaded, this didn't seem to be of much benefit. I'm only using the TIMEOUT_PAGE any time a screen changes.

I also noticed that some of the TIMEOUT_PAGE had 600 seconds. I reduced them as well to 60.

While researching this issue I came across a suggestion to uncheck (1) show javascript during replay, (2) scroll to object when found, and (3) highlight object when found as a way to speed up iMacros. If I remember correctly, this suggestion was from a few years ago and an earlier version of iMacros. I couldn't find these options anywhere. Are they still available?

I noticed that iMacros shifts the page left and right and up and down as it is selecting / extracting different fields. Does this have any impact? Is there a way to "turn this off"?

Thanks for your help and suggestions...........
Yep, the deprecated Commands are still backward compatible, so that won't really make a Difference, especially if you are still at iMB v10.3..., I think only from v9.0.3 for FF that a few Commands started to get removed from the "Code", it's possible that iMB v12.0 is following that "Trend" as well...

The 'show javascript' Option is only for iMacros for FF I would think, but the 2 other Options about 'Scroll' and 'Highlight' are for all 4 Browsers I would think... They can play a Role if you are doing more than 20 or maybe 50 Extractions on the same Page and that Page requires "a lot" of (Vertical) Scrolling, but that will hardly play a Role in your Case... And yep, the Up/down/Left/Right Shifting comes from the 'Scroll to Object' Option...

More efficient I would think will be if you "rethink" the Workflow of your Script to avoid the constant "Switch" between Scripting Interface/'.vbs' Compiler => iMacros Compiler to generate and parse '.iim' Macro on the fly => Browser for one Extraction => back to '.vbs' => go to Excel for a few Actions => back to '.vbs' => generate and parse another mini-Macro => Browser => etc...

You could try activating the iMacros Profiler to get some "Insight" in the Response Times for all Actions in your Script if they indeed increase steadily after some time your Macro has been running to find out if the "lazy" Actor is the Scripting interface or the '.vbs' Compiler or the iMB Browser or Excel or even the OS for File-IO Functionality, or maybe a Combination of several (F)actors and try to find a Solution/Workaround with that Info..., like killing and restarting iMB that I already mentioned...
But hum, the Profiler slows the overall iMacros Performance by about 30% in my "Experience", as it involves a lot of File-IO from the OS... It's more for "Troubleshooting" Purpose in my Opinion... A somewhat lighter Alternative if to build your own Logging (like I do), based on Timestamps and a few '!STOPWATCH' Flags place at a few "strategic" places in your Script that you then save yourself in your own 'SAVEAS' File (or include in the same File where you already save the Extractions, but you currently don't use the 'SAVEAS TYPE=EXTRACT' Mechanism...), but hum..., that will still be quite "heavy" in your Case with 1 'EXTRACT' = 1 Macro... => = 1 Logging...

And it's completely possible that iMB experiences some Memory Leak after a while, you are still at iMB v10.3, I guess the Performance for iMB probably got improved in iMB v11.0 then v11.5 and now v12.0 that were released after "your" v10.3 Version which is already 5 or 6 years old I think... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply