PaleMoon extract scrape get SLOWER after it loops

Support for iMacros. The iMacros software is the unique solution for automating every activity inside a web browser, for data extraction and web testing.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

PaleMoon extract scrape get SLOWER after it loops

by a2515125 on Mon Aug 21, 2017 7:49 pm

hey

I am satisfied with PaleMoon browser which is more stable than FF, definitely not crashes that often
BUT it seems can not solve the fact that iMacro will get slower after loops hundred or thousand times

I have this code to extract 35000 data in the same page (POS=1....2....35000)
every 20 loops need to press "see more" button

it was very fast at the beginning like extracting 20 data in 3 seconds
but after I woke up, it was very slower, extracting 1 data /5-10 seconds

I already unchecked the JavaScript on setup / clean history & cookie/ restart the code but make no difference
the CPU only consumes 8% / RAM 20%

How can I improve or solve this getting slower problem

Windows 10/PM 26.3.3/iMacros 8871 / Double CPU Xeon E5 2670/RAM 32G/SSD 240G


Code: Select all
SET !REPLAYSPEED FAST
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 0
SET !LOOP 15578

URL GOTO=javascript:window.scrollBy(0,50000)

'set loop to scrape'

TAG POS={{!LOOP}} TYPE=DIV ATTR=CLASS:media-left
TAG POS=R-1 TYPE=A ATTR=HREF:*   EXTRACT=HREF

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\admin\Desktop\iMacroscript FILE=123456.csv

'move next page'
SET next EVAL("var x; var y=\"{{!LOOP}}\";if(y%19==0){x=1}else{x=0}; x;")


TAG POS={{next}} TYPE=BUTTON ATTR=CLASS:btn<SP>btn-link<SP>*






a2515125
 
Posts: 84
Joined: Mon Dec 05, 2016 8:37 pm

Re: PaleMoon extract scrape get SLOWER after it loops

by chivracq on Tue Aug 22, 2017 10:10 am

a2515125 wrote:Windows 10 / PM 26.3.3 / iMacros 8871 / Double CPU Xeon E5 2670/RAM 32G/SSD 240G

hey

I am satisfied with PaleMoon browser which is more stable than FF, definitely not crashes that often
BUT it seems can not solve the fact that iMacro will get slower after loops hundred or thousand times

I have this code to extract 35000 data in the same page (POS=1....2....35000)
every 20 loops need to press "see more" button

it was very fast at the beginning like extracting 20 data in 3 seconds
but after I woke up, it was very slower, extracting 1 data /5-10 seconds

I already unchecked the JavaScript on setup / clean history & cookie/ restart the code but make no difference
the CPU only consumes 8% / RAM 20%

How can I improve or solve this getting slower problem

Code: Select all
SET !REPLAYSPEED FAST
SET !ERRORIGNORE YES
SET !EXTRACT_TEST_POPUP NO
SET !TIMEOUT_STEP 0
SET !LOOP 15578

URL GOTO=javascript:window.scrollBy(0,50000)

'set loop to scrape'

TAG POS={{!LOOP}} TYPE=DIV ATTR=CLASS:media-left
TAG POS=R-1 TYPE=A ATTR=HREF:*   EXTRACT=HREF

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\admin\Desktop\iMacroscript FILE=123456.csv

'move next page'
SET next EVAL("var x; var y=\"{{!LOOP}}\";if(y%19==0){x=1}else{x=0}; x;")

TAG POS={{next}} TYPE=BUTTON ATTR=CLASS:btn<SP>btn-link<SP>*

Hum..., I use Pale Moon as well (same v26.3.3 Version, + iMacros for FF v8.8.2, Win10-x64) and I don't do 35,000 Extracts on the same Page, the Max I sometimes do in a row is 20,000 with a "simple" new 'URL GOTO' on a different URL each time (incremented by 1), runs in about 10 hours, and I didn't notice any slow down at the end, while I have other Scripts running 24/7 in other Profiles at the same time that I maybe only restart once every 3 weeks sometimes...

But if you are extracting your Data from some Site like FB/LinkedIn/Twitter/Instagram/etc which are stupidly overloaded with several Mb's of JavaScript Scripts running on them for Notification/Chat/See more..., then, tja..., I'm not surprised...!

Try using their Mobile Site if they have any, as it is often much lighter and without all those JS Scripts that finally crash any Browser (and I think FF would even do worse than PM)...

Or you'll have to think about a "more robust" Strategy, where from a '.BAT' File or from the Win10 Task Scheduler, you kill and restart the Browser and your Script every 2h for example, stopping it for example 2min before the "Switch" and saving the current Loop_Nb to some Temp File that will be picked up at the first Run after the Switch, but hum..., that's a bit cumbersome, and you'll need to be a bit "Creative", ah-ah...! You cannot "really" loop an '.iim' Script from the Scheduler for example, you will probably have to use a '.js' Script to handle the Looping yourself...

Mini-thing you can try as well, I notice you are on v8.8.7 for FF while I use v8.8.2 myself (on PM v26.3.3 as well, + Win10-Home_x64), I don't really expect any Improvement as I think all Changes between v8.8.2 and v8.8.7 were related to 'BOM', but you never know...!
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6489
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: PaleMoon extract scrape get SLOWER after it loops

by a2515125 on Tue Aug 22, 2017 11:30 am

got it ....thanks

the website I extract is similar to FB....haha

Now I understand why after you explained which remind me of FB has same pb when extract the thumb-up members

I wonder if using python to extract the same data 35000 in the same page would get the same result of terrible speed :?: :?:
a2515125
 
Posts: 84
Joined: Mon Dec 05, 2016 8:37 pm

Re: PaleMoon extract scrape get SLOWER after it loops

by chivracq on Tue Aug 22, 2017 12:03 pm

a2515125 wrote:got it ....thanks

the website I extract is similar to FB....haha

Now I understand why after you explained which remind me of FB has same pb when extract the thumb-up members

I wonder if using python to extract the same data 35000 in the same page would get the same result of terrible speed :?: :?:

I sometimes see a PM or FF Profile/Instance crash by itself or increase CPU Use to 40% and become completely unresponsive with any Page taking 10 Sec to react to a simple Click while no iMacros Script ran/runs in it, only from some FB or similar Page(s) loaded in it, and the only Solution at that moment is to restart the Instance...

=> I'm not sure Python will help then, because you will still need 1750 times to call the JavaScript that triggers the 'See more...'...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6489
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: PaleMoon extract scrape get SLOWER after it loops

by a2515125 on Tue Aug 22, 2017 7:10 pm

maybe we need one script for press all "see more" button, pressing 1750 times at the first

and restart to run second script for extraction would be faster.........haha
a2515125
 
Posts: 84
Joined: Mon Dec 05, 2016 8:37 pm


Return to General Support & Discussions

Who is online

Users browsing this forum: Google [Bot] and 7 guests

-->