Trying to redo loop if EANF

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
NotguyinVideo
Posts: 5
Joined: Mon Aug 03, 2020 10:36 pm

Trying to redo loop if EANF

Post by NotguyinVideo » Mon Aug 03, 2020 10:45 pm

Hi guys, first post, trying to self teach iim and its going pretty well but i need to redo some of my web scraping and im a bit stumped. Heres the raw code, I use it to enter 2 variables and then press go and i want to scrape the output The issue is My internet is rotating Ip every so often and if it times up just right when it rotates then the output is EANF, for some reason if i get too many of those then iim just locks up and it doesn't work anymore, I like to set this up to run all night and set it to run slow so it never messes up. I think what i need to do though is something along the lines of

If Extract = #EANF# then redo this loop using same variables

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\hirea\Downloads FILE=Output.csv

This is the line that extracts the output and occasionally cant find the expected Anchor, I want some sort of test at this point in the loop that says if #EANF# then go back up to "TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"first Variable" CONTENT={{!COL1}}"

Heres my config
Windows 10 Pro Version 1909 English
imacros Browser V12.6.505.4525 have license but was using Internet explorer plugin and not sure how to migrate liscense
Demos seem to work
Not sure how to test VBS scripting interface
get the same issue in IE explorer and imacros browser The issue is my cycline IP kills the internet for a few seconds and inevitably my iim is trying to scrap data at that exact moment

VERSION BUILD=12.5.2018.1105
'
URL GOTO="the website im scraping"
SET !DATASOURCE C:\Users\hirea\Desktop\"the file with the 2 inputs"
SET !LOOP 1
'Increase the current position in the file with each loop
SET !DATASOURCE_LINE {{!LOOP}}
SET !WAITPAGECOMPLETE no
TAB T=1
SET !ERRORIGNORE YES
TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"first Variable" CONTENT={{!COL1}}
TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"2nd Variable" CONTENT={{!COL2}}
WAIT SECONDS=2
TAG POS=1 TYPE=BUTTON:SUBMIT ATTR=TXT:"Give me the output"
ADD !EXTRACT {{!COL1}}
ADD !EXTRACT {{!COL2}}
WAIT SECONDS=2
TAG POS=1 TYPE=DIV ATTR=CLASS:"output" EXTRACT=TXT
WAIT SECONDS=2
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\hirea\Downloads FILE=Output.csv
URL GOTO="the website im scraping"
WAIT SECONDS=6
Last edited by NotguyinVideo on Tue Aug 04, 2020 12:09 pm, edited 1 time in total.
chivracq
Posts: 9430
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Trying to redo loop if EANF

Post by chivracq » Tue Aug 04, 2020 3:48 am

NotguyinVideo wrote:
Mon Aug 03, 2020 10:45 pm
Hi guys, first post, trying to self teach iim and its going pretty well but i need to redo some of my web scraping and im a bit stumped. Heres the raw code, I use it to enter 2 variables and then press go and i want to scrape the output The issue is My internet is rotating Ip every so often and if it times up just right when it rotates then the output is EANF, for some reason if i get too many of those then iim just locks up and it doesn't work anymore, I like to set this up to run all night and set it to run slow so it never messes up. I think what i need to do though is something along the lines of

Code: Select all

If Extract = #EANF# then redo this loop using same variables

SAVEAS TYPE=EXTRACT FOLDER=C:\Users\hirea\Downloads FILE=Output.csv   
This is the line that extracts the output and occasionally cant find the expected Anchor, I want some sort of test at this point in the loop that says if #EANF# then go back up to "TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"first Variable" CONTENT={{!COL1}}"

Code: Select all

VERSION BUILD=12.5.2018.1105

URL GOTO="the website im scraping"

SET !DATASOURCE C:\Users\hirea\Desktop\"the file with the 2 inputs"
'
SET !LOOP 1
'Increase the current position in the file with each loop 
SET !DATASOURCE_LINE {{!LOOP}}
'
SET !WAITPAGECOMPLETE no
'
TAB T=1
'
SET !ERRORIGNORE YES
'
TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"first Variable" CONTENT={{!COL1}}

TAG POS=1 TYPE=INPUT:TEXT ATTR=NAME:"2nd Variable" CONTENT={{!COL2}}

WAIT SECONDS=2

TAG POS=1 TYPE=BUTTON:SUBMIT ATTR=TXT:"Give me the output"

ADD !EXTRACT {{!COL1}}
ADD !EXTRACT {{!COL2}}

WAIT SECONDS=2

TAG POS=1 TYPE=DIV ATTR=CLASS:"output" EXTRACT=TXT
'
WAIT SECONDS=2
'
SAVEAS TYPE=EXTRACT FOLDER=C:\Users\hirea\Downloads FILE=Output.csv
'
URL GOTO="the website im scraping"

WAIT SECONDS=6

FCIM...! :mrgreen: => for me to read, read my Sig..., apart from checking your Post for Spam before approving it... :wink:

And hum..., most Scripts posted are usually too compact, but yours is on the contrary too "one-Line-by-one-Line" with too many extra=unnecessary (empty) Lines..., => can you compact it a little bit, to make it readable "by Block"...? :?

("Readability" is one of the most important Factors on a Tech Forum, for (Advanced) Users trying to hep you (from your OP), and if the Thread gets a Solution, for other Users searching the Forum with a similar Qt/Pb/Scenario... :idea: )

>>>

And I have very bad Int. Co., not sure when I'll be able to check the Forum (and your Thread) again... :oops:
(Int. Co. now from a Friend tethering for me...)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
NotguyinVideo
Posts: 5
Joined: Mon Aug 03, 2020 10:36 pm

Re: Trying to redo loop if EANF

Post by NotguyinVideo » Tue Aug 04, 2020 12:10 pm

Sorry, went back in and provided more info and tried to clean up my coding
chivracq
Posts: 9430
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Trying to redo loop if EANF

Post by chivracq » Wed Aug 05, 2020 12:45 am

NotguyinVideo wrote:
Tue Aug 04, 2020 12:10 pm
Sorry, went back in and provided more info and tried to clean up my coding

OK, seen that, perfect, and Thanks... :D

And "sorry", but I don't even read en don't try to do any "Thinking" when FCI is not (clearly) mentioned... It's usually a "Waste of Time" for "everybody" otherwise, as I then need to go into several Layers of if-if-if, and you won't understand half of my Post(s), ah-ah...! :twisted:
(... Like hum..., this is I'm "afraid" probably going to happen in this Thread very similar to yours actually, but where the User is not really "able" to mention their FCI...)

>>>
NotguyinVideo wrote:
Mon Aug 03, 2020 10:45 pm
Heres my config

Code: Select all

Windows 10 Pro Version 1909 English
imacros Browser V12.6.505.4525
have license but was using Internet explorer plugin and not sure how to migrate liscense
Demos seem to work
Not sure how to test VBS scripting interface
get the same issue in IE explorer and imacros browser The issue is my cycline IP kills the internet for a few seconds and inevitably my iim is trying to scrap data at that exact moment

Alright good, I don't requote your Script that you've compacted (nearly "too much" now, but OK, never mind...) to keep the Thread "readable"... 8)

So, OK, you are then on iMB v12.6 'Trial' + Win10_Pro_#1909_Eng if I understand correctly...

And hum..., you'll have to "solve" the Licensing Issue, as if you are now on iMB v12.6 'Trial', the Trial Popups will soon start "nagging" you, and they will halt your Script while running "all night long" until you dismiss the Popup(s), defeating that Purpose... :shock:

You have a full Section in the FAQ about that kind of Licensing "Scenarios" and migrating/transferring a License from one PC/Installation to another one..., and if you don't come out, open a separate Thread in the correct Sub-Forum (=> 'Installation & Licensing' Sub-Forum), and @TechSup will be able to help you... :idea:

Your iMB v12.5 "Full" License should also be valid on FF/CR/IE to give you the 'PE' (Full) Version for those 3 Browsers, and even on IE, the v12.6 'Free' Version actually already supports the '!DATASOURCE' Command that you are using, so that could also be an "Option", even if you don't manage to migrate/import your License... Hum..., I'm not completely sure about the 'SAVEAS TYPE=EXTRACT' Command..., I think it is also supported in v12.6 'Free' for IE..., but I'm not "completely sure", you'd need to test/try... But those 2 Commands definitely don't work in v10.0.2 for FF 'Free' + v10.0.5 for CR 'Free', for those 2 Browsers, you then need the 'PE' Version...

>>>

OK, now "we go", ah-ah...!! Hum-hum..., OK, I understand your Scenario, hum..., quite Interesting actually, and I "also" have a very "shitty" Internet Connection, well... for other Reasons, but I'm a bit of a "Specialist" in terms of "Reliability" with Scripts and iMacros, ah-ah...! But you are "in good Hands", I'm "afraid", re-ah-ah...! :P

Alright, you'll need to implement a few Conditional "Mechanisms" to your Script, and to redo a Line/Row in your DataSource, you cannot use the "raw" '!LOOP' Var for '!DATASOURCE_LINE' anymore for '{{!COL1}}' and '{{!COL2}}' but '!DATASOURCE_LINE' will need to pull from some 2nd/secondary mini/Temp-DataSource which Row to use...
For that mini/Temp secondary DataSource, you'll need to use only 1 or 2 Rows in it, and to re-delete and re-create it on every Loop as 'SAVEAS' always creates (a) new Row(s) otherwise (appended) and finding or keeping track of the last Row to use is not very "straightforward"... (It can be done, I've already provided a Solution, but pfff..., will be "easier" for you if you don't go this Way, I would think..., just saying..., ah-ah...!)

For your "main" = current 'SAVEAS', you could "decide" if you want to keep it "like it currently is", => with some "#EANF#" Data in it, or if you maybe want to double that Process, one 'SAVEAS' saving all the Data for every Loop, that could serve as "Logging", and one 'SAVEAS' saving "only" the "clean"/usable Data, search the Forum for "conditional+saveas", I've already provided the Solution for that Func... 8)
=> You basically need to use 'EVAL()' (about to become your "Best Friend", ah-ah...!) to dynamically compute the FileName to use for the 'SAVEAS'. And if you want to double the 'SAVEAS', you'll also need to keep the Content of '!EXTRACT' in some Temp Var as the first 'SAVEAS' will clear (by Design) the Content of '!EXTRACT' and the 2nd 'SAVEAS' would otherwise always save nothing... :!:

>

All your 'WAIT' Statements, meant to slow your Script I understand, are not very "efficient"... I would rather keep only one 'WAIT' Statement, that can also be made Conditional when your IP is changing and the 'EXTRACT' returns "#EANF#", again "conditional+wait" will give you the Solution if you search the Forum on those 2 Terms... :idea:

Some "Interesting" Idea maybe, would be to also use your Temp secondary DataSource to keep track of the current + previous Value for the 'WAIT', to make that 'WAIT' Statement "double" Conditional, the first time you get some "#EANF#" in the 'EXTRACT', make it 5 Sec for example, then 10/15/20 Sec on the next Loops, until a Max of 20 or 30 Sec, until 'EXTRACT' works again and resets the 'WAIT' to 0/1/5, whatever Value you want to use when the Script "works fine" and reliably... :idea: 8) :P :twisted:

>

And I would think you don't "really" need the 2 'URL GOTO' Statements, both at the Beg + End of your Script, the first one at the Beg of your Script is "enough" I would think...

>

OK, "Good Luck" further, and post your "Final" Script, or dare to shout if you didn't understand anything or if you get stuck somewhere... :wink:
Oh...!, and I've lost my Co. of course...! OK, I'll try posting that Reply a bit "later"... :oops:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
NotguyinVideo
Posts: 5
Joined: Mon Aug 03, 2020 10:36 pm

Re: Trying to redo loop if EANF

Post by NotguyinVideo » Wed Aug 05, 2020 2:10 am

Thank you for the post, Im going to go through this and try to muddle my way through some additional lines, the conditional wait sounds like itll be me entire solution actually. Could you elaborate on the ip checking functionality, that likely could solve my issues as well.

I was using IE as the browser and didn't have the free edition popups there but thought possibly it was the browser that was causing a lot of the issues so i am now trying to use Imacros browser but it seems the license doesn't automatically transfer so i need to hunt down the Key somewhere.
chivracq
Posts: 9430
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Trying to redo loop if EANF

Post by chivracq » Wed Aug 05, 2020 5:04 am

NotguyinVideo wrote:
Wed Aug 05, 2020 2:10 am
Thank you for the post, Im going to go through this and try to muddle my way through some additional lines, the conditional wait sounds like itll be me entire solution actually. Could you elaborate on the ip checking functionality, that likely could solve my issues as well.

I was using IE as the browser and didn't have the free edition popups there but thought possibly it was the browser that was causing a lot of the issues so i am now trying to use Imacros browser but it seems the license doesn't automatically transfer so i need to hunt down the Key somewhere.

Hum, "muddle my way" sounds like the "funny" beginning of some "exciting" Adventure, ah-ah...! :P

"the conditional wait sounds like itll be me entire solution actually..."
=> Hum, yep, why not, that would indeed be the "dirty but easy/iest" Solution, with some hard-coded and constant 5-7-10 Sec for the 'WAIT', when 'EXTRACT' returns "#EANF#", if the Data from your DataSource is "reliable" and you know for sure that "#EANF#" from the 'EXTRACT' means that your IP is changing at that time...

The Implementation is fairly easy, => stg like the following for example...:

Code: Select all

SET !EXTRACT NULL
TAG POS=1 TYPE=DIV ATTR=CLASS:"output" EXTRACT=TXT
SET Cond_WAIT EVAL("var s='{{!EXTRACT}}'; var z; if(s=='#EANF#'){z=10;} else{z=1;}; z;")
'PROMPT EXTRACT:<SP>_{{!EXTRACT}}_<BR><BR>Cond_WAIT:<SP>_{{Cond_WAIT}}_
WAIT SECONDS={{Cond_WAIT}}
Activate the 'PROMPT' Line if you want to check/debug the 'EVAL()'...

>>>

About the Popups, only iMB has a 'Trial' Version and triggers those Popups, the 3 other Browsers (=> IE/FF/CR) have a 'PE' (= 'Personal Edition' = "Full") Version, and a 'Free' Version, but they don't trigger any Popup(s)...

User-Defined Vars are for example not supported in the 'Free' Versions, you would then need to change my "Cond_WAIT" Var to "!VAR1", as only '!VAR[1-3]' are supported as Names for Vars in those 'Free' Versions...
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
NotguyinVideo
Posts: 5
Joined: Mon Aug 03, 2020 10:36 pm

Re: Trying to redo loop if EANF

Post by NotguyinVideo » Wed Aug 05, 2020 3:52 pm

So i figured first things first fix the license issue, So i got my chrome IIM working and licensed and it seems to work just fine now with no crashes. Im letting it run now for a few hours and we shall see but my initial basic coding may work just fine in chrome with no crash. WOOHOOOO

Stand by for my post later today where im like nah i was wrong.....
chivracq
Posts: 9430
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Trying to redo loop if EANF

Post by chivracq » Wed Aug 05, 2020 4:09 pm

NotguyinVideo wrote:
Wed Aug 05, 2020 3:52 pm
So i figured first things first fix the license issue, So i got my chrome IIM working and licensed and it seems to work just fine now with no crashes. Im letting it run now for a few hours and we shall see but my initial basic coding may work just fine in chrome with no crash. WOOHOOOO

Stand by for my post later today where im like nah i was wrong.....

OK, Thanks for the Follow-up and the Update... :D
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
NotguyinVideo
Posts: 5
Joined: Mon Aug 03, 2020 10:36 pm

Re: Trying to redo loop if EANF

Post by NotguyinVideo » Wed Aug 05, 2020 5:25 pm

Seems to run pretty good. I think I’ll follow up for some optimizations but I think most of my issue is internet explorer bugginess
chivracq
Posts: 9430
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Trying to redo loop if EANF

Post by chivracq » Wed Aug 05, 2020 9:25 pm

NotguyinVideo wrote:
Wed Aug 05, 2020 5:25 pm
Seems to run pretty good. I think I’ll follow up for some optimizations but I think most of my issue is internet explorer bugginess

OK, good to hear then... :D

But I still gave you some "pretty good" Advice anyway, about Logging and Conditional 'WAIT' to increase the "Reliability" of your Script...
I actually became a bit of a "Specialist" or "Advanced" User with iMacros for a very big part because of "unreliable" Internet Connection, always having to find some "Creative" Solution/Workaround for Failing Connection..., and now I can find some "Solution" for nearly every Scenario some Thread gets opened on the Forum, ah-ah...! :twisted:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Post Reply