Will this software scrap this site in the way that I want ?

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
zog
Posts: 2
Joined: Tue Apr 30, 2019 5:29 am

Will this software scrap this site in the way that I want ?

Post by zog » Tue Apr 30, 2019 5:42 am

Hey all,
Will this software scrape this site ?
https://westernpower.com.au/faults-outa ... tages-map/
I would like to scrap it automaticly every 5 mins to compile a database of just how long certain areas are without power.
It needs a zoom to see all the data so that is why I am asking before buying anything.
zog
Posts: 2
Joined: Tue Apr 30, 2019 5:29 am

Re: Will this software scrap this site in the way that I want ?

Post by zog » Wed May 01, 2019 3:17 am

I guess not.
No replies.
Goldclownfish
Posts: 18
Joined: Mon Apr 29, 2019 3:50 am

Re: Will this software scrap this site in the way that I want ?

Post by Goldclownfish » Wed May 01, 2019 4:39 am

Well for one thing if your took the time to help yourself. You would see in fact they offer a FREE 30 trial of the software on the website. Next time spend a little time researching before u expect someone to hand it to you. It in fact took you longer to post then it would have to look for yourself. :roll:
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Will this software scrap this site in the way that I want ?

Post by chivracq » Wed May 01, 2019 3:29 pm

zog wrote:
Tue Apr 30, 2019 5:42 am
Hey all,
Will this software scrape this site ?
https://westernpower.com.au/faults-outa ... tages-map/
I would like to scrap it automaticly every 5 mins to compile a database of just how long certain areas are without power.
It needs a zoom to see all the data so that is why I am asking before buying anything.
zog wrote:
Wed May 01, 2019 3:17 am
I guess not.
No replies.
Goldclownfish wrote:
Wed May 01, 2019 4:39 am
Well for one thing if your took the time to help yourself. You would see in fact they offer a FREE 30 trial of the software on the website. Next time spend a little time researching before u expect someone to hand it to you. It in fact took you longer to post then it would have to look for yourself. :roll:
Yeah...!, your "I guess not. No replies." is a bit of an "impatient" Conclusion, ah-ah...! The Forum is "Users helping Users", and "sbd" needs to go and do some Testing on your Site before "we" can give you a YES or NO Answer, and that sbd can better be yourself as the Site seems to be quite dynamic with Content and "we" don't know what to "expect"...
=> Like @Gcf mentioned, you can download the Full iMB v12.5 Browser (Trial for 30 days), and you have 'Free' Add-ons for IE/FF/CR. 8)
The 'Free' Add-ons for FF and CR are a bit limited in Functionality (saving the Data to a local '.CSV' for example is not included), but you can still test if locating and extracting the Data that you want already works... :idea:
(You can then use the 'PROMPT' and/or '!CLIPBOARD' Commands to check that Content as the 'SAVEAS' Command won't work in the 'Free' Versions...)

But OK, I was already a "nice sbd" yesterday to already have a look at your Site, and today again, and I "now" understand why you mentioned "Zooming", as yesterday there were "only" 2 Outages in total, 1x Red and 1x Yellow, and quite "far" from each other, while today there are 5 in total, and with 2 quite close to each other (in Perth), that get grouped together and displayed as a "2", to then give a "4" if I zoom out 4x times, and a final "5" if I zoom out 6x times.

The Site reacts well to the 'TAG' and 'EVENT' Modes to tag and extract Data and to interact with the Red/Yellow Flags (to open their corresponding Popup and extract that specific Data), count the total Nb of Outages, zoom in/zoom out, automatically zoom in when clicking on a Group Nb, and also with the 'Center Map' Button if you want to use its Functionality, or that I used myself as 'Anchor' for 'Relative Positioning' to first extract the total Nb of Outages (after first zooming out 6x times) to then further zoom in again to open all Red or Yellow Flags to extract their corresponding Data.

It works, (so the/my Answer is YES to your Qt, ah-ah...! :wink: ), and it's quite an interesting "Scenario" and Site actually, but you'll need to "build" a bit of "Intelligence" in your Script to make it "reliable" and to cover all possible Cases, with 0 - 2 - 4 - 5 - 10, maybe 20 or more Outages(?), with Red and Yellow Flags and maybe some other Colours, I don't know...?, etc... 8)
[I did my Testing using iMacros for FF v8.8.2 + Pale Moon v26.3.3 (=FF47) + Win10_x64.]

Now you need to "dig in the mud" yourself, and "we"'ll be here on the Forum to help you if you get stuck "somewhere", but nbd will be writing your Script for you, ah-ah...! :wink:
Hum, and if you expect "real Support", only the License for iMB gives you access to TechSupport, the 'PE' License for the FF and CR Add-ons still relies on the Forum for Help. :idea:

(And I'm the most active Helper on the Forum, but I only help Users using the Forum "a bit correctly", which means: 'correct Sub-Forum when they open a new Thread' + 'descriptive Thread Title' (and I'm quite "allergic" to "Help needed...!!" or "Urgent...!!" etc for example...) + 'FCI mentioned' + 'no Social Media/Forums with Like/Follow/Comment' + 'no Games/Votes/Financial Gain' + 'neat Follow-up' + 'sharing their Solution/final Script' + 'really getting stuck after trying their best' + 'being "nice" of course'... :wink: )
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
Post Reply