[SOLVED] URL Goto appends HTTP:// to URL, can it be stopped?

Discussions and Tech Support specific to the iMacros Firefox add-on.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

[SOLVED] URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Sat Feb 18, 2017 9:12 pm

I have a piece of code that goes directly to the source page of a website in order to avoid excess load on the target servers and my internet (theres a lot of media on the website and I can get all the data I need from the source page without calling on the dozens of videos and images)

The code looks like

Code: Select all

Tag [Position of relevant link] EXTRACT=TXT
SET !VAR1 view-source:
Add !VAR1 {{!Extract}}
URL goto={{!VAR1}}
When I put a prompt in to check var1, it looks correct (view-source:https://addressofsite.com/etc) but when iMacros tries to go there I get an error message that http://view-source:https//etc. isn't a valid link.

Is there any way to get imacros not to prepend http:// to my addresses?

(details: Windows 10, iMacros for Firefox 9.0.3, Firefox 51.0.1 32 bit)

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Solution:
Download Firefox Nightly to run iMacros edited

To Edit iMacros
Run %APPDATA%
go to the URL about:support#extensions-tbody to find the ID for iMacros
Open the XPI file with the iMacros ID for a name with a Zip program like 7-zip
In Folder content there is a file called Chrome_Player.JS
find "MacroPlayer.prototype.ActionTable["url"]" and change url= "http://" + url to url = url
Save the edit and away we go.
Last edited by Housingassocmonkey on Mon Feb 20, 2017 7:59 am, edited 1 time in total.
chivracq
Posts: 9306
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by chivracq » Sun Feb 19, 2017 7:18 am

Housingassocmonkey wrote:I have a piece of code that goes directly to the source page of a website in order to avoid excess load on the target servers and my internet (theres a lot of media on the website and I can get all the data I need from the source page without calling on the dozens of videos and images)

The code looks like

Code: Select all

Tag [Position of relevant link] EXTRACT=TXT
SET !VAR1 view-source:
Add !VAR1 {{!Extract}}
URL goto={{!VAR1}}
When I put a prompt in to check var1, it looks correct (view-source:https://addressofsite.com/etc) but when iMacros tries to go there I get an error message that http://view-source:https//etc. isn't a valid link.

Is there any way to get imacros not to prepend http:// to my addresses?

(details: Windows 10, iMacros for Firefox 9.0.3, Firefox 51.0.1 32 bit)
Ah...!, good, you've added your FCI, but it's always a bit "dangerous" to "silently" edit your Posts as I usually only check them once... But I was anyway going to reply as I find your Thread "interesting", ah-ah...!

So, OK, you got me busy for a few hours, ah-ah...!, and I found a few Solutions, but they strangely don't all work in the 2 FCI's I tested them or they behave differently, and I don't have v9.0.3 for FF installed, so you'll have to test by yourself which one(s) (still) work(s) in your Config as v9.0.3 is dropping Support for "URl GOTO=javascript:...".

But considering the Reason you give for wanting to use the Page Source because of all Images and Videos, you could use an Ad-Blocker on this Page or Site like ABP/ABE/uBlock which allows to specify exactly what you want to block...

A simple Solution to block Images from iMacros, I'm not too completely sure about Videos..., is to use the 'FILTER' Command:

Code: Select all

FILTER TYPE=IMAGES STATUS=ON
'... Do your stuff...
FILTER TYPE=IMAGES STATUS=OFF
The 2 FCI's I used to test:
- iMacros for FF v8.8.2, Pale Moon v26.3.3 (=FF47), Win10-x64.
- iMacros for FF v8.9.7, FF51, Win10-x64.
The PM Config is my "main"/Prod/Default FCI...

First I checked your Pb and I can indeed reproduce the Behaviour you observe... Handy btw when you ask a Qt if you can mention the exact Error Msg which is:
BadParameter: The URL syntax is not correct: 'http://view-source:http://forum.imacros.net/viewtopic.php?f=11&t=27257', line 4 (Error code: -911)
But, hum..., if the main Requirement is not to load the whole Page because of all Media Files, then only on PM with v8.8.2 I find usable Solutions which don't work on FF51 with v8.9.7 and I don't expect them to work with v9.0.3 either, and all other Solutions actually need the Page to be already (fully) loaded...

The Solutions working on PM are:

Code: Select all

URL GOTO=javascript:window.location="view-source:http://forum.imacros.net/viewtopic.php?f=11&t=27257"

Code: Select all

URL GOTO=javascript:void(window.location="view-source:http://forum.imacros.net/viewtopic.php?f=11&t=27257")

Code: Select all

URL GOTO=javascript:window.open("view-source:http://forum.imacros.net/viewtopic.php?f=11&t=27257")
All other Solutions either work in PM or in FF or in both but differently, and they all require the Page to be already loaded...:

Code: Select all

URL GOTO=javascript:void(window.open('view-source:'+location.href))

Code: Select all

URL GOTO=javascript:window.open('view-source:'+location.href)

Code: Select all

URL GOTO=javascript:void(window.location="view-source:"+window.location)

Code: Select all

URL GOTO=javascript:window.location="view-source:"+window.location
This one is interesting...!:

Code: Select all

SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=HTML ATTR=* EXTRACT=HTM
'SAVEAS TYPE=HTM FOLDER=D:\TEMP FILE=View-Source.TXT
SAVEAS TYPE=EXTRACT FOLDER=D:\TEMP FILE=View-Source.TXT

TAB OPEN
TAB T=2
URL GOTO=file:///D:\TEMP\View-Source.TXT
'SAVEAS TYPE=HTM' didn't work because iMacros still appends an '.htm' Extension to the File...

And this one as well, using the 'EVENT' Mode:

Code: Select all

EVENT TYPE=KEYPRESS CHAR="u" MODIFIERS="CTRL"
And otherwise, to answer your exact Qt, i.e. if it is possible to stop iMacros from prepending an URL with 'http://"?, then, pffff...!, yep it is, but it is a bit cumbersome and you will need to "hack" the Add-on and "customize" the 'URL GOTO' Command which is declared in 'MacroPlayer.js':

Code: Select all

MacroPlayer.prototype.RegExpTable["url"]="^goto\\s*=\\s*("+im_strre+")\\s*$";
MacroPlayer.prototype.ActionTable["url"]=function(cmd){var param=imns.unwrap(this.expandVariables(cmd[1])),scheme=null;
if(!/^([a-z]+):.*/i.test(param)){param="http://"+param;}var ios=imns.Cc["@mozilla.org/network/io-service;1"].getService(imns.Ci.nsIIOService);try{ios.newChannel(param,null,null);}catch(e){Components.utils.reportError(e);
throw new BadParameter("The URL syntax is not correct: '"+param+"'");}try{gBrowser.loadURI(param,null,null);}catch(e){var s=e.toString();
if(/NS_ERROR_FILE_NOT_FOUND/.test(s))throw new RuntimeError("File "+param+" not found",930);else throw e;}};
(From v8.8.2 for FF.)

Pb is the Dash in "view-source" which triggers the "if(!/^([a-z]+):.*/i.test(param))" Statement, ah-ah...!
But, good luck if you go this way, ah-ah...!!

But I would think that the 'FILTER' Solution is the easiest and should meet your Needs, I would think..., well, if 'FILTER' is not broken in v9.0.3 which is pretty buggy, ah-ah...!
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Sun Feb 19, 2017 7:46 am

Many thanks for such a detailed answer. Sadly AdBlock is not an option as the website blocks access if it detects an adblocker. And filter images doesn't cut enough of the bandwidth. Apologies for silently editing FCI, the sticky said not to reply to your own posts lest they appear answered so I did it with an edit.

I don't know what PM is but I can't replicate any of your javascript solutions on Firefox. If you could tell me how I find MacroPlayer.js I'd be happy to solve this by modifying the URL function directly.
chivracq
Posts: 9306
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by chivracq » Sun Feb 19, 2017 4:44 pm

Housingassocmonkey wrote:Many thanks for such a detailed answer. Sadly AdBlock is not an option as the website blocks access if it detects an adblocker. And filter images doesn't cut enough of the bandwidth. Apologies for silently editing FCI, the sticky said not to reply to your own posts lest they appear answered so I did it with an edit.

I don't know what PM is but I can't replicate any of your javascript solutions on Firefox. If you could tell me how I find MacroPlayer.js I'd be happy to solve this by modifying the URL function directly.
Hum..., about AdBlock not being an Option, Sites blocking Access/Functionality to/of their Site usually do it with some external Script... that you need to identify... and to block, ah-ah...! And I guess this Site is full of JavaScript if you say that 'FILTER' only cuts a small part of the Bandwidth...

Well, "PM" is "Pale Moon", like I mentioned, the Browser I use mainly, it is forked on FF and iMacros for FF works on it..., until v8.8.7 on PM27. It is optimized for Speed and Performance and Reliability compared to FF and doesn't suffer from Memory Leak for example like FF does... :shock:

But OK, if you want to "hack" the iMacros Add-on directly, why not, ah-ah...!, I've explained the "Procedure" in the following Thread. (Hum, Decb 2013, more than 3 years ago already, oops...!):
- Re: Extend Imacros with new commands?
Make sure to make a Backup of iMacros Folders and Macros as if you make any "Mistake", iMacros will become unusable at all and you'll have to reinstall it.

And mini-bit of "Thinking", you might get "lucky" but I think/fear it's not simply a question of adding the Dash from "view-source:" to the RegEx Condition for adding "http://", but the Statement(s) after that part perform(s) some kind of Checking on the Validity of an URL and I wouldn't be surprised if "view-source:..." still gets rejected, so, be careful and it might be a bit more complex than it first looks... :twisted:

"URL GOTO=about:config" for example gets accepted in both FCI's I use, PM and FF, but "URL GOTO=about:preferences" only works in FF (FF51 + v8.9.7) while it still triggers the "BadParameter" RuntimeError in PM (PM26 + v8.8.2), as I guess "about:preferences" got introduced in some recent FF Version only...:
BadParameter: The URL syntax is not correct: 'about:preferences', line 5 (Error code: -911)
(Notice there was no "http://" added to the URL...)

But good luck and I'll be interested of course if you can post your modified Version of the part of Code I posted from 'MacroPlayer.js'... Oh...!, hum, and post the original Code from v9.0.3 as well as the one I posted was from v8.8.2.
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Sun Feb 19, 2017 6:00 pm

So I found the relevant JAR and edited the JS. I changed URL="http://"+URL to URL=URL. Checked the code in an online compiler and it looks like it does what it's supposed to.

However. Firefox won't run iMacros because the edit invalidates the certificate ("iMacros for Firefox could not be verified for use in Firefox and has been disabled."). Even restoring the backup won't revalidate the certificate so I'm going to have to reinstall iMacros.

Which is a shame because that edit would have been the perfect solution. I'm currently downloading the developer version of Firefox to see if the extended tools will let me run an unsigned add on.
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Sun Feb 19, 2017 6:40 pm

iMacros is not compatible with the developer version of Firefox. So that's that out of the window. There is a version called "Nightly" that some people suggest might work, although it says it automatically reports activity to Mozilla so swings and roundabouts.
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Sun Feb 19, 2017 7:07 pm

Firefox Nightly works with an edited .JS file and allows direct data extraction from the source.

Does this forum have a way to mark a problem as solved?
chivracq
Posts: 9306
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by chivracq » Mon Feb 20, 2017 1:22 am

Shitty Co. at this moment, any "Reply/Quote' takes about 10 min to load, and doesn't work and I have to repeat the "Operation" 20 times, but OK, I've seen your Replies, I will reply when I have a "decent" Co..., ah-ah...!
Yep, I now, gigantically decadent for a Web-Automation Specialist, ah-ah...! OOPS...! :oops:
(Is my normal Life actually since 5-8-10 years, I only have about avg 6-8h Access to Internet everyday and still manage to do about 40h-60h "Activity" on Internet everyday, ah-ah...! Everything about Efficiency... ah-ah...!, and I even find some time to help other Users (on this Forum...), Oops...!)
EDIT: Oh...! finally managed to post this Post after pfff, maybe 25 times trying, ah-ah...!
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
chivracq
Posts: 9306
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by chivracq » Mon Feb 20, 2017 4:59 am

Housingassocmonkey wrote:So I found the relevant JAR and edited the JS. I changed URL="http://"+URL to URL=URL. Checked the code in an online compiler and it looks like it does what it's supposed to.

However. Firefox won't run iMacros because the edit invalidates the certificate ("iMacros for Firefox could not be verified for use in Firefox and has been disabled."). Even restoring the backup won't revalidate the certificate so I'm going to have to reinstall iMacros.

Which is a shame because that edit would have been the perfect solution. I'm currently downloading the developer version of Firefox to see if the extended tools will let me run an unsigned add on.
Oh ja, forgot about that..., sorry just found out about it myself very recently (1 "fucking" week ago when I updated from FF47 to FF51), Pb is FF itself, can be circumverted (OK Spellchecker doesn't like it, ah-ah...! (= circumvent + revert)), but pffff, always a hassle...! :shock: :shock: :shock:

Did you try Ctrl^Drag on the Add-ons Interface, was supposed to be the "Developers" Workaround...?
Housingassocmonkey wrote:iMacros is not compatible with the developer version of Firefox. So that's that out of the window. There is a version called "Nightly" that some people suggest might work, although it says it automatically reports activity to Mozilla so swings and roundabouts.
Housingassocmonkey wrote:Firefox Nightly works with an edited .JS file and allows direct data extraction from the source.

Does this forum have a way to mark a problem as solved?
Yep, easy way to mark a Thread as "[Solved]" is simply to add this Tag to your Thread Title... (from your first Post in the Thread)

But, hum..., I would like a little more Info... Please...!! 8)
So simply replacing "URL="http://"+URL" to "URL=URL" solved the whole Pb...!? 8)
=> Concerning the "Pb" I was afraid of about the Checks on "view-source", that you might be "screwing" other Scenarios, but hum, I think you located a Bug as well, if 'view-source:" is accepted by the Validation after the 'IF' Statement, then there is no Reason that the 'IF' Statement stops it, ah-ah...!
Last edited by chivracq on Mon Feb 20, 2017 2:54 pm, edited 1 time in total.
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Housingassocmonkey
Posts: 7
Joined: Sat Feb 18, 2017 9:03 pm

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by Housingassocmonkey » Mon Feb 20, 2017 7:28 am

Yes, the issue wasn't the "-". I'm not sure what that statement does but there is a very short if statement that basically says if the URL doesn't start with HTTP:// then add it. Remove that and it doesn't add it.

It still throws an error if you put junk in the URL, but it can go to url's that don't start HTTP://
chivracq
Posts: 9306
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: URL Goto appends HTTP:// to URL, can it be stopped?

Post by chivracq » Mon Feb 20, 2017 7:22 pm

Housingassocmonkey wrote:Yes, the issue wasn't the "-". I'm not sure what that statement does but there is a very short if statement that basically says if the URL doesn't start with HTTP:// then add it. Remove that and it doesn't add it.

It still throws an error if you put junk in the URL, but it can go to url's that don't start HTTP://
Yep, the Issue is the Dash I think, but never mind, good that you managed to find a workable Workaround for you... 8)

Make sure to disable Automatic Updates for iMacros or you'll have to reapply your Change each time iMacros gets updated, ah-ah...! :idea:
Some thing would apply if you "downgraded" to v8.9.7, because you will probably find out pretty quickly by yourself, but v9.0.3 is pretty buggy and limited compared to v8.9.7 which is much more stable. oops...! :oops:

I have posted in the 'Bugs and EnhReq's' Thread for FF with a Ref to this current Thread...
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
Post Reply