Extract part of an URL and modify this

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
pauloswear
Posts: 7
Joined: Tue May 19, 2020 4:07 pm

Extract part of an URL and modify this

Post by pauloswear » Tue May 19, 2020 7:56 pm

Hi, i am newby yet using scripts... i am trying to extract a part of an URL and modify like this example:
  • - generic website openned with macro function TAG, example: genericsite . com/task/www. genericsite2 .com/user/
    - i want extract only user part: /user/
    - after all this i want add: www. genericsite2 .com/ before user
    final result: www. genericsite2 .com/user/
I am running this code on imacros build 10021450, firefox 76, windows 10 x64 pro ENG.

The code that i am using is:

Code: Select all

CLEAR
SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !REPLAYSPEED FAST
SET !TIMEOUT_PAGE 60
SET !TIMEOUT_STEP 0
SET !LOOP -9999999
SET !ENCRYPTION NO


'----------ID DA CONTA----------------
SET !VAR1 35372878524
'----------AÇÃO A EXECUTAR------------
SET !VAR2 1
'-------------------------------------


TAB T=1
TAB CLOSEALLOTHERS
URL GOTO=https://websitegeneric.com.br/painel/conectar
WAIT SECONDS=2
TAG POS=1 TYPE=SELECT ATTR=ID:conta_id CONTENT=%{{!VAR1}}
TAG POS=1 TYPE=SELECT ATTR=ID:acao_id CONTENT=%{{!VAR2}}
WAIT SECONDS=4
TAG POS=1 TYPE=P ATTR=TXT:Ver<SP>link
WAIT SECONDS=4

'HERE I NEED EXTRACT URL AND MODIFY URL

TAB T=2
WAIT SECONDS=3
TAG POS=1 TYPE=BUTTON ATTR=TXT:Seguir
TAG POS=1 TYPE=BUTTON ATTR=TXT:Follow
WAIT SECONDS=4
TAB T=1
TAB CLOSEALLOTHERS
WAIT SECONDS = 2
TAG POS=1 TYPE=BUTTON ATTR=TXT:Confirmar
WAIT SECONDS=180
chivracq
Posts: 9425
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract part of an URL and modify this

Post by chivracq » Tue May 19, 2020 9:28 pm

pauloswear wrote:
Tue May 19, 2020 7:56 pm
Hi, i am newby yet using scripts... i am trying to extract a part of an URL and modify like this example:
  • - generic website openned with macro function TAG, example: genericsite . com/task/www. genericsite2 .com/user/
    - i want extract only user part: /user/
    - after all this i want add: www. genericsite2 .com/ before user
    final result: www. genericsite2 .com/user/
I am running this code on

Code: Select all

imacros build 10021450, firefox 76, windows 10 x64 pro ENG.
The code that i am using is:

Code: Select all

CLEAR
SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
SET !REPLAYSPEED FAST
SET !TIMEOUT_PAGE 60
SET !TIMEOUT_STEP 0
SET !LOOP -9999999
SET !ENCRYPTION NO


'----------ID DA CONTA----------------
SET !VAR1 35372878524
'----------AÇÃO A EXECUTAR------------
SET !VAR2 1
'-------------------------------------


TAB T=1
TAB CLOSEALLOTHERS
URL GOTO=https://websitegeneric.com.br/painel/conectar
WAIT SECONDS=2
TAG POS=1 TYPE=SELECT ATTR=ID:conta_id CONTENT=%{{!VAR1}}
TAG POS=1 TYPE=SELECT ATTR=ID:acao_id CONTENT=%{{!VAR2}}
WAIT SECONDS=4
TAG POS=1 TYPE=P ATTR=TXT:Ver<SP>link
WAIT SECONDS=4

'HERE I NEED EXTRACT URL AND MODIFY URL

TAB T=2
WAIT SECONDS=3
TAG POS=1 TYPE=BUTTON ATTR=TXT:Seguir
TAG POS=1 TYPE=BUTTON ATTR=TXT:Follow
WAIT SECONDS=4
TAB T=1
TAB CLOSEALLOTHERS
WAIT SECONDS = 2
TAG POS=1 TYPE=BUTTON ATTR=TXT:Confirmar
WAIT SECONDS=180

Alright, this one opened in the "correct" Sub-Forum, and you have read the Forum Rules and now also mention your FCI..., very good...! :D

Hum, 'Free'/'PE' is still missing from your FCI...(?)

But OK, yep, what you want is fairly easy to implement, you simply need the '!URLCURRENT' + 'EVAL()' Commands, and inside 'EVAL()' some JavaScript String Method(s)...
Different Solutions include using (directly or a Combination of) 'replace()' / 'split()' / 'substring()' / 'substr()' / 'slice()' / 'search()' / indexOf()' / 'lastIndexOf()..., and there are probably other Solutions also... :idea:
I would go for 'split()' myself, but it's a "personal" Preference and they would all work... 8)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
pauloswear
Posts: 7
Joined: Tue May 19, 2020 4:07 pm

Re: Extract part of an URL and modify this

Post by pauloswear » Tue May 19, 2020 10:21 pm

I know it's not cool to ask, but could you help me with the code that performs this function? I don't understand javascript or imacros
Its a freeware version btw.
Thanks!
chivracq
Posts: 9425
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract part of an URL and modify this

Post by chivracq » Tue May 19, 2020 10:58 pm

pauloswear wrote:
Tue May 19, 2020 10:21 pm
I know it's not cool to ask, but could you help me with the code that performs this function? I don't understand javascript or imacros
Its a freeware version btw.
Thanks!

OK for iMacros for FF v10.0.2 'Free' about your FCI...

Well, nope, I don't write Code or their Script for other Users, or very rarely, you need to do "your part of the Job", I (only) help Users who "try their best" and "really get stuck"..., but you haven't tried anything yet..., ah-ah...! :wink:
The Forum contains already Dozens of similar Threads like yours with complete Solutions btw, if you search the Forum a bit... :idea:

And "generic Scenario" / "generic Site", => you only get "generic Advice" from me, tja...! :P

And hum..., I'm already being "nice" because I usually don't help at all for Social Media and Like/Follow/Comment/etc... 8)
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
pauloswear
Posts: 7
Joined: Tue May 19, 2020 4:07 pm

Re: Extract part of an URL and modify this

Post by pauloswear » Thu May 21, 2020 4:22 am

Ok, i was trying to learn some JS and imacros.. I was thinking in split the last part of url and put this in a matrix..

i did this function, and this worked, but i still dont know how to implement this on imacros
function getUrl() {
var link = window.location.href.slice(0, -1)
link = link.split('/')
var user = link.pop() || link.pop();
var urlfinal = 'www.url.com' + "/" + user
console.log(urlfinal);
return urlfinal;
}
I am still trying to get a progress...

edit:

still thinking in something like this
SET !VAR3 EVAL("'www.url.com' + '/'+window.location.href.slice().split('/').pop()")
chivracq
Posts: 9425
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract part of an URL and modify this

Post by chivracq » Thu May 21, 2020 11:37 am

pauloswear wrote:
Thu May 21, 2020 4:22 am
Ok, i was trying to learn some JS and imacros.. I was thinking in split the last part of url and put this in a matrix..

i did this function, and this worked, but i still dont know how to implement this on imacros

Code: Select all

       function getUrl() {
    var link = window.location.href.slice(0, -1)
    link = link.split('/')
    var user = link.pop() || link.pop();
    var urlfinal = 'www.url.com' + "/" + user
    console.log(urlfinal);
    return urlfinal;
}

I am still trying to get a progress...

edit:

still thinking in something like this

Code: Select all

SET !VAR3 EVAL("'www.url.com' + '/'+window.location.href.slice().split('/').pop()")

Alright, you are indeed making some Progress... :D

The Function taken from SOF is a bit Overkill but your 'EVAL()' is getting very close...!

A few Remarks:
1- The 'window.location.href' is pure JS, and that would/will work in pure JS or from iMacros in a '.js' Script (not supported anymore in v10.0.2 for FF, '.js' Scripts are only supported until v9.0.3 for FF) or with the 'URL GOTO=javascript' Syntax, but passing Vars back from 'URL GOTO=javascript' to the '.iim' Script is a bit cumbersome and not very straightforward.
And I'm not sure you can access the DOM from 'EVAL()' and that 'window.location.href' will work in 'EVAL()'. You will probably get some Error that "var "window" is undefined/not referenced"...

But "Luckily" for you, this is what the Built-in Var '!URLCURRENT' that I had already mentioned, does for you, ah-ah...! You don't need to reinvent the Wheel, ah-ah...! :idea:

2- Then OK, to isolate the "user" Data/String from the first URL, you go for 'split()' as the "main" Mechanism, good-good..., like I had mentioned... :D
... And you combine it with 'pop()' + 'slice()'... :o
Hum, OK, why not..., I had not thought of this Way, I find it a little bit cumbersome in this Case, (as 'split()' alone would already get the Result), but it's not a "bad Idea"..., especially using 'pop()' that will return the last Item of the Array returned by the 'split()'. 8)

And you indeed then need the "slice(0,-1)" like in the Function, to first remove the last "/" after the "user" in the URL.
=> The "slice()" without Argument(s) that you use in your 'EVAL()' does nothing, you do need the 2 Args like in your Function to remove the final "/", otherwise, the "pop()" will return an Empty String as the URL ends on a "/" which is the Char you split the URL/String on... :idea:

Alright, keep up the "good Work", you are getting very close...! :wink:
Last edited by chivracq on Thu May 21, 2020 1:47 pm, edited 1 time in total.
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
pauloswear
Posts: 7
Joined: Tue May 19, 2020 4:07 pm

Re: Extract part of an URL and modify this

Post by pauloswear » Thu May 21, 2020 12:33 pm

Ok, I switched the browser and iMacros to last Basilisk and iMacros 8.9.7
Finally i did this

Code: Select all

SET !VAR3 EVAL("'www.website.com' + '/'+'{{!URLCURRENT}}'.slice(0, -1).split('/').pop()")
URL GOTO={{!VAR3}}
This worked, but now i want to do some implementations like if and else.. If the "text" is present in url, i want to execute a comand, else, i want to go back to line 0 of code...
I was reading about TIMEOUT_STEP 0. Is there some internal function to do this job or can i use {{!TIMEOUT_STEP}} in the parameter to go back to step 0?
chivracq
Posts: 9425
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extract part of an URL and modify this

Post by chivracq » Thu May 21, 2020 2:06 pm

pauloswear wrote:
Thu May 21, 2020 12:33 pm
Ok, I switched the browser and iMacros to last Basilisk and iMacros 8.9.7
Finally i did this

Code: Select all

SET !VAR3 EVAL("'www.website.com' + '/'+'{{!URLCURRENT}}'.slice(0, -1).split('/').pop()")
URL GOTO={{!VAR3}}
This worked, but now i want to do some implementations like if and else.. If the "text" is present in url, i want to execute a comand, else, i want to go back to line 0 of code...
I was reading about TIMEOUT_STEP 0. Is there some internal function to do this job or can i use {{!TIMEOUT_STEP}} in the parameter to go back to step 0?

Yep, very good...!! :D

And..., like I mentioned that 'split()' alone could do the job also, instead of your "slice(0, -1).split('/').pop()", you could also simply use "split('/')[5]", as "user" happens to be located exactly between 2x '/'... But your Solution is perfect...

>>>

Then, yep, iMacros for FF v8.9.7 is indeed the "best" Version to use, it is much more Functional than v10.0.2 'Free', and it indeed works on 'Basilisk' v2020, and on 'Pale Moon' v28 also... (I use it myself also, but still on FF v55.0.3.)

>>>

For your 'if'/'else', I would think you can better start a New Thread as it won't have anything/much to do with the "original" Content/Purpose of this current Thread..., but hum, '!TIMEOUT_STEP' is a Timeout Command, it has nothing to do with "going back to Step 0"... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE').
- I don't even read the Qt if that (required) Info is not mentioned...!
- Script & URL help a lot for more "educated" Help...
pauloswear
Posts: 7
Joined: Tue May 19, 2020 4:07 pm

Re: Extract part of an URL and modify this

Post by pauloswear » Thu May 21, 2020 2:55 pm

mm... I will try a little bit before open new topic.. Maybe the best solution is set timeout to 0, this will give me a new loop

Btw, thanks for all support
Post Reply