Searching through a webpage using iMacros / Javascript

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

Searching through a webpage using iMacros / Javascript

by Benzin on Thu Jun 23, 2016 10:05 am

Hey everyone,

I know many of you probably hate it when new people come here and expect to get their problems solved, but I've spent so many hours on this problem and something tells me that it can't be that difficult to solve.

What I was trying to do is something very simple: I wanted to create a macro with iMacros that is able to go through a webpage, search for a certain word and if the word is found, click on a button. Else, refresh the page and start again.

I realized that there is no easy way to use if/else on iMacros, so that I'll need to add this with Javascript. Still, my major problem is found in the macro. Basically, if I use the extract option on iMacros, it will only skim through the HTML code of the page, if I understand it correctly. This means that it will need a certain TYPE. The problem is that this webpage I am looking to skim through does update time after time but does not show these new "results" in the HTML code. I think it uses some kind of Javascript or php, not showing the text I want to find in the HTML code. How do I still program iMacros to simply search through the "visible" text on the page (in a CTRL+F kind of way)?

The macro I had implemented in my .js file that is responsible for extracting looks like this:

VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=A ATTR=TXT:*Text* EXTRACT=TXT

In this example I try and search for the word [...]Text[...] If I try and use the wildcard TYPE=* it simply does not work.

So: Is there a way to search on a page and not only the HTML code with iMacros? How would I do it?
If it doesn't work wit iMacros, is there a way to simply do it with Javascript?

Thanks in advance.
Benzin
 
Posts: 3
Joined: Thu Jun 23, 2016 10:03 am

Re: Searching through a webpage using iMacros / Javascript

by chivracq on Thu Jun 23, 2016 12:20 pm

Benzin wrote:Hey everyone,

I know many of you probably hate it when new people come here and expect to get their problems solved, but I've spent so many hours on this problem and something tells me that it can't be that difficult to solve.

What I was trying to do is something very simple: I wanted to create a macro with iMacros that is able to go through a webpage, search for a certain word and if the word is found, click on a button. Else, refresh the page and start again.

I realized that there is no easy way to use if/else on iMacros, so that I'll need to add this with Javascript. Still, my major problem is found in the macro. Basically, if I use the extract option on iMacros, it will only skim through the HTML code of the page, if I understand it correctly. This means that it will need a certain TYPE. The problem is that this webpage I am looking to skim through does update time after time but does not show these new "results" in the HTML code. I think it uses some kind of Javascript or php, not showing the text I want to find in the HTML code. How do I still program iMacros to simply search through the "visible" text on the page (in a CTRL+F kind of way)?

The macro I had implemented in my .js file that is responsible for extracting looks like this:

Code: Select all
VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !EXTRACT_TEST_POPUP NO
TAG POS=1 TYPE=A ATTR=TXT:*Text* EXTRACT=TXT


In this example I try and search for the word [...]Text[...] If I try and use the wildcard TYPE=* it simply does not work.

So: Is there a way to search on a page and not only the HTML code with iMacros? How would I do it?
If it doesn't work wit iMacros, is there a way to simply do it with Javascript?

Thanks in advance.

FCIM...! :mrgreen:
=> iMacros for FF v8.9.7, FF46/47...?, OS...?

You don't need any JavaScript, there are several ways to achieve what you want in pure '.iim'.

1- iMacros has a 'SEARCH' Command that you can use for that.

2- I don't like the 'SEARCH' Command that I find complicated to use, I prefer to issue a
Code: Select all
TAG POS=1 TYPE=* ATTR=* EXTRACT=TXT
or
Code: Select all
TAG POS=1 TYPE=* ATTR=* EXTRACT=HTM
and using 'EVAL()', to do a 'split()' on your Keyword(s), and then check the 'length' of the Array returned by 'split()'.

3- You can as well do an 'EXTRACT' like in '2-' and using 'EVAL()' again, check on '==/!=' "#EANF#" to spit out a 1/0 to reuse for the 'TAG POS=n' to click your Button (+ '!ERRORIGNORE').

4- Or simply use (Double) Relative Positioning (+ '!ERRORIGNORE') but you indeed cannot use 'TYPE=*' as the Anchor as it will find the highest HTML Element containing the whole Page and there are no other Elements outside of it to use for Relative Positioning, you cannot even get out of it. A 'TYPE=DIV' would be more appropriate, here is an Example that will open the 'SEARCH FORUM' Page if "Positioning" is found on this Current Page:
Code: Select all
VERSION BUILD=8820413 RECORDER=FX
TAB T=1
URL GOTO=http://forum.imacros.net/viewtopic.php?f=8&t=26381

TAG POS=1 TYPE=DIV ATTR=TXT:*Positioning*
WAIT SECONDS=2
TAG POS=R1 TYPE=* ATTR=TXT:*

TAG POS=R-1 TYPE=A ATTR=TXT:Search<SP>Forum
(Tested on iMacros for FF v8.8.2, Pale Moon v26.1.1 (=FF43), Win10-x64.)
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6474
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Searching through a webpage using iMacros / Javascript

by Benzin on Thu Jun 23, 2016 3:43 pm

First of all, thanks a lot for your reply!

Sorry that I didn't post all the info. Yes, I'm running iMacros for FF v8.9.7, on FF 47, Windows 7 64bit.

I tried using your 4- approach, because it was the only one I managed to understand (kinda). It worked! I am really impressed and disappointed in myself.

I have a few questions though:

If I want to have it run in a loop and not stop when the text is not found on the page, do I use "SET !ERRORIGNORE YES"?

The last minor problem I am encountering is the following: Sometimes multiple strings of text with buttons on the side appear after the page is reloaded. For some reason the macro finds the text on the first result, but doesn't click the first button. Actually, it seems to always be clicking the last one.

The site pretty much looks like this:

http://i.imgur.com/bjvan7s.jpg

This is my macro:

Code: Select all
VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !ERRORIGNORE YES
URL GOTO=https://workplace.clickworker.com/en/clickworker/dashboard

TAG POS=1 TYPE=DIV ATTR=TXT:*Text 1*

WAIT SECONDS=2
TAG POS=R1 TYPE=* ATTR=TXT:*

TAG POS=R-1 TYPE=A ATTR=TXT:Start!



If I set my macro like this, it doesn't click on the first Start! button in the Text 1 row. It seems to always click on the last Start! button, basically the one in the Text 3 row. How do I avoid this?

And the most important question: How do I make it stop looping once the button click was successful? Or maybe have it show a pop-up.
Benzin
 
Posts: 3
Joined: Thu Jun 23, 2016 10:03 am

Re: Searching through a webpage using iMacros / Javascript

by chivracq on Thu Jun 23, 2016 7:07 pm

Benzin wrote:First of all, thanks a lot for your reply!

Sorry that I didn't post all the info. Yes, I'm running
Code: Select all
iMacros for FF v8.9.7, on FF 47, Windows 7 64bit.


I tried using your 4- approach, because it was the only one I managed to understand (kinda). It worked! I am really impressed and disappointed in myself.

I have a few questions though:

If I want to have it run in a loop and not stop when the text is not found on the page, do I use "SET !ERRORIGNORE YES"?

The last minor problem I am encountering is the following: Sometimes multiple strings of text with buttons on the side appear after the page is reloaded. For some reason the macro finds the text on the first result, but doesn't click the first button. Actually, it seems to always be clicking the last one.

The site pretty much looks like this:

http://i.imgur.com/bjvan7s.jpg

This is my macro:

Code: Select all
VERSION BUILD=8970419 RECORDER=FX
TAB T=1
SET !ERRORIGNORE YES
URL GOTO=https://workplace.clickworker.com/en/clickworker/dashboard

TAG POS=1 TYPE=DIV ATTR=TXT:*Text 1*

WAIT SECONDS=2
TAG POS=R1 TYPE=* ATTR=TXT:*

TAG POS=R-1 TYPE=A ATTR=TXT:Start!


If I set my macro like this, it doesn't click on the first Start! button in the Text 1 row. It seems to always click on the last Start! button, basically the one in the Text 3 row. How do I avoid this?

And the most important question: How do I make it stop looping once the button click was successful? Or maybe have it show a pop-up.

Yep, use '!ERRORIGNORE' indeed to keep your Macro looping...

Well for the Script, I just produced an Example working on this Current Page of the Forum, I cannot access your Site which requires Login&Password, but every Script is always very dependent on the HTML Structure of the Page.
My Example takes the first/highest DIV which contains the Keyword and it apparently works as well in your Case, but you need to record your Actions and there is probably some other inner DIV or SPAN Element containing your Keyword for which even Simple Relative Positioning might be enough.
Upload some HTML Saveas of your Page if you don't come out by yourself...
- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...
chivracq
 
Posts: 6474
Joined: Sat Apr 13, 2013 6:07 am
Location: Amsterdam (NL)

Re: Searching through a webpage using iMacros / Javascript

by Benzin on Fri Jun 24, 2016 9:13 am

Thanks again for your reply.

I have uploaded a html file of the page I start from:
Click

It's basically what I showed in the pic yesterday, but though not all elements are shown if you open the file. Shouldn't be a problem though.
Regarding the problem with the macro always clicking the last button: Could that have something to do with relative positioning? Sadly, I didn't fully understand how relative positioning works.

My biggest problem is that I need the macro to stop looping once it has successfully clicked the button. Because this way it just keeps doing the same thing, hence going back to the dashboard. This is why I thought about doing if with some kind of if/else, but I didn't manage to do that myself.
Benzin
 
Posts: 3
Joined: Thu Jun 23, 2016 10:03 am


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: Bing [Bot] and 5 guests

-->