I need to extract data from a site. There is a simple SELECT option, an INPUT, and a SEARCH BUTTON. So far so good.
Now, the result pages, are not paginated in the traditional way. There is no link to Page 1/2/...., only next page button, which is an input, and it acts like a submit. This input has a NAME attribute, which, on the last page is missing. This is fine with me, because the macro will stop automatically once finished.
I need to go through all the pages, and extract the data. Now bear with me, I've started to look into iMacros only yesterday, so I'm not really familiar with all the options..
This is the code I've came up so far, and it works for the first page:
Code: Select all
VERSION BUILD=8970419 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET !ERRORIGNORE YES
URL GOTO=https://www.eofcom.admin.ch/eofcom/public/searchEofcom_InaFree.do
' select option 2 from the dropdown
TAG POS=1 TYPE=SELECT FORM=ID:chofcomelicensingpresentationnainaformEofcomInaSearchForm ATTR=NAME:nrt CONTENT=%2
' insert value into input box
TAG POS=1 TYPE=INPUT:TEXT FORM=ID:chofcomelicensingpresentationnainaformEofcomInaSearchForm ATTR=NAME:pnp CONTENT=0002
' submit search
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:chofcomelicensingpresentationnainaformEofcomInaSearchForm ATTR=NAME:doSearchFreeByNumber
' *** this is the block I would need to loop through
' get out the text from the result table
TAG POS=1 TYPE=TBODY ATTR=TXT:* EXTRACT=TXT
' remove all the spaces
SET !EXTRACT EVAL("'{{!EXTRACT}}'.replace(/\s/g, \"\");")
' show the result
PROMPT "{{!EXTRACT}}"
' goto next page
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:chofcomelicensingpresentationnainaformEofcomInaSearchForm ATTR=NAME:getNextInaPage
' *** end block of loop
Code: Select all
SET !LOOP 25
From my researches over the net couldn't figure out how to achieve that. Any help would be greatly appreciated.
I am using the free iMacros Firefox extiension (latest, downloaded yesterday), in Debian 8.5, but I also have an XP machine with the trial iMacros Version 11.1.
Note: before someone would complain about legal issues related to scraping data, I have to mention that my company contacted the owners of the site, and asked for an API, which is not available, and they suggested to use the search page instead.
Thanks.