Extracting multiple pages from a website

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
databasepro
Posts: 18
Joined: Sat May 27, 2006 5:18 pm

Extracting multiple pages from a website

Post by databasepro » Sat May 27, 2006 5:23 pm

I am completely new to IMacros, but I think it could be very helpful to me in a variety of projects I am working on.

I have tried repeatedly to get this to work, but there is something I am missing.

What I want to do is to:

1) go to this website http://www.legis.state.la.us/billdata/b ... &startno=1

2) beginning with the first bill, I want to extract the entire "table", line by line to a text file. Each column would be separated by a comma. Each line, representing all of the information for one bill, ending in a hard return.

3) at the bottom of the page there is the word "More". This brings you to the next page of data. I want to continue the extracting and saving process until I have gone through every page on the website.

Thanks in advance for any help on this.

Eventually, it would be nice to have this automated within Microsoft Access. Which version of iMacros would I need.

/K
User avatar
Tech Support
Posts: 4948
Joined: Tue Sep 20, 2005 7:25 pm
Contact:

Post by Tech Support » Mon May 29, 2006 10:14 am

Hi,

would need to loop the macro. The macro would consist basically of three lines, one line with the EXTRACT command, one with the SAVEAS command and one with the TAG command which clicks the more link. You couls run this macro until it stops (with an error), because at some point the will be no more "More" link.

If you want to start and control iMacros from Access, you will need the Scripting Edition.

Regards
---
Lasse Clausen, iOpus Support
databasepro
Posts: 18
Joined: Sat May 27, 2006 5:18 pm

Thanks for the reply

Post by databasepro » Fri Jun 02, 2006 8:54 pm

I still am not getting how I would go from one row to the next getting each line of data and saving to the csv file.

And then I would have to go to the next page.

Do I use the play loop mechanism or do I manually code it in the macro?

I am still stuck.

Thanks for any help.
User avatar
Tech Support
Posts: 4948
Joined: Tue Sep 20, 2005 7:25 pm
Contact:

Post by Tech Support » Tue Jun 06, 2006 6:51 am

Hi,

usually you start by recording one example case of set of pages you want to automate. You then need to look at the macro code created and find those parts where you need to use variables instead of hard coded text, like the name you search for or the certain results link you clicked. It might also be helpful to split one macro into several small ones.

If you extract an entire table and then use the SAVEAS TYPE=EXTRACT command, all table HTML is converted into the format you described, ie column separators are commas and row separators are newlines.

Regards
---
Lasse Clausen, iOpus Support
Post Reply