Internet Job Board needing to scrape Multiple Websites

Find a consultant or advertise your services here
Post Reply
Posts: 1
Joined: Fri Aug 28, 2015 2:53 pm

Internet Job Board needing to scrape Multiple Websites

Post by dfreye » Fri Aug 28, 2015 3:51 pm

We currently use Kapow Katalyst to scrape 26 separate websites for job postings. Because of the price of Kapow, we are looking for alternatives to do website scraping. We have looked at iMacros and have even had a javascript built with it within the firefox add-on but think we may need more power over it.

We need everything to be completely automated, either by leveraging the .NET component so we can write these using Visual Basic that we can put into a SSIS package that we can schedule on a daily basis or some other version of iMacros that will give us what we need as far as automation goes.

Each site will write to its own .xml file or .csv file or better yet, could write the jobs directly to our SQL Database into separate tables based on the client.

We use one set of requirements for all our sites to make it easy, but some sites may lack one component or another which we try and make up for on the backend within SQL.

We need to be able to run multiple instances at one time so that all are finished within a set time. Typically I have been controlling the timing by batching 4 or more sites together based on time it takes to run each.

What we would like is to give you the opportunity to show us what can be done to scrape different sites. We would like you to setup our environment on an existing Amazon EC2 for development if its complicated and write the first scrape. We would like for this to be a reasonable flat charge and if everything works the way we need it, we can give you a list of our other 25 sites and requirements and keep you busy while converting our robots to iMacros. After these are all converted, you would be our go-to person for any new clients and scrapes we add for a flat-rate per scrape build. Our current Kapow guy has been with us for 3 years so we would want the same kind of relationship. Typically we have one or two a week but sometimes it could be slow.

Please feel free to reach out to me directly at Our website is so you can get an idea of what we do and what we are looking for for each job.
Post Reply