I need to download a lot of xls files(around 10000)

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Moderators: Community Moderators, iMacros Moderators

Forum rules
Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Answering your own posts (e.g. attempting to "bump" your topic) drops your topic from the list of unanswered threads, so it may actually receive less views.

I need to download a lot of xls files(around 10000)

by fengfan on Tue Mar 29, 2016 8:38 pm

I need to download a lot of xls files(around 10000), and it takes around 30 seconds to finish one downloading routine.
So the total time I need is 10000x30s = 80hours!!! It's a really frustrating number.
I wonder if I can speed up this process?? Thanks in advance
fengfan
 
Posts: 1
Joined: Tue Mar 29, 2016 8:25 pm

Re: I need to download a lot of xls files(around 10000)

by thecoder2012 on Thu Mar 31, 2016 2:56 am

fengfan wrote:I need to download a lot of xls files(around 10000), and it takes around 30 seconds to finish one downloading routine.
So the total time I need is 10000x30s = 80hours!!! It's a really frustrating number.

Any example is very helpful.

fengfan wrote:I wonder if I can speed up this process?? Thanks in advance

Use more tabs in your browser with imacros.
Join 9kw.eu Captcha Service now and let your iMacros continue downloads and scripts while you sleep. - Custom iMacros? Contact me!
User avatar
thecoder2012
 
Posts: 109
Joined: Sat Aug 15, 2015 10:14 am
Location: Internet

Re: I need to download a lot of xls files(around 10000)

by techimac on Sat Apr 02, 2016 2:51 pm

fengfan wrote:I need to download a lot of xls files(around 10000), and it takes around 30 seconds to finish one downloading routine.
So the total time I need is 10000x30s = 80hours!!! It's a really frustrating number.
I wonder if I can speed up this process?? Thanks in advance


scrape all download links and then use download manager which allows to input bulk url's (HiDownload is one)
techimac
 
Posts: 189
Joined: Fri Feb 20, 2015 2:27 pm


Return to Data Extraction and Web Screen Scraping

Who is online

Users browsing this forum: No registered users and 3 guests

-->