Extracting prices from flight comparison sites - Possible?

Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.
Forum rules
iMacros EOL - Attention!

The renewal maintenance has officially ended for Progress iMacros effective November 20, 2023 and all versions of iMacros are now considered EOL (End-of-Life). The iMacros products will no longer be supported by Progress (aside from customer license issues), and these forums will also no longer be moderated from the Progress side.

Thank you again for your business and support.

Sincerely,
The Progress Team

Before asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information: CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST
Post Reply
millenialme
Posts: 4
Joined: Mon Apr 16, 2018 9:15 pm

Extracting prices from flight comparison sites - Possible?

Post by millenialme » Mon Apr 16, 2018 9:24 pm

Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

Can it be done?
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extracting prices from flight comparison sites - Possibl

Post by chivracq » Mon Apr 16, 2018 10:11 pm

millenialme wrote:Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

Can it be done?
Yep of course, iMacros is for Web-Automation and Data Extraction...! :D

:arrow: Why do you think we have a Sub-Forum dedicated to "Data Extraction and Web Screen Scraping"...!?, that you managed to find very correctly to open your Thread...! :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
millenialme
Posts: 4
Joined: Mon Apr 16, 2018 9:15 pm

Re: Extracting prices from flight comparison sites - Possibl

Post by millenialme » Tue Apr 17, 2018 6:36 am

chivracq wrote:
millenialme wrote:Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

Can it be done?
Yep of course, iMacros is for Web-Automation and Data Extraction...! :D

:arrow: Why do you think we have a Sub-Forum dedicated to "Data Extraction and Web Screen Scraping"...!?, that you managed to find very correctly to open your Thread...! :idea:
Brilliant!

I'm very new and lack a lot of knowledge when it comes to this. I experienced some iMacro use in a previous job and saw the potential for it to be used in this way. I'm in no way an expert in stuff like this, quite the contrary.

Is there anyone here capable of scripting something like this? Or do any of you know a freelancing type site which is reliable for posting jobs such as this?
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extracting prices from flight comparison sites - Possibl

Post by chivracq » Tue Apr 17, 2018 6:12 pm

millenialme wrote:Brilliant!

I'm very new and lack a lot of knowledge when it comes to this. I experienced some iMacro use in a previous job and saw the potential for it to be used in this way. I'm in no way an expert in stuff like this, quite the contrary.

Is there anyone here capable of scripting something like this? Or do any of you know a freelancing type site which is reliable for posting jobs such as this?
Yep, it sounds like a "useful" Use of iMacros indeed, and a "Fun" Project..., but you should give it a try yourself, it's not very-very complicated, and it's Fun to make your own Script... 8)

You can post (open a new Thread) otherwise/anyway in the 'iMacros Consultants' Sub-Forum and you'll probably get a few Developers interested... (And a few have already posted their Contact Info in that Sub-Forum...)
(And mention a way to contact you as PM is disabled on the Forum...)

(I'd be willing to do it if you don't get any Reactions or don't find anybody after, say..., 1 week or so..., but...,, hum..., Scripts I make for other Users usually end up as a "never-ending" Project in my Exp as I'm a bit too much of a "Perfectionist" myself I guess, and in my Exp, Users always come up with extra Requirements halfway, which requires a Rewrite/Rethinking of the Script, so you'd need to formulate upfront exactly what you want and what your Req's are... (And it's much "easier" to make a Script for yourself than for sbd else, ah-ah...!))

OK, good luck anyway...! :wink:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extracting prices from flight comparison sites - Possibl

Post by chivracq » Wed Apr 18, 2018 6:25 am

Hum-hum-hum...!, just got a "Creative" Idea, ah-ah...! :oops:

But, hum... (lots of "hum", hum-hum...!, I think it usually means I'm "thinking" while typing...), I like your "Project", as I find it "useful", and a perfect Use of iMacros, or even a perfect Usecase for iMacros... 8)

So if you are not already working on your Script yourself and still want to "get it done", my "Creative Idea" would "misuse" a little bit your Project for some little "Project" of mine, but that would get you your Script for say, half the Price you had in mind, and you would even get "the Best of all Worlds", ah-ah...! :twisted:
=> You can send me a Mail @ [My_Pseudo_on_the_Forum]@gmail.com and I can explain the Details... :idea:
Hum, and you can start yourself in your Mail with what Price you had in mind and wording your Req's (with concrete Examples and URL(s)) for your Project of course... :idea:
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
chivracq
Posts: 10301
Joined: Sat Apr 13, 2013 1:07 pm
Location: Amsterdam (NL)

Re: Extracting prices from flight comparison sites - Possibl

Post by chivracq » Wed Apr 18, 2018 5:08 pm

Ah OK, I notice you opened a Thread in the 'Consultants' Sub-Forum about 1/2h after my previous Post, so you are not going to try to make the Script yourself, and I guess you are not really interested by my "Creative Idea", ah-ah...! OK, fair enough... 8)

For those interested, Thread in the 'Consultants' Sub-Forum:
- Looking for help on a project - Paid opportunity
millenialme wrote:I am looking to develop a script for extracting price data from flight comparison sites such as Google flights.

I love to travel and have spent hours in the past trying to find good deals for flights to various countries. Like a lot of people I have signed up to Jack's flight club Scotts cheap flight etc but I'd like to be able to target the places I actually want to travel to.

After working a job recently I was introduced to iMacros and saw the potential for this application. However, my very limited knowledge of writing scripts has made it very difficult to get this to work. That's why I'm looking for some help!

The intension is to run the script a few times each week to try and find the best deals for flights from UK airports to various countries around the world.
An example would involve running a search for:

Code: Select all

Flights from LHR to LAX 
Date 07/06/18 - 21/07/18
The prices for these flights would then be extracted into excel to be processed to find the cheapest deals

I have spent time collating information for every 3 letter airport code on the planet ready for use as well as dates to be input.

Any help or ideas about how to approach this would be appreciated!

Please contact me if you're interested, want more details or would like to talk numbers - josshillary94@gmail.com
(You could have posted a mini-Update and that Link yourself, @OP, minus-Point for me to get "motivated" for your Project... :idea: )

Hum..., no URL(s) mentioned from which Site(s) you usually use and want to extract your Data from... (I had asked for those...!)
iMacros is very dependent on the HTML Structure of a Site/Page, it makes a "huge" Difference if you were using some Site able to check several Air Companies, or if you were checking several Air Companies one by one from their own Web-Site... (Every Web-Site needs a separate Script...)

The Req's are not very "precise"... (Sorry, was me trying to be "diplomatic" a wee bit, I meant "completely vague", ah-ah...!)
=> Duration of your Stay not mentioned...?, Return Flight as well...?, + within the 6 weeks Date-span mentioned or are the Dates only for Departure...?, Input Destination as a fixed List of 1 or a few specific Countries/Cities or for all Countries/Cities in the World...?
You should provide as well the fixed List of Departure Airports in your Order of Preference... etc...

Advice of the Day:
Every 5 min you don't spend on defining your Req's precisely will cost you at least $100 with any Professional Developer, in back&fro Communication trying to clarify, or they will "take stg for granted" and will implement it how they "interpreted" your vague Formulation which in 80% of the cases won't be what you expected, Developer spent a few hours implementing the 1st Version, not good, need a 2nd Version, Developer a bit pissed off because of wasted time, not paid, Motivation goes down, and you'll end up with some "half-Product"...! :idea:

You should mention what Browser(s) you intend you use... (and your OS...!)
Recommended Browser Choice:
1. FF (Firefox), until FF56 + iMacros for FF v8.9.7 (or v9.0.3). (Free) (There is no iMacros Version at the moment working on FF57(+)...)
2. IE11 + iMacros for IE v12.0. (Free)
3. CR65 (CR=Chrome) + iMacros for CR v10.0.1 PE. (PE = Personal Edition = About $100, not available for MacOS nor Linux, only Win32/64.)

[LHR = London Heathrow (UK) / LAX = Los Angeles (US)]
- (F)CI(M) = (Full) Config Info (Missing): iMacros + Browser + OS (+ all 3 Versions + 'Free'/'PE'/'Trial').
- FCI not mentioned: I don't even read the Qt...! (or only to catch Spam!)
- Script & URL help a lot for more "educated" Help...
millenialme
Posts: 4
Joined: Mon Apr 16, 2018 9:15 pm

Re: Extracting prices from flight comparison sites - Possibl

Post by millenialme » Thu Apr 19, 2018 7:41 pm

Thanks for all of your advice!

The reason my post is so vague is because purely because I came up with the idea but need somewhere to start!

My knowledge of iMacros is so limited, all I know is that I can achieve this by using a well written script, however I do not possess the skill to make such a script.

As for details:

Browser - early Firefox in order to use free version of iMacros. However, if this largely impacted functionality I would be willing to use alternatives.

Google flights - https://www.google.co.uk/flights

The break down of dates for flights would be:
Long weekends (Friday to Monday)
1 week trips
2 weeks trip

e.g. an example search may involve:
Search 1
(NAVIGATE TO) Departing (INPUT) LHR
(NAVIGATE TO) Arriving (INPUT) LAX
(NAVIGATE TO) Departing date (INPUT) 31/05/18
(NAVIGATE TO) Return date (INPUT) 14/05/18
Then to extract data for the price of the cheapest flight and possibly also the airline an time of flight.

This information would then be used to collate the best deals for a number of destinations. E.g. three 'trip duration' variations would run for each, long weekend, 1 week and 2 weeks. 5 different destinations would be checked during 1 session. For example one week I may focus on North America and want prices for LA, Florida, NYC, Seattle and Boston. These searches would be made for a certain time window, e.g. various dates between May 31st and June 31st.

How is that looking? Things a little more clear now?

Apologies for my rookie mistakes!
Post Reply