Supporting airline ticket purchase: Buy now or wait?

Date of analysis: May 28, 2008

Table of Contents

1. Problem statement
2. Data collection and processing
   2.1. Data collection
   2.2. Missing value processing
   2.3. Dependent variables and Independent variables
   2.4. Data reduction: factor analysis
   2.5. Final data
3. Analysis
   3.1 Scatter plot diagram
   3.2 Decision Tree
4. Summary

Problem statement

Airplane ticket prices fluctuate according to various conditions, and airlines do not disclose pricing policies, so it is difficult to forecast prices. Therefore, even if you board the same plane, the amount of money you pay for a seat varies greatly. Smart buyers want to pay the least amount of money to buy a plane ticket, but it is very difficult to decide when to buy a plane ticket.

Online travel companies, such as Priceline, offer low fare alert services to help customers purchase airline tickets. When customers are interested in flights, they will be notified by email if the price have reached the one specified in advance. However, since these services are based on what has already happened, there is a disadvantage that if the prices continue to rise, they may miss the right time to buy an appropriate ticket.

To help customers buy airline tickets, I analyzed the data affecting a ticket price and used data mining techniques to find patterns. Based on the prediction that the airplane ticket will rise or fall tomorrow, I can assist customers to make the decision whether it is good to buy an airplane ticket now (Buy) or better to wait (Wait).

Data collection and processing

Data collection

The data related to airplane ticket fares were directly collected from KAYAK and Priceline. These data are collected from March 1, 2008 to May 28, 2008. The fare of airplane is so fluctuating that the fare is updated about 4 ~ 7 times a day, so the lowest fare among these prices is used. A summary of the data collected in relation to the airfare is as follows.

Data on airplane fares and the factors affecting ticket prices (e.g., season, week, time, and the number of stops) are collected. I also collected external factors (e.g., oil prices) that can affect ticket prices.

Dependent variables and Independent variables

The following variables were collected for a total of 253 instances.

Dependent Variables (DVs)

• buy (b) or wait (w)
  • Buy: Today price < Next day price, Wait: Today price >= Next day price
• ticket_price
  • Lowest price on the search date

Independent Variables (IVs)

Missing value processing

In addition, the price of an airplane ticket is influenced by oil price. The oil prices from March 1, 2008 to May 28, 2008 were obtained from the iPath S&P GSCI Crude Oil Tot Ret Idx ETN (OILNF) in the finance section in the US Yahoo site. Oil prices are not traded on weekends and holidays, there are missing values. To handle this, the data for these days was filled at the prices traded the day before. That is, May 25, 2008 is the Memorial Day in the US, so there is no oil price data even though it is Monday, so it is filled with the transaction price on May 23, 2008.

Data reduction: factor analysis

Factor analysis was performed to reduce the variables and three variables of stop, trip_leave_time, and quarter were removed. First, the stop variable is eliminated because the the number of stops does not affect the price significantly since the collected data includes not only direct flights but also the multiple airline within a one airline. For the trip_leave_time was collected for the assumption that crowded use of daytime airplanes so the airfare is expected to be more expensive at day time than the one in the night; however, it was removed because it did not have a significant correlation with the price. The quarter was collected with the expectation that the company would put the airplane ticket at the end of the quarter at a cheap price to increase the total revenue; but it was removed because it did not show any correlation.

We did not perform the selection using the principal component analysis, because many variables are not taken into consideration, and the remaining variables are considered to be meaningful.

Final data

airfare.csv

Analysis

Scatter plot diagram

The price of airplane tends to increase as the travel date approaches.
When the number of days remaining until the date of travel is the same, Thursday and Sunday start at a lower price, while Friday and Saturday start at a higher price.

To investigate the factors affecting the airplane ticket prices, I plotted scatter plot diagrams with IVs on the X axis and DV on the Y axis.

Scatter plot diagram sorted by trip_leave_week (X-axis: remain_days, Y-axis: ticket_price)
What is noteworthy is that, even if the number of days remaining until the date of travel (remain_days) is the same, the price is different according to the day of departure (trip_leave_week). For example, while 80 days remain equally, Thursday (purple) and Sunday (yellow) start at a lower price, while Friday (blue) and Saturday (green) start at a higher price.

Scatter plot diagram sorted by trip_leave_date (X-axis: remain_days, Y-axis: ticket_price)
Likewise, even if the number of days remaining until the date of travel (remain_days) is the same, the price is different according to the date of departure (trip_leave_date). For example, while 80 days remain equally, 05/29/2008 (blue) and 06/01/2008 (green) start at a lower price, while 07/04/2008 (yellow) and 08/02/2008 (purple) start at a higher price. These differences are due to peak season and non-peak season. July 4 is the Independence Day of the United States and August 2 is the peak season for the summer vacation season.

Decision Tree

K-means clustering
We use K-means clustering to distinguish the data set and find the characteristics for each cluster. The size of K starts from 2 and clustering is repeated until the lift value is no longer increased. As a result of performing the cluster from K = 2 to K = 8, it stopped at K = 8 because the lift value is no longer improved.

• Metrics

- precision = a/(a+c)  
- recall = a/(a+b)  
- lift = a/(a+c) / (a+b)/(a+b+c+d)

• Data set

• cluster K = 2

• cluster K = 3

• cluster K = 4

• cluster K = 5

• cluster K = 6

• cluster K = 7

• cluster K = 8

CRT (Classification and Regression Tree)

The decision tree shows that the major factors affecting airline ticket prices are oil_price and remain_days.
From this model, we can determine you may buy the ticket today if oil_price > 64.62 per barrel, remain_days > 50, and yesterday_oil_price < 69.57.

• CRT tree based on the clusters of K = 1, 3, 5

• Model summary
  

• Validation (10-Fold Cross Validation)
  

Summary

From the result, I notice that the price of airplane tends to increase as the travel date approaches. When the number of days remaining until the date of travel is the same, Thursday and Sunday start at a lower price, while Friday and Saturday start at a higher price.

The prediction result that the tomorrow airplane price will rise or fall compared to the today price can be useful to customers who want to purchase airplane tickets. As mentioned above, it is difficult to predict the timing of the purchase of the optimal airline ticket because the airline pricing is very complicated due to many factors. Much more of data for longer period needs to be collected for building a more accurate prediction model.