Using Web Scraping Technology to Automatize Cost Data Collection for Calculating Construction Cost Indexes

Construction Cost Index (CCI) and Building Cost Index (BCI) published by Engineering News Record (ENR) are two of the very popular cost indexes used in the industry. Many state DOTs publishes their highway construction cost indexes to keep track of the current trend of cost escalation. The ENR BCI and ENR CCI are prepared by collecting cost data from agents of the 20 cities in the country. Such manual process of collecting cost data can be costly. The proposed method of price data collection, based on a computer technology called “web scraping,” can automatically collect material prices and labor wage rates from the websites of material suppliers and labor unions. The material price data and labor rate can then be used to calculate various construction cost indexes. As a proof of concept, an application has been developed that collects price data from one of the popular chain store websites that sells the construction materials in small scale. The collected data is then stored in a MySQL database for further use. The benefit of using this technology is ability to calculate the indexes as often as required. Also, it would be easy to extend the geographical coverage of the indexes based on the availability of the suppliers publishing the cost data online for that location. The limitation of the technology is that if there are changes in the structure of the website which are beyond the changes in just a layout of the website, then the system is likely to break requiring a manual fix.