To Automate Post-Sale Flights Funnel For Performing Various Tasks On User’s Behalf
Background
Due to the COVID-19 pandemic, aviation industry was one of the most badly impacted industries, thousands of flight cancellations were happening every day. Being one of the biggest travel aggregators, it demanded a lot of resources for handling such cases at MakeMyTrip. To solve this problem, we formed Post-Sale-Automation team so that we could automate tedious tasks like cancelling flights on behalf of travel agent, checking the booking status on behalf of user, performing health declaration & web-check-in followed by downloading the boarding-pass on behalf of user, buying post-booking add-on services on behalf of user and checking flight departure status.
How?
Some of the above was possible by integrating different supplier APIs which could perform web-check-ins, download boarding passes, cancel the flights, re-schedule the flights and book post-sale add-ons. But there were some suppliers who didn’t have these ready to integrate APIs, so to not lose customer experience for these suppliers, we built web-scraping framework based on the idea that we can imitate user’s or agent’s behaviour using web browser robots in order to login into airline’s website and then performing the desired tasks. We started this with Air-Asia cancellations using Selenium and once it got successful, we created web-scrapers for Go-Air, Spice-Jet, Indigo, etc which would login and then perform various tasks like signing health declaration, performing web-check-in, downloading boarding pass, cancelling & re-scheduling flights (fully and partially), booking post-sale add-ons, checking flight departure status, etc. All of these either via agency login or user login, and these services were consumable by MMT and GI both using REST!
Challenges
There were a couple of challenges that we faced like,
• Airline websites seeking verification via Captcha: We solved it by capturing a screenshot of the captcha code and then using AI to read the content of the same and then finally by submitting the decoded value to by-pass the captcha.
• Scraper failures due to website’s slow speed and intermittent errors: We solved this by putting incremental waits on the configured elements and then logging proper errors corresponding to the errors shown by the website along with retry mechanism.
• Low web-scraping success rate: In order to tackle this, we created some pre-validation apis, which would validate the PNR requests before hitting the airline website, it reduced failures by huge amount.
• High scraper latency: Web-scrapers have high latency due to various reasons including airline’s website’s responsiveness and number of steps we perform on each page, we tried to reduce the number of steps and then used incremental waits instead of blind waits. Also, we added proper web-page failure identification and intelligent retry mechanism to complete the task in the single request only.
• High Memory & CPU Utilization: Though we were using headless Chrome browser but still we had to go with dev-shm storage instead of RAM to save intermittent Chrome crashes and to reduce infra consumption for various scrapers running on the server.
• Chrome and Chrome Driver Versioning: We utilised Nexus repository to unify the Chrome version across different scrapers and to avoid accidental Chrome upgrades causing failures in production.
Conclusion
When we didn’t have any APIs to integrate, web-scraping saved us and made us much ahead in competition by enriching our customers experience. This exercise resulted in a fully automated system that could take care of most the post-sales tasks so quickly and with minimal human intervention while saving Go-MMT’s Post-sale man-hours and also making our customers happy!
Not a robot!
Sanyam Jain
