![]() Lastly, I’m not a fan of web scraping tools such as Selenium or Puppeteer due to their relative “chunkiness” and speed. With this in mind, I decided to use Colly, a fast and elegant web scraping framework for Golang due to its simplicity and the great developer experience it provides. However, using Scrapy in this scenario seems like overkill to me goal was rather simple and does not require any complex features such as using a handling JavaScript rendering, middlewares, data pipelines, etc. My initial thought was to use Scrapy - a feature-rich and extensible web scraping framework with Python. Otherwise, we would need to use a JavaScript rendering (headless) browser such as Splash or Selenium to scrape the content. Instead, we would need to check the ‘Network’ tab to see if it’s making any HTTP API calls to retrieve the content data. Sidetrack for a moment - what if the site content is rendered using JavaScript? Then, we won't be able to scrape the desired data directly. What if the site is rendered using JavaScript dynamically-loaded content)Įasy for us, the Michelin Guide website content is loaded using server-side rendering. Generally speaking, there are 2 main distinctions of how content is being generated/rendered on a website: Open Chrome DevTool → Cmd/Ctrl + Shift + P → Disable JavaScript This helps me to quickly identify how content is being rendered on the website. The first step that I often take after opening up the DevTool was to immediately disable JavaScript and do a quick refresh of the page. Part of the process of selecting the right library or frameworks for web scraping was to perform DevTooling on the pages. On top of that, using a SaaS often comes with a price along with its second (often unspoken) cost - its learning curve! Developer Tools (DevTool) I prefer to build my scraper due to flexibility reasons. Octoparse) in the market that requires no code at all. Heck, there’s even a tonne of Web Scraping SaaS (e.g. Today, there is a handful of tools, frameworks, and libraries out there for web scraping or data extraction. With each page containing 20 restaurants, our scraper will be visiting about ~325 pages the last page of each category might not contain 20 restaurants. Looking at the website’s data, there should be a total of 6,502 restaurants (rows). Firstly, what is the total number of restaurants that are expected to be present in our dataset? The different Michelin Awards that we are interested in Let’s do a quick estimation of the scraper. Here’s an example of our restaurant model: // model.go On the other hand, having the restaurants’ address, longitude, and latitude are particularly useful when it comes to mapping them out on maps. Having that said, feel free to submit a PR if you’re interested! I’d be more than happy to work with you. In this scenario, I am leaving out the restaurant description ( see "MICHELIN Guide’s Point Of View”) as I don’t find them particularly useful. Award (1 to 3 MICHELIN Stars and Bib Gourmand).What are we collectingīefore starting this web-scraping project, I made sure that there are no existing APIs that provide these data at least as of the time of writing this.Īfter scanning through the main page along with a couple of restaurant detail pages, I eventually settled for the following (i.e. Hence, the data collected has to be consistent, accurate, and parsed correctly. So, what does “high-quality” mean? I want anyone to be able to use the data directly without having to perform any form of data munging. Leave a minimal footprint as possible to the website.Collect “high-quality” data directly from the official Michelin Guide website.Now that that is out of the way, let’s start! Colly is unbelievably elegant yet easy to use, I’d highly recommend you to go through the official documentation to get started. Overviewīefore we start, I just wanted to point out that this is not a complete tutorial about how to use Colly. The final dataset is available free to be downloaded here. What follows is my thought process on how I collect all restaurant details from the Michelin Guide using Golang with Colly framework. Inspired by this Reddit post, my initial intention was to collect restaurant data from the official Michelin Guide (in CSV file format) so that anyone can map Michelin Guide Restaurants from all around the world on Google My Maps (see an example). Gaining just one can change a chef's life losing one, however, can change it as well. Through the years, Michelin stars have become very prestigious due to their high standards and very strict anonymous testers. 9 min read Photo by Fabrizio Magoni / UnsplashĪt the beginning of the automobile era, Michelin, a tire company, created a travel guide, including a restaurant guide.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |