When I built first version of the website, it was a quick and dirty project hacked together to compare prices of Apple product in different countries. I didn’t even submit it to ProductHunt because of how unfinished it looked and functioned.
Ever since then, Apple has release few iteration of iPhones, MacBooks, and iPads. At first the site scrape Apple website data well on its own, but as more and more product names changed, a lot of breaking and conflicting data happened. For example, previous generation iPad Pro is now called iPad Pro 10.5”. There’s now few more insights I gathered that I didn’t figured out at first when building the first one.
How to get the prices ⛏
Like in previous version, I rely on Structured Data (JSON+LD) that’s embedded on Apple website to get the prices. Basically, if you need to know prices of any products, you can use Google Shopping to search for the product, follow the linked shop page, and use Google Structured Data Testing Tool to see if they have info that you need on the page.
For this project, I need a page that list all model variants, and the variant page itself to get more info, like product image, full name, and price. In Apple website, the pages are:
Variant list page (this list all model variants)
Variant page (it will redirect to SEO-friendly URL)
Next, I need prices in different country. Apple made their URL structure very intuitive and consistent. You just need to add each country code to the URL to change country. As for country list that have Apple Online Store presence, there’s Choose Your Country page. You can verify every country by checking if the variant list page not returning a 404.
Figuring out tax rate & refunds 💸
I already knew tax refund is one of quickest way to get cheaper price, but I didn’t know how Apple incorporate the price or how tax rates in every country works. While chatting with a friend, she mentioned that price in Australia usually works out cheaper after tax refund, for Malaysian. This surprises me because I always thought everything was more expensive in Australia.
After scraping all the prices, and deducting taxes, prices in Australia sit near the top of the table. In Apple website, some countries have tax-inclusive, and some have tax-exclusive. By counting the price after deducting or adding the taxes, the actual price margin between countries got narrower, and it made more sense. But still, there’s few outliers, like Brazil and Turkey where it still more expensive than the rest.
Figuring out rates between countries is a manual process. Some country has efficient tax refund process, like Japan where they simply 0-rated the tax when you show your passport. But others, I might need help from the crowds to verify if tax refunds really are covered for Apple purchases.
CMS for managing dynamic data 🗂
In previous version, I use a seed file to populate every Apple lineups and models. This became tedious, as I need to edit the file by hand. This process is manual, since knowing what Apple call their new product, and what is the product page URL, can’t be automated.
I’ve been experimenting with Google Spreadsheet as CMS for few years. I even tried to make a website builder backed by Google Spreadsheet as data source before. There are few products attempting to solve the same problem. The problem with Google Spreadsheet is the API not robust enough, no way to link images, and you can’t really link between sheets, just like traditional relational database.
Here come Fieldbook and Airtable. While Fieldbook is shutting down, I left with only one choice. While it’s not perfect, I manage to make full use of the API to populate data into the “base.” It’s a godsend. I’m able to run a single command from my machine to sync the data if I make any edits in the future.
You can see all the data here: airtable.com/shrXanxII1uosNzhC.
Getting accessory prices 🖱
I thought scraping for all Apple-made accessories were impossible. They don’t really have a consistent listing page, where you can scrape every single SKU in every country. And they have client-side rendering, which is unfriendly to web scrapers. Well it turns out it was a blessing in disguise.
Client-side rendering = JSON data endpoint
You could watch for client-side requests and figure out where is the data endpoint. Once I got them, I am able to populate it all into Airtable. Every product name from the JSON is localized, so assigning them standardized name need to be done manually.
Development & deployment 🏗
I’ve been using Refile for file attachment, and Figaro for config file. Rails has been bundling ActiveStorage in Rails 5.1 introduced Credentials.yml in Rails 5.2. ActiveStorage is quite dynamic as it allows you to send MiniMagick image transformation commands straight from the view itself. This is one of the command chain for one of the thumbnail.
variant.thumbnail.variant(combine_options: {
resize: "480x480>",
extent: "480x480", // extend image to this size
background: "white", // add background to extended area
gravity: "south" // pull image to this direction in new area
})
While credentials.yml allows you to commit your encrypted credentials into the repo, and deploy them together without the need to pass them to your teammates manually. I took the opportunity to implement them in this project and able to deploy them both with little effort.
Future roadmap 🚧
As of now, there’s no third-party price source that I’m pulling from. Some of the most populated countries are still not in Apple Online Store, like India, Vietnam, the whole African continent, except from resellers and eCommerce shops. To extract and normalize data from these sources would be a big undertaking. There is no free API I found to get these data.
Another option probably to just accept user-submitted prices. While reasonable, the mechanics to verify and approve them would also be tricky.
There’s also potential partnership with concierge service to connect buyer and seller to buy Apple products in different country, especially might be beneficial to expensive countries like Brazil and Turkey.
