BigQuery, Airbyte, Intercom, and another story of Google Cloud Platform cost optimization

My first story on the Google Cloud Platform cost optimization was about the hidden secrets of Looker Studio (formerly Google Data Studio). Let me tell you another story on how to find the source of soaring costs. Some new cool features for cost optimization are now available on GCP so let’s start. The support team lead reached out to me asking if we can get the content of comments from Intercom....

November 18, 2022 · 6 min · Alex Danilin

Building data pipelines in Google BigQuery with Dataform

I’ve been using Google BigQuery since its’ public release and I like where it has been going through all these years. The team behind the product is doing an amazing job and I do not remember any public feature with no apparent use case. One of the most noticeable BigQuery evolution branches for me is from simple storage and query engine to internal scheduled queries mechanism and a non-linear interconnected query logic for data manipulation later on....

October 10, 2022 · 7 min · Alex Danilin

How to Stop Wasting Money on Google BigQuery in Google Data Studio

Update. I updated this article with a new case of BigQuery cost optimization. One of my responsibilities as a product manager is tracking the influence of product on key metrics like revenue or MRR. I also prefer to share this data with my team for us to be on the same line while we are developing new features or improving something that already exists. A dashboard with a number of key metrics is good choice here....

November 26, 2021 · 6 min · Alex Danilin

How to Use Self-made Service Account Key with Expiration Date on Google Cloud Platform

One of the best practices for storing service account keys is to rotate them on a regular basis. You can do that manually but it would be much better to have a kind of mechanism that doesn’t let you miss the date of key renewal. I will use some gcloud console commands here but it is not necessary as you can do the same in Google Cloud Platform (GCP) interface....

November 23, 2021 · 3 min · Alex Danilin

How to Grant an Access for R googlesheets4 package Using Google Cloud Platform Service Account

One of my weekly tasks brought me to Google Cloud Platform instance to deploy a tiny R script as a cron job. This script makes some data transformations and writes the results to Google Sheets. I use googlesheets4 package to communicate with Google Sheets and it works frictionless on my local machine. But Ubuntu cloud instance is not a local Windows machine. I do not use interactive mode on instances and browser as well so the common OAuth authorization is not the case here....

January 29, 2020 · 2 min · Alex Danilin