Over the last few months I have been solving a series of technical challenges that allows making BigQuery derived data visualisations public, completely within the browser, using free tools. The important thing is that your BigQuery credentials remain protected, yet you still do not need to run a server, or pay any money!

A powerful browser based literate coding and dataviz environment is provided by Observable.

My tutorial covers

  • Creating a Service Account
  • Password protecting it with with SubtleCrypto
  • Instantiating a BQ client in the browser using GAPI
  • Logging into Firebase Storage using a service account
  • Setting up Firebase Security Rules.
  • Reading and Writing to Firebase storage

Checkout the full tutorial here. It’s written in a notebook so the tutorial is executable and you can fork it to make your own BQ integration.

Originally published at https://dev.to on March 26, 2021.

I used to work at Google Cloud, I also worked on a Functions-as-service (FaaS) product (Firebase Functions). I have seen that customers kept complaining about the same things:

  • Deploy times are too long
  • Cold starts are too frequent
  • FaaS is hard to debug/mock

I then went on to work as Cloud consultant for all three major clouds, I also heard customers complain about the Cloud in general, particuarly

  • The unlimited risk of a big bill from Cloud providers.
  • The opaqueness of operation

As a consumer of digital services. I have have been outraged by

  • ISPs selling my data
  • Apps selling…

In this article, I take an in-depth look at using Camunda BPMN (Business Process Modeling Notation) to build a non-trivial internal tool for my workplace. Camunda BPMN process definitions are simultaneously a graphical and textual representation of an executable workflow. I find the technology very interesting as it works both a visual communication tool and as a low code rapid development system. It excels at long running operations and has a focus on reliability and observability. The supporting tooling that allows querying running processes both graphically and/or via REST APIs.

The tool I built here for evaluation purposes touches on…

A while ago I started writing an identity aware proxy (IAP) to secure a binary with authentication. However, what started as a minimal auth layer has grown with features. What I have come to appreciate is that the reverse proxy is a great layer to do a variety of cross-cutting concerns like auth, at-least-once delivery and adapting. Furthermore, I have found OpenResty provides amazing performance and flexibility, AND it fits the serverless paradigm almost perfectly.

Concretely I have been working on extending the IAP to ingest and reshape signals from Slack and Zapier, tunnel them through a Write Ahead Log…

A long time ago, our CEO used to have one-to-one discussions with every single Futuricean over the span of a calendar year. He did it to keep his finger on the pulse of what’s happening in the company and what people are thinking about. It was a great way to maintain an understanding of whether our autonomous organization was really sailing in the same direction and aligned behind a shared strategy. As organizations grow, we need to look for ways to scale the appropriate level of control and understanding. Our old tools are no longer effective.

That brings us to…

Seemingly every project I work on gets bogged down at some point on authentication. Why must it take so much time! Why must I solve the same problems so many times now! I found myself nodding furiously while reading Google’s BeyondCorp papers, “we should solve it in the proxy layer!”. Let’s stop wasting time with Oauth2 language clients, much like we did with TLS a while ago. Solve it well once, and reuse it for everything henceforth.

Why? Because Identity Aware Proxies are awesome for security, they allow you to take insecure binaries and wrap them with a pre-audited secure…

The goal I am working towards is a common language for designers and developers. The latest footstep is OpenAPI integration with BPMN diagrams, making it simpler to build human/service workflows by importing APIs from public specifications. Here is a snapshot of the latest demo prototype.

What is BPMN?

What you see is the Camunda Modeling programs view of a BPMN diagram. It’s a visual front end for an industry standard called Business Process Modeling Notation (BPMN 2.0). BPMN is powerful, it’s a flexible language for representing human/system interactions, with a number of implementations from different vendors (i.e. no vendor lock in). I am…

I always wanted a Minecraft server but they always seemed a bit expensive. So I created this Terraform recipe to allow my kids and neighbors to play together economically but without compromising performance. The key was using Google Cloud pre-emptible VM that shutdown automatically within 24 hours.

So I never forget to switch it off after use, which saves me a ton of money as we don’t play that often. The disk attached to the VM sticks around, so we never lose our world.

Find the Terraform recipe on Github (MIT license).

If you need help setting it up I…

Here is a recipe to get the Camunda business workflow engine up-and-running fast using Terraform and serverless hosting technologies. Full code on Github

Persistence with Cloud SQL

Camunda requires a relational database. I used Google’s managed Cloud SQL. Hardly any ops required! I put Camunda in its own database namespace, so we can reuse the SQL service for other applications. I also provisioned a unique user, so we can track Camunda’s access uniquely.

Customizing the Camunda Docker Image With Cloud Build

Camanda very helpfully supplies a basic docker image, but we need to customize it. We will typically need to do this for two reasons. First, we need to supply a…

BigQuery is an amazing tool, it is capable of summoning an army to crunch your numbers in an instant. However, this power is also a footgun, as you can easily execute very expensive queries and not notice until the end-of-the-month bill arrives. This is a professionally unacceptable situation to be in, so I developed this list of best practices to ensure BigQuery projects are delivered in a safe state.

Look at the cost before running the query

If you use the web console, the (data) cost of the query will flash up in green before you execute it. Always check this. Always be mindful of how much a…

Tom Larkworthy

Cloud consultant at Futurice GmbH. Ex-Firebase, Ex-Google Cloud.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store