Over the last few months I have been solving a series of technical challenges that allows making BigQuery derived data visualisations public, completely within the browser, using free tools. The important thing is that your BigQuery credentials remain protected, yet you still do not need to run a server, or pay any money!
A powerful browser based literate coding and dataviz environment is provided by Observable.
My tutorial covers
Checkout the full tutorial here. It’s written in a notebook so the tutorial is executable and you can fork it to make your own BQ integration.
Originally published at https://dev.to on March 26, 2021.
I used to work at Google Cloud, I also worked on a Functions-as-service (FaaS) product (Firebase Functions). I have seen that customers kept complaining about the same things:
I then went on to work as Cloud consultant for all three major clouds, I also heard customers complain about the Cloud in general, particuarly
As a consumer of digital services. I have have been outraged by
In this article, I take an in-depth look at using Camunda BPMN (Business Process Modeling Notation) to build a non-trivial internal tool for my workplace. Camunda BPMN process definitions are simultaneously a graphical and textual representation of an executable workflow. I find the technology very interesting as it works both a visual communication tool and as a low code rapid development system. It excels at long running operations and has a focus on reliability and observability. The supporting tooling that allows querying running processes both graphically and/or via REST APIs.
The tool I built here for evaluation purposes touches on…
A while ago I started writing an identity aware proxy (IAP) to secure a binary with authentication. However, what started as a minimal auth layer has grown with features. What I have come to appreciate is that the reverse proxy is a great layer to do a variety of cross-cutting concerns like auth, at-least-once delivery and adapting. Furthermore, I have found OpenResty provides amazing performance and flexibility, AND it fits the serverless paradigm almost perfectly.
Concretely I have been working on extending the IAP to ingest and reshape signals from Slack and Zapier, tunnel them through a Write Ahead Log…
A long time ago, our CEO used to have one-to-one discussions with every single Futuricean over the span of a calendar year. He did it to keep his finger on the pulse of what’s happening in the company and what people are thinking about. It was a great way to maintain an understanding of whether our autonomous organization was really sailing in the same direction and aligned behind a shared strategy. As organizations grow, we need to look for ways to scale the appropriate level of control and understanding. Our old tools are no longer effective.
That brings us to…
Seemingly every project I work on gets bogged down at some point on authentication. Why must it take so much time! Why must I solve the same problems so many times now! I found myself nodding furiously while reading Google’s BeyondCorp papers, “we should solve it in the proxy layer!”. Let’s stop wasting time with Oauth2 language clients, much like we did with TLS a while ago. Solve it well once, and reuse it for everything henceforth.
The goal I am working towards is a common language for designers and developers. The latest footstep is OpenAPI integration with BPMN diagrams, making it simpler to build human/service workflows by importing APIs from public specifications. Here is a snapshot of the latest demo prototype.
What you see is the Camunda Modeling programs view of a BPMN diagram. It’s a visual front end for an industry standard called Business Process Modeling Notation (BPMN 2.0). BPMN is powerful, it’s a flexible language for representing human/system interactions, with a number of implementations from different vendors (i.e. no vendor lock in). I am…
I always wanted a Minecraft server but they always seemed a bit expensive. So I created this Terraform recipe to allow my kids and neighbors to play together economically but without compromising performance. The key was using Google Cloud pre-emptible VM that shutdown automatically within 24 hours.
So I never forget to switch it off after use, which saves me a ton of money as we don’t play that often. The disk attached to the VM sticks around, so we never lose our world.
Find the Terraform recipe on Github (MIT license).
Camunda requires a relational database. I used Google’s managed Cloud SQL. Hardly any ops required! I put Camunda in its own database namespace, so we can reuse the SQL service for other applications. I also provisioned a unique user, so we can track Camunda’s access uniquely.
BigQuery is an amazing tool, it is capable of summoning an army to crunch your numbers in an instant. However, this power is also a footgun, as you can easily execute very expensive queries and not notice until the end-of-the-month bill arrives. This is a professionally unacceptable situation to be in, so I developed this list of best practices to ensure BigQuery projects are delivered in a safe state.
If you use the web console, the (data) cost of the query will flash up in green before you execute it. Always check this. Always be mindful of how much a…
Cloud consultant at Futurice GmbH. Ex-Firebase, Ex-Google Cloud.