Dataform-Versioning for BigQuery
GCP Dataform is a powerful tool for managing your data infrastructure, with Bigquery, on Google Cloud Platform (GCP). With Dataform, you can easily create, test, and deploy data models and pipelines using a simple, SQL-like syntax. Usually when operating with Bigquery, we create queries and save them as views, or saved queries. Transformative queries can […]
Orchestration on the Cloud
For any data transformation pipeline, it is not a one step process. The conversion of data requires, comparison, datatype conversion, data extraction, comparison, selection and flattening of records, and these steps could be re-run against hundreds of tables in loop.
Google BigQuery – Dos and Don’ts
Moving from a standard on-premise, aka desktop based DB IDE to a cloud based DB is fun. And why not!! You have a virtual scalable super-computer at your disposable, where your prior queries crunching millions of records, which could take an hour to execute, can now be completed in mere minutes.