Data Engineering Next Door

DISCLAIMER: complete setup of sftp server on GCP was done via Terraform except for adding secret value in secret manager. The background you might want to skip reading Unfortunately needed to an sftp server to get some files. i know very data engineer of me. and very wicked of the 3rd party data providers .. And no they did not have an API. And no they could not send in a cloud storage....

The background you might want to skip reading I had multiple api sources which had some what the same response format and given the architecture in GCP, I had a number of similar python functions that were duplicated across multiple cloud functions. We all know what a mess that is. Did the research, came across two ways to solve this A monolithic repo with common code in a separate folder and one folder per cloud function....

The background you might want to skip reading I did not go with the usual path of deploying cloud functions using terraform by zipping source code, uploading code to google cloud storage and then creating/deploying cloud functions. Not to forget the service accounts and permissions that have to be set. Why? Here are my reasons: Terraform still does not provide an inbuilt feature to check if there is a change in codebase....