https://nanta-data.dev/en/tags/gcp/Recent content in Gcp on nanta - Data EngineeringHugo -- gohugo.ioen© 2026 nantaFri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/bigquery-data-transfer-airflow/Fri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/bigquery-data-transfer-airflow/We built a pipeline to load S3 mart tables into BigQuery using Data Transfer Service. During PoC, DTS scheduling was managed by GCP. For production, we moved it into Airflow — creating a transfer object each batch tick and deleting it after completion. User feedback drove improvements: multi-day lookback windows, concurrent execution quota management via slot pools, and empty source path detection through GCP logging API.