https://nanta-data.dev/en/tags/aws/Recent content in Aws on nanta - Data EngineeringHugo -- gohugo.ioen© 2026 nantaMon, 09 Mar 2026 00:00:00 +0000https://nanta-data.dev/en/posts/ec2-instance-architecture-comparison/Mon, 09 Mar 2026 00:00:00 +0000https://nanta-data.dev/en/posts/ec2-instance-architecture-comparison/In the 2026 cloud VM benchmarks, AMD EPYC Turin (C8a) dominated both single-threaded and multi-threaded performance. Should we migrate our Graviton (ARM) data platform infrastructure to C8a? After analyzing the vCPU vs physical core distinction, Spot price-to-core efficiency, and workload characteristics — the conclusion is that a hybrid strategy beats a full migration.https://nanta-data.dev/en/posts/emr-on-eks-lakeformation-poc/Tue, 03 Mar 2026 00:00:00 +0000https://nanta-data.dev/en/posts/emr-on-eks-lakeformation-poc/We needed to add job-level access control to EMR-on-EKS Spark jobs. Ranger was ruled out due to EMR-on-EKS’s structural limitations — no master node, no plugin installation path. We chose LakeFormation, and hit 10 issues during PoC: service label selector mismatches, FGAC blocking RDD operations/UDFs/synthetic types, cross-account Glue restrictions, and more. Here’s how we identified each cause and found workarounds.https://nanta-data.dev/en/posts/bigquery-data-transfer-airflow/Fri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/bigquery-data-transfer-airflow/We built a pipeline to load S3 mart tables into BigQuery using Data Transfer Service. During PoC, DTS scheduling was managed by GCP. For production, we moved it into Airflow — creating a transfer object each batch tick and deleting it after completion. User feedback drove improvements: multi-day lookback windows, concurrent execution quota management via slot pools, and empty source path detection through GCP logging API.https://nanta-data.dev/en/posts/eks-topology-aware-hints/Fri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/eks-topology-aware-hints/We evaluated Kubernetes Topology Aware Hints to reduce cross-AZ network costs on EKS. Hints were correctly applied to EndpointSlices, but had no actual effect. AWS Load Balancer Controller’s IP target mode bypasses kube-proxy entirely, and our primary internal workloads — Spark, Trino, Airflow — are all single-zone or stateful, meaning the traffic paths where hints get referenced simply don’t exist in our environment.https://nanta-data.dev/en/posts/kafka-rack-awareness-spark/Fri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/kafka-rack-awareness-spark/We tried to apply Kafka rack awareness to Spark jobs to reduce cross-AZ network costs. Getting the AZ information was solved easily via AWS IMDS, but Spark itself doesn’t support rack-aware Kafka partition assignment. The related Jira ticket is open but the PR was closed.https://nanta-data.dev/en/posts/s3-table-buckets-poc/Fri, 27 Feb 2026 00:00:00 +0000https://nanta-data.dev/en/posts/s3-table-buckets-poc/AWS S3 Table Buckets offer managed Iceberg tables with automatic compaction. We ran a PoC to see if they could solve our CDC table compaction problem. We validated Trino, Spark, and Kafka Connect integration, examined auto-compaction behavior, and assessed costs. The conclusion: not a fit for every table, but valuable specifically for CDC workloads with unpredictable partition-level updates.