Tom Xiao

Stage 3 · Data Engineer · data platform case · public

Data on EKS

A Hadoop-era stack (Hive, Trino, Ranger, LDAP and a SQL IDE) rebuilt as Helm charts, so it deploys and is secured like any other Kubernetes service. The charts are public, and the security design was covered on the AWS Big Data Blog.

RolePlatform team member: the cut-over after a node failure, the consolidation, the test design and the monitoring plan
StatusPublic charts. The platform behind them runs in production
PrincipleOne identity, one policy, every engine

One query, two people

Every query passes the same two checks before it reaches an engine: who is this, and may they. Everything here is sample data.

query path · same SQL, two peoplesample data

analyst-aSELECT * FROM finance.balances_daily

queryLDAP identityRanger policyengine (Hive, Trino)
allowed · rows returned

analyst-bSELECT * FROM finance.balances_daily

queryLDAP identityRanger policyengine (Hive, Trino)
denied by policy · engine never reached
done · one allowed, one denied

How it is put together

One identity store

  • LDAP holds the users and groups
  • Every engine asks the same directory who is calling

One policy store

  • Ranger holds who may see which table and column
  • A rule is written once and applies to Hive and Trino alike

Charts, not hand-built servers

  • Each component is a Helm chart on Amazon EKS
  • Installing one is a command, and upgrading is a diff

Before and after the merge

In production the same stack had been running twice. Putting both on one EKS platform changed the cost and the nightly schedule. Figures are approximate.

Before

Two platforms doing the same job

Platform APlatform B

Two copies of the same stack to run, patch and pay for.

Daily ETL window~8 h
After

One platform on EKS

One platform

One identity store, one policy store, one way to deploy.

Daily ETL window~5 h
Saved
~US$400/day

The weekend an old node failed

The old platform lost a node. Because the new one already existed as charts, the switch was done in a single weekend instead of being planned for months.

An old node fails
Cut over to the EKS platform in one weekend
One platform left to run

What I took from it

  • Put the access check in one place and every engine behind it inherits it.
  • Infrastructure that is a chart can be rebuilt in a weekend. Infrastructure that is a person's memory cannot.
  • Tests and monitoring are what make a quick cut-over safe.

One identity. One policy. Every engine.

Next