Project information
- Company: Unit8
- Client: A global Swiss reinsurance company
- Categories: Data Engineering, Data Governance
- Main technologies: Python, PySpark, Palantir Foundry
Summary
On the Palantir Foundry platform of a global Swiss reinsurance company, any employee can create a "digital service" and plug it into company or external data through data connections and third-party applications. Self-service is fast, but it left nobody with a clear picture of where data came from and where it went.
Working with the client's architecture and compliance teams, I built in PySpark the datasets that give that picture, and the compliance checks that keep it accurate.
What I Built
- A lineage view of the whole platform: one dataset showing how data flows between digital services, inside and outside Foundry, built from the configuration of every data connection and third-party application.
- A compliance loop: owners must declare where their data comes from, and most never did. Unmapped sources are now detected daily, the owner is warned, reminded 7 days before the deadline, and loses access after 30 days.
- Data the platform didn't expose: some metadata was only visible in Foundry's web interface. I traced the network calls behind it and turned the undocumented APIs into a transform that produces a clean dataset.
Impact
- From hours to one query: tracing a service's data flow used to mean clicking through each connection's configuration; it is now a lookup in one dataset.
- Every data connection and third-party application checked daily, a class of non-compliance that no existing alert covered.
- No more manual chasing: warnings, reminders and access revocation run on their own, through the client's existing notification pipeline.