Data sources flowing into a digital service, which passes data on to its consumers

Project information

  • Company: Unit8
  • Client: A global Swiss reinsurance company
  • Categories: Data Engineering, Data Governance
  • Main technologies: Python, PySpark, Palantir Foundry

Summary

On the Palantir Foundry platform of a global Swiss reinsurance company, any employee can create a "digital service" and plug it into company or external data through data connections and third-party applications. Self-service is fast, but it left nobody with a clear picture of where data came from and where it went.

Working with the client's architecture and compliance teams, I built in PySpark the datasets that give that picture, and the compliance checks that keep it accurate.

What I Built

  • A lineage view of the whole platform: one dataset showing how data flows between digital services, inside and outside Foundry, built from the configuration of every data connection and third-party application.
  • A compliance loop: owners must declare where their data comes from, and most never did. Unmapped sources are now detected daily, the owner is warned, reminded 7 days before the deadline, and loses access after 30 days.
  • Data the platform didn't expose: some metadata was only visible in Foundry's web interface. I traced the network calls behind it and turned the undocumented APIs into a transform that produces a clean dataset.

Impact

  • From hours to one query: tracing a service's data flow used to mean clicking through each connection's configuration; it is now a lookup in one dataset.
  • Every data connection and third-party application checked daily, a class of non-compliance that no existing alert covered.
  • No more manual chasing: warnings, reminders and access revocation run on their own, through the client's existing notification pipeline.