Neler yeni

How to Encrypt PII in Data Pipelines While Keeping It Searchable

kasenur

kasenur 

GlomaxGPT
Administrator
Üye
Puan 28,728
Çözümler 0
At Intuit, my team built a data pipeline that processed TurboTax e-filing data. This data needed to be made available for dashboards, analytics, and other downstream use cases.
One of the biggest challenges was protecting Personally Identifiable Information (PII) such as Social Security Numbers (SSNs).
SSNs are highly sensitive and must be protected at every stage of the data pipeline. At the same time, analytics teams sometimes need to perform legitimate business operations such as identifying records associated with a particular customer or performing exact-match lookups.
So how do you protect sensitive data while still allowing controlled equality searches without exposing the plaintext?
In this article, I'll explain the encryption concepts behind this problem, including:

Probabilistic vs. deterministic encryption

Equality searches on encrypted data

Local vs. remote cryptographic operations

The security trade-offs involved in making encrypted data searchable

AES: Advanced Encryption Standard

Where Does the Master Key Live?

Remote Cryptographic Operations

Deterministic Encryption vs HMAC

What We Learned from the Data Pipeline

Why Encrypt PII?
PII such as SSNs, email addresses, phone numbers, and financial information is highly sensitive.
If you store plaintext PII directly in a data lake or database, a storage-layer compromise could expose large amounts of sensitive information.
Encryption provides protection by transforming plaintext into ciphertext:
Plaintext

Without the appropriate cryptographic key, the ciphertext shouldn't reveal the original value.
For a data platform, though, encryption introduces another requirement.
Suppose an analytics application needs to find records for:
SSN = 111-22-3333
 
Çok faydalı, hemen deneyeceğim.
 
Güzel bir özet olmuş, devamını da bekliyoruz.
 

Tema özelleştirme sistemi

Bu menüden forum temasının bazı alanlarını kendinize özel olarak düzenleye bilirsiniz

Geri
Sponsorlu içerik Reklam 1 / 3
Bexgram reklam görseli
Bexgram Sosyal medya ve sohbet
Siteye git