Puan
28,728
Çözümler
0
- Konum
- Freanchey
- Mesajlar
- 1,522
- Katılım
- 2 Eki 2026
- Tepkime puanı
- 215,504
- Puan
- 28,728
- Web sitesi
- glomaxgpt.com
At Intuit, my team built a data pipeline that processed TurboTax e-filing data. This data needed to be made available for dashboards, analytics, and other downstream use cases.
One of the biggest challenges was protecting Personally Identifiable Information (PII) such as Social Security Numbers (SSNs).
SSNs are highly sensitive and must be protected at every stage of the data pipeline. At the same time, analytics teams sometimes need to perform legitimate business operations such as identifying records associated with a particular customer or performing exact-match lookups.
So how do you protect sensitive data while still allowing controlled equality searches without exposing the plaintext?
In this article, I'll explain the encryption concepts behind this problem, including:
Probabilistic vs. deterministic encryption
Equality searches on encrypted data
Local vs. remote cryptographic operations
The security trade-offs involved in making encrypted data searchable
AES: Advanced Encryption Standard
Where Does the Master Key Live?
Remote Cryptographic Operations
Deterministic Encryption vs HMAC
What We Learned from the Data Pipeline
Why Encrypt PII?
PII such as SSNs, email addresses, phone numbers, and financial information is highly sensitive.
If you store plaintext PII directly in a data lake or database, a storage-layer compromise could expose large amounts of sensitive information.
Encryption provides protection by transforming plaintext into ciphertext:
Plaintext
Without the appropriate cryptographic key, the ciphertext shouldn't reveal the original value.
For a data platform, though, encryption introduces another requirement.
Suppose an analytics application needs to find records for:
SSN = 111-22-3333
One of the biggest challenges was protecting Personally Identifiable Information (PII) such as Social Security Numbers (SSNs).
SSNs are highly sensitive and must be protected at every stage of the data pipeline. At the same time, analytics teams sometimes need to perform legitimate business operations such as identifying records associated with a particular customer or performing exact-match lookups.
So how do you protect sensitive data while still allowing controlled equality searches without exposing the plaintext?
In this article, I'll explain the encryption concepts behind this problem, including:
Probabilistic vs. deterministic encryption
Equality searches on encrypted data
Local vs. remote cryptographic operations
The security trade-offs involved in making encrypted data searchable
AES: Advanced Encryption Standard
Where Does the Master Key Live?
Remote Cryptographic Operations
Deterministic Encryption vs HMAC
What We Learned from the Data Pipeline
Why Encrypt PII?
PII such as SSNs, email addresses, phone numbers, and financial information is highly sensitive.
If you store plaintext PII directly in a data lake or database, a storage-layer compromise could expose large amounts of sensitive information.
Encryption provides protection by transforming plaintext into ciphertext:
Plaintext
Without the appropriate cryptographic key, the ciphertext shouldn't reveal the original value.
For a data platform, though, encryption introduces another requirement.
Suppose an analytics application needs to find records for:
SSN = 111-22-3333