Benchmark Data

The Collaborative Research Cycle uses the Diverse Communities Data Excerpts serve as the target data for this program. All deidentification techniques in our directory have been run on this input data, and the resulting examples of deidentified data are available in the archive below.

The Diverse Community Data Excerpts includes three benchmark datasets – the Massachusetts data is from north of Boston, the Texas data is from near Dallas, and the National data is a collection of very diverse communities from around the nation. The data is derived from the 2019 American Community Survey; the 24 features in the complete scheme were chosen because they capture many of the complexities from real-world data, while still being small, and simple enough to make more formal analysis feasible. The data folder includes lovely postcard documentation about the communities and a JSON data dictionary to make it easy to configure your privacy technique. The usage guidance section in the readme has helpful configuration hints (watch out for 'N').

Additionally data from 2018 has been provided as a control; this may be useful for configuring differentially private algorithms or calibrating privacy metrics. The 2018 data covers the same schema (features and geography) but does not share any individuals with the 2019 data.

Deidentified Data Archives

Download the Research Acceleration Bundle and explore! This archive of deidentified data samples and evaluation metric results provides a broad, representative sample of data deidentification as a whole.

The target data for this project are the NIST Diverse Communities Excerpts , curated data drawn from the American Community Survey. The archive is comprised of deidentified versions of the Excerpts data as generated by a wide variety of deidentification algorithms, libraries and privacy definitions. Check out our Algorithm Summary Table for a high level glimpse of the current archive contents.

The bundle also includes tools and reports designed to support meta-research which we hope will lead to a more foundational understanding of the mechanics of privacy, utility and equity on diverse populations. Scroll down this page for a tour of the exploration tools we provide along with the bundle.

• Link to the Research Acceleration Bundle respository and documentation.
• Direct download link for all deidentified data and individual reports (698 MB).
• Direct download link for all meta-reports comparing techniques with discussion (669 MB).

Deidentification Algorithm Summary Table

This table provides a very high level summary of the deidentification algorithms in our archive. Unique Exact Match (UEM) is a simple privacy metric that counts the percentage of singleton records in the target that are also present in the deidentified data; these uniquely identifiable individuals leaked through the deidentification process. The Subsample Equivalent (SsE) utility metric uses an analogy between deidentification error and sampling error to communicate utility; a score of 5% indicates the edit distance between the target and deidentified data distributions is similar to the sampling error induced by randomly discarding 95% of the data. Edit distance is based on the k-marginal metric for sparse distributions.

Note that this isn't a leaderboard--- you can select any column in the dropdown menu and reorder the table according to that column. Algorithms with high utility (high SsE) may have a lot of privacy leakage (high UEM), algorithms with low privacy leakage (low UEM) may have poor utility (low SsE). Algorithms that have only been run on small subsets of the schema may perform differently on larger feature spaces (Avg Feat. Space Size). And, in general, SsE and UEM are very simple, reductive metrics. If you're curious about a deidentification method, we recommend checking out its full evaluation results in the metareports archive

Select Column for Sorting

Library	Algorithm	Team	# Entries	# Feat. sets	Avg. Feat. Space Size	ε values	Utility: SSE	Privacy Leak: UEM
rsynthpop	catall	Rsynthpop-categorical	2	1	2e+08	100.0	65.0	81.33
rsynthpop	ipf_NonDP	Rsynthpop-categorical	1	1	3e+08		50.0	15.82
rsynthpop	catall_NonDP	Rsynthpop-categorical	1	1	2e+08		50.0	63.37
tumult	DPHist	CRC	2	2	6e+07	10.0	45.5	99.91
subsample	subsample_40pcnt	CRC	15	5	4e+25		40.67	39.93
rsynthpop	cart	CRC	12	4	3e+20		40.0	16.14
UTDallas-AIFairness	smote	UT Dallas DSPL	9	1	2e+26		40.0	17.51
sdcmicro	pram	CRC	12	3	1e+11		38.33	56.27
smartnoise-synth	aim	CRC	16	6	4e+25	10.0	34.69	10.38
Anonos Data Embassy SDK	Anonos Data Embassy SDK	Anonos	3	1	2e+26		30.0	0.01
MostlyAI SD	MostlyAI SD	MOSTLY AI	6	1	2e+26		30.0	0.01
aindo-synth	aindo-synth	Aindo	3	1	2e+26		30.0	0.01
smartnoise-synth	aim	CRC	16	6	4e+25	5.0	28.12	9.77
rsynthpop	cart	CBS-NL	3	1	2e+08		21.67	28.6
smartnoise-synth	mst	CRC	16	6	2e+19	10.0	19.69	7.89
rsynthpop	ipf	CRC	3	1	2e+08	100.0	18.33	16.97
Genetic SD	Genetic SD	DataEvolution	10	2	9e+25	10.0	17.5	0.18
rsynthpop	ipf	CRC	3	1	2e+08	10.0	16.67	14.29
smartnoise-synth	mst	CRC	18	6	4e+25	5.0	14.44	7.42
smartnoise-synth	aim	CRC	16	6	4e+25	1.0	12.19	7.22
ydata-sdk	YData Fabric Synthesizers	YData	33	4	1e+26		11.85	9.7
LostInTheNoise	MWEM+PGM	LostInTheNoise	1	1	5e+26	1.0	10.0	0.0
smartnoise-synth	mst	CRC	16	6	2e+19	1.0	10.0	5.81
synthcity	bayesian_network	CRC	12	4	6e+25		7.17	17.86
Genetic SD	Genetic SD	DataEvolution	9	2	9e+25	1.0	5.56	0.04
ydata-synthetic	ctgan	DCAI Community	1	1	6e+14		5.0	0.33
subsample	subsample_5pcnt	CRC	4	4	1e+26		5.0	4.97
rsynthpop	ipf	Rsynthpop-categorical	1	1	2e+08	2.0	5.0	10.68
Sarus SDG	Sarus SDG	Sarus	1	1	2e+08	10.0	5.0	13.99
synthcity	privbayes	CRC	9	3	1e+11	1.0	4.89	5.17
sdv	ctgan	CBS-NL	6	1	2e+26		4.33	0.0
smartnoise-synth	mwem	CRC	5	5	2e+11	10.0	4.2	3.52
synthcity	privbayes	CRC	9	3	1e+11	10.0	3.78	4.32
smartnoise-synth	pacsynth	CRC	9	3	1e+11	10.0	3.44	8.9
sdv	tvae	CRC	13	4	6e+25		3.15	5.76
sdv	fastml	CRC	4	2	9e+24		3.0	1.15
synthcity	tvae	CRC	12	4	6e+25		3.0	4.69
smartnoise-synth	patectgan	CRC	12	4	2e+19	10.0	3.0	5.66
sdcmicro	kanonymity	CRC	21	3	1e+11		2.67	23.41
rsynthpop	ipf	CRC	3	1	2e+08	2.0	2.33	3.05
smartnoise-synth	patectgan	CRC	18	6	4e+25	5.0	2.33	4.49
smartnoise-synth	pacsynth	CRC	15	5	2e+11	5.0	2.13	4.32
synthcity	pategan	CRC	12	4	6e+25	10.0	2.0	2.59
synthcity	adsgan	CCAIM	1	1	5e+26		1.0	0.0
sdv	copula-gan	CRC	1	1	3e+24		1.0	0.0
sdv	ctgan	CRC	6	1	5e+26		1.0	0.0
synthcity	dpgan	CCAIM	1	1	5e+26	1.0	1.0	0.0
sdv	gaussian-copula	Blizzard Wizard	2	1	5e+11		1.0	0.0
synthcity	pategan	CCAIM	1	1	5e+26	1.0	1.0	0.0
rsynthpop	catall	Rsynthpop-categorical	2	1	2e+08	1.0	1.0	0.01
smartnoise-synth	patectgan	CRC	11	4	2e+19	1.0	1.0	0.02
smartnoise-synth	pacsynth	CRC	4	2	1e+08	1.0	1.0	0.38
sdv	gaussian-copula	CommunityData	1	1	2e+08		1.0	0.52
subsample	subsample_1pcnt	CRC	4	4	1e+26		1.0	0.98
synthcity	adsgan	CRC	12	4	6e+25		1.0	1.54
rsynthpop	ipf	CRC	1	1	2e+08	1.0	1.0	1.72
synthcity	pategan	CRC	12	4	6e+25	1.0	1.0	1.99
rsynthpop	ipf	Rsynthpop-categorical	1	1	2e+08	1.0	1.0	3.53
rsynthpop	catall	Rsynthpop-categorical	2	1	2e+08	10.0	1.0	60.39
tumult	DPHist	CRC	1	1	1e+06	1.0	1.0	74.47
tumult	DPHist	CRC	1	1	1e+06	2.0	1.0	88.39
tumult	DPHist	CRC	1	1	1e+06	4.0	1.0	98.05

---

Tutorial notebooks

Included in the version 1.1 of the CRC Research Acceleration Bundle, we provide a notebooks directory containing a complete set of tools to make the archive widely accessible for programmatic navigation and analysis.

Notebook Utility List (libs)

The notebooks folder includes a library of utilities to assist with navigating the deidentified data archive (Data and Metrics Bundle):

• Index.csv file with metadata (library, algorithm type, privacy type, research paper doi, parameter settings, etc) on every technique and deidentified data sample in the archive.
• Utility to provide easy access to the index.csv file, including help function with metadata definitions.
• Utility to assist navigation through metrics available in report.json files. Utility to assist navigation through archive file hierarchy.
• Utility to easily generate bar charts and scatterplots with configurable colored highlighting by deid technique metadata. Utility to make human-readable display labels for deid samples
• Utility to configure and display collections of metric visualizations from the SDNist reports

For newcomers to python/pandas:

Welcome! We also teach everything you need to know about pandas dataframes:

• Reading data from csv file, creating new data frames
• Displaying excerpts of data frames
• Adding new columns, updating and operating on columns
• Filtering rows, iterating over rows
• Sorting rows based on selected columns

Introduction Tutorial:

We teach all the basics for performing meta-analysis on the deidentified data archive:

1. Setup notebook.
2. Load deid datasets index file (index.csv).
3. Select specific deid. datasets from the index dataframe.
4. Working with the deidentified data csv files.
5. Working with the target data csv files.
6. Compare target and deid datasets.
7. Use index.csv to highlight plots by algorithm properties.
8. Access SDNIST evaluation reports.
9. Show relationship between two evaluation metrics.
10. Identify specific data samples of interest.
11. Show images from SDNist evaluation reports.
12. Get evaluation metrics for specific samples of interest.

Quickstart Example Jupyter Notebooks:

For folks more familiar with pandas and jupyter notebooks, we include quickstart example notebooks to demonstrate common tasks.

Example Notebook 1: Analyzing k-marginal score of deidentified datasets.

We demonstrate using a notebook to collect utility scores from the report.json files for all of the deidentified data submissions in the archive. We use the CRC plotting utility to display these scores as bar plots based on different algorithm properties.

Notebook Link

Example Notebook 2: Imposter plot (propensity scores and inconsistencies).

We show how to collect the counts of individuals in the 100% confidence propensity bin (obviously synthetic records, 'imposters') across all deidentified data submissions. We then provide a scatterplot comparing imposter count with inconsistencies count. This notebook demonstrates accessing metrics in report.json files and metric result .csv files, and using the CRC plotting utility to make highlighted scatter plots.

Notebook Link

Example Notebook 3: Race distribution metric

We show how to check deidentification impact on race distribution by directly counting individuals in the target and deidentified data csv files. We filter the index.csv metadata to print a data frame containing this score alongside relevant algorithm properties.

Notebook Link

Example Notebook 4: Privacy utility trade-off

We use the data and metrics archive to empirically explore the classic concept of the “privacy/utility trade-off curve”. This notebook collects utility and privacy metrics from the report.json files and uses the CRC plotting tool to produce scatterplots highlighted by technique properties.

Notebook Link

---

Pair-wise PCA Inspection Tool

Pairwise PCA is a relatively new visualization metric that was introduced by the IPUMS International team during the HLG-MOS Synthetic Data Test Drive. It lets us look at the high dimensional data distribution using a set of 2D scatterplots along principle component axes. The plots look at the deidentified data and target data from the same angle (ie, using axes from the target data), so we can directly see where their distributions differ from each other.

The pairwise PCA tool lets you interactively explore these plots using a GUI interface. You can install it by following the directions here .

We provide a brief usage guide below.

Step 1: Select your deidentified data

Select a deidentified data file and the appropriate target data file (MA, TX or National). Select a feature set. The tool will let you know whether your data contains the all the features for that feature set- it's fine if there are a few missing features, it will only use the ones that are present.

Step 2: Explore the plot

The pair plots on left side of the screen allows to select a single PCA component pair. The middle section of the screen shows the scatter plot of the selected pair. The single PCA component pair plot can be hovered with mouse to select a specific region of the plot enclosed in the yellow circle. The selected data points will be listed in the right section of the screen.

Step 3: Highlight the plot

In the features list you can click on a feature to highlight the plots. If you click on a specific value, it will highlight only the selected group. You can check if you'd like to highlight on all pair plots (this may run slower). You can take a screenshot using the screenshot button, and it will be stored in the PCA tool's screenshots folder for you.

Step 4: Check out the component definitions

In the component definitions tab, you'll find the definitions for all five component axes used to make the plots. You can see which features had high or low weight (influence) and whether that influence was in the positive or negative direction. Try highlighting a plot by its high weight features and see what happens! How does that compare to the low weight features?