Step By Step Guide: Running Zingg with Snowflake Iceberg Tables

Run Zingg on Snowflake Iceberg tables, from setup and training to matching and writing results back.

Snowflake
Step-by-step guide

Bringing together data from different sources like websites, offline stores, and CRMs often creates a common challenge: duplicate records. Without a shared way to identify customers, businesses end up with fragmented and inconsistent data, making it hard to get a clear picture of their audience. This problem has long been a hurdle for organizations, but with tools like Zingg and Apache Iceberg, it's now easier to detect and merge duplicates, ensuring clean and reliable customer data.

Why Zingg?

Zingg is an open-source entity resolution tool designed for modern data stacks. By connecting to diverse data sources such as Databricks, Snowflake, Cassandra, AWS S3, and Azure, Zingg ensures simple integration and supports multiple formats like Parquet, JSON, and CSV. Its ability to deduplicate and unify records across data silos empowers organizations to create consistent and accurate data views of customers, suppliers, and more.

Why Apache Iceberg?

Apache Iceberg is a high-performance table format designed for large-scale analytics. It simplifies data management by supporting schema evolution, time travel, and efficient querying without manual partition management. Iceberg's ability to handle petabyte-scale datasets makes it a perfect fit for modern data lakes.

The Synergy of Zingg and Iceberg

Combining Zingg’s entity resolution with Iceberg’s robust data architecture delivers major advantages:

• Unified Data Mastering

Zingg performs intelligent entity resolution on Iceberg tables, deduplicating records to provide consistent entity views.

• Scalability & Performance

Iceberg delivers fast optimized query execution, while Zingg’s ML algorithms handle efficient matching at scale.

• Easy Integration

Zingg integrates directly with Iceberg tables—no complex transformations required.

• Enhanced Data Governance

You get data lineage, schema evolution, and trustworthy master data in one workflow.

• Future-Ready Architecture

The combination ensures scalable, adaptable pipelines ready for evolving data needs.

Step-by-Step Guide: Running Zingg on Docker with Snowflake Iceberg Tables

We will use Azure for storing the Iceberg data, with Snowflake managing the Iceberg tables. You can also use other services like AWS.

Setting Up Iceberg Tables on Snowflake

1. Create Azure Blob Storage Account

  • Select Primary service > Azure Blob storage
  • Keep the rest of the settings as default
  • Note the Storage Account name – this will be used later in Snowflake setup
Azure Create a storage account review page for Blob storage, the first step in running Zingg with Snowflake Iceberg tables


2. Create Storage Container

  • Navigate to Storage Accounts > Data Storage > Container
  • Create a new container
  • Note the container name – this will also be used in the Snowflake setup
Azure storage account Containers page with the New container pane open, used for Snowflake Iceberg tables


3. Setup Snowflake

  • Create a new Warehouse or use the default COMPUTE_WH warehouse
  • Create a Database for Iceberg
  • Create External Volume
Snowflake worksheet with a CREATE OR REPLACE EXTERNAL VOLUME azure_zingg statement for Azure blob storage


4. Connect External Volume to Azure

  • Run the command: DESC VOLUME <volume_name>
  • Under the property values, find and click the URL (AZURE_CONSENT_URL) to authorize Snowflake to use Azure storage
  • Note down the AZURE_MULTI_TENANT_APP_NAME value before the underscore
Snowflake DESC EXTERNAL VOLUME result with the AZURE_CONSENT_URL and AZURE_MULTI_TENANT_APP_NAME values highlighted
  • Once authorized, use the AZURE_MULTI_TENANT_APP_NAME value (before the underscore) in the Azure storage → Add role assignment section
Azure Access Control (IAM) role assignments page with the Add button and a Storage Blob Data Owner app assignment


5. Verify Connectivity

SYSTEM$VERIFY_EXTERNAL_VOLUME('azure_zingg');

Snowflake SYSTEM$VERIFY_EXTERNAL_VOLUME result for azure_zingg returning success true


6. Create Iceberg Table

Create your Iceberg table in Snowflake with the appropriate schema.

Snowflake SQL creating the zingg_iceberg Iceberg table with string columns, azure_zingg external volume and base location


7. Load Data into the Iceberg Table

Normally this is done from some external source, but we are just loading a small dataset from test.csv.

Snowflake SQL inserting rows from CSV_DATA into the ZINGG_ICEBERG table


Setting Up Zingg

Before setting up Zingg, you need to create a config.json file. This is the primary configuration file used to set up Zingg's matching and output processing.

Configuration File Structure

1. Field Definitions (fieldDefinition)

This section defines the fields involved in the matching process, their properties, and how they are used:

  • fieldName: The name of the field being configured
  • matchType: Specifies the type of matching for the field:
    • fuzzy: Allows partial matches or minor differences (e.g., "Jon" vs. "John")
    • exact: Requires an exact match of the values
    • dont_use: Excludes the field from matching
  • fields: Refers to the field name in the dataset
  • dataType: Specifies the data type of the field, typically string for textual data
2. Output Configuration (output)

Defines how and where the results of Zingg's matching process are stored:

  • name: Logical name for the output
  • format: Specifies the output format; in this case, Snowflake's connector for Spark (net.snowflake.spark.snowflake)
  • props: Contains Snowflake connection details
3. Input Data Source (data)

Specifies the source data configuration:

  • name: Logical name for the input dataset
  • format: Input format, similar to the output format
  • props: Connection details, similar to the output configuration
Zingg config.json with field definitions, Snowflake output and input properties, numPartitions 4 and label sample size 0.5


Props Configuration

To integrate with Snowflake, specify the required JAR files in a configuration file named props.conf. This ensures the correct dependencies are loaded when Zingg interacts with Snowflake.
‍

props.conf spark.jars line listing the Snowflake JDBC and Spark Snowflake connector JAR files for Zingg


Working with Docker

Pulling the Zingg Docker Image

The Zingg image can be pulled from Docker Hub using the following command.
‍

Terminal command docker pull zingg/zingg:0.4.0

Starting the Docker Container

Mount the required directories and start the Zingg container:

Terminal command docker run mounting the zingg-0.4.0 folder and starting the Zingg container with bash


Zingg Workflow Phases

Phase 1: Find and Label

Zingg identifies sample record pairs during the Find Training Data phase, which users label as matches or non-matches. These labeled samples are then used to train Zingg's machine-learning model.
Run the Find And Label phase using the following command:

Terminal command running the Zingg findAndLabel phase with config.json and props.conf

‍Labeling Details:

The interactive learner minimizes user effort by selecting representative pairs. Once the job finishes, proceed to the next step, where pairs are manually labeled.

Zingg terminal log of the findAndLabel run reading ZINGG_ICEBERG from Snowflake and writing training data
Phase 2: Training

In the training phase, the labeled data is used to train Zingg's machine-learning models. Run the training phase with:

Terminal command running the Zingg train phase with config.json and props.conf

‍
Phase 3: Matching

The trained models are then used to predict duplicate or matching records within the dataset. Zingg generates an output table that includes both the original columns and additional metadata.
Run the matching phase using:

Terminal command running the Zingg match phase with config.json and props.conf


Understanding the Output

To store the output in an external Iceberg table, ensure the table is created in advance. The output includes:

  • Source Table Columns: Preserved in their original format
  • Z_CLUSTER: A unique identifier assigned to matching records, grouping duplicates
  • Z_MINSCORE: Indicates the minimum similarity score for any record in a cluster
  • Z_MAXSCORE: Indicates the maximum similarity score for any record in a cluster
Snowflake ZINGG_ICEBERG_OUT table preview with Z_MINSCORE, Z_MAXSCORE and Z_CLUSTER columns beside the customer fields

Final Notes

This approach combines Zingg's smart data deduplication and unification with Iceberg's ability to handle large datasets, making it easier to manage and clean up data. By using Docker, you get a portable environment to run Zingg, and Snowflake ensures your data is stored and processed securely.
‍
Explore the Zingg GitHub repository to start running entity resolution on your own Snowflake Iceberg tables.
‍
These technologies offer a modern, flexible, and easy-to-maintain setup that helps businesses manage their data more efficiently. This solution not only improves data quality but also ensures your data pipelines are ready for future needs, making it simpler to make better data-driven decisions.

Keep Going

More Guides

See all guides

Stuck On A Step?Ask The Community

The Zingg team and community answer questions on Slack. The documentation covers every phase, setting and connector.