Managed Syncs

Managed Syncs allows you to configure Supaglue to sync data from your customers' third-party providers into a destination database hosted in either your cloud, or Supaglue's cloud.

It's the fastest way to read data into your application and allows you to offload third-party provider authentication, APIs, rate limits, retries, and more to Supaglue.

How it works

1. Use the Management Portal to create a sync configuration (SyncConfig) that defines "what" and "how" to sync from a third-party Provider to a Destination.
2. Your customer connects via our Managed Authentication feature.
3. Supaglue starts fetching data from your customers’ third-party Providers and landing them in a Destination database (e.g. Postgres), that you can then query from your application.

Core concepts

Provider

A Provider is a third-party SaaS tool that Supaglue can connect to, to sync data.

Use the Connectors -> Providers page in the Management Portal to configure your provider. For providers who support OAuth-based authentication, you'll be asked to provide your OAuth app credentials.

Management Portal

https://app.supaglue.io/application/62605dc1-148e-4c53-a850-82e10f71ed23/connectors/providers/crm/salesforce

JSON

{
  "id": "...",
  "application_id": "...",
  "category": "crm",
  "auth_type": "oauth2",
  "name": "salesforce",
  "config": {
    "oauth": {
      "oauth_scopes": ["id", "api", "refresh_token"],
      "credentials": {
        "oauth_client_id": "3MVG9ux34Ig8G5epaZ61C6cIBMigzByvA_dRqIK173k80emeb13bs.bjU1IxUt8AOAKrZHcymU770sHlopQG_",
        "oauth_client_secret": "EE04112033912B1B62A3BE1DAD7D01903C7256A611A636008AD68434B421CD3C"
      }
    }
  }
}

Destination

A Destination is a database that Supaglue syncs third-party provider data to. You can choose one in your infrastructure or Supaglue's infrastructure.

Use the Connectors --> Destinations page in the Management Portal to set your destination credentials:

Management Portal

https://app.supaglue.io/application/62605dc1-148e-4c53-a850-82e10f71ed23/connectors/destinations/postgres

JSON

{
  "id": "...",
  "application_id": "...",
  "name": "My Postgres Destination",
  "type": "postgres",
  "config": {
    "host": "production.cluster-cdhnnutnlctj.us-west-2.rds.amazonaws.com",
    "port": 5432,
    "database": "postgres_prod_db",
    "schema": "public",
    "user": "myuser"
  }
}

info

Before syncing data, Supaglue generates the destination tables and columns in your database if they do not already exist.

See data models for more detail on destination schemas.

SyncConfig

A SyncConfig ties a Provider (source) to a Destination. It defines what objects to sync and how to sync them (frequency, strategy, etc).

Management Portal

https://app.supaglue.io/application/62605dc1-148e-4c53-a850-82e10f71ed23/syncs/sync_configs/4427eb3f-3c64-4d1e-bf3a-0646475fc71c

JSON

{
  "id": "...",
  "application_id": "...",
  "provider_id": "7f72ec07-e5c1-47fd-8cf5-e71dd13873af",
  "destination_id": "6e7baa88-84dd-4dbc-902a-14522c2984eb",
  "config": {
    "default_config": {
      "period_ms": 60000,
      "strategy": "full then incremental",
      "auto_start_on_connection": true
    },
    "standard_objects": [
      {
        "object": "Account"
      }
    ],
    "custom_objects": [
      {
        "object": "Car__c"
      }
    ]
  }
}

The SyncConfig above defines references a Salesforce Provider config and a Postgres Destination config that does the following:

• Sync Account standard object from Salesforce.
• Sync the Car__c custom object from Salesforce.

Sync strategies

Supaglue offers two sync strategies that you can configure in your sync configuration:

• full only: Every sync fetches all records from the Provider.
• full then incremental: The initial sync performs a full sync, then incremental after using a persistent high watermark.
  • full_sync_every_n_incrementals: Setting this configuration allows you to perform incremental syncs and a full refresh after every n incremental syncs. This strategy is helpful to ensure the eventual consistency of records deleted on the provider side that some incremental syncs do not support.

full then incremental is used where we can, but for specific objects and Providers, we may only be able to offer full only sync strategies. We list these in the Providers documentation.

Query patterns

Direct query

The quickest way to query data is to query the raw JSONB data that Supaglue lands: this is well-suited if you can easily do transforms upon reading the raw data in your application using code. You may optionally add indexes to help query performance.

Logical SQL view

You can create Postgres views if you need to rename fields or run transformations easily expressed using SQL.

Postgres Generated Columns

Building on Pattern 2, if your SQL transformations are expensive to run at query time, you can use Postgres Generated Columns to speed up queries by pushing the transformation work to be maintained at write time.

Transformation pipeline

If you have well-structured tables that you wish to write Supaglue-synced data into, you can paginate over the Supaglue-synced tables, run your transformations, and write the transformed data into your tables.

Read more about transformation pipelines and patterns in our transformation tutorial section.

Destination options

You can decide where to host the database that Supaglue syncs to:

1. Self-host your database: Supaglue writes raw data directly into your database.
2. Use a Supaglue-hosted database + API: Supaglue writes to a Supaglue-hosted database and provides an API to paginate, transform, and write data into your database schema.

Option 1: Your database

Supaglue writes raw data directly into your database.

Option 2: Supaglue-hosted database + Data Listing API

Supaglue writes raw data into a Supaglue-hosted database and provides a convenient API to paginate over records, which is helpful if you don't have an existing transformation pipeline.

Choosing between the destination options

There are several differences between the two methods of hosting Destinations:

Dimension	Your application database	Supaglue-hosted database + Data Listing API
Transformations	Ideal if you have existing transformation pipelines	Ideal if you don't have an existing transformation pipeline
Compliance (data residency)	Your customer data is never stored in Supaglue at-rest	Your customer data is stored in Supaglue at-rest
Security	Your database needs to be accessible from the public internet	Your final target database can be behind a VPC
Operational	You need to ensure your database is up	Supaglue will ensure the uptime and sync reliability
Performance/Load	Supaglue determines how fast to sync to your database	You want to control how fast to sync to your database