Managed Syncs allows you to configure Supaglue to sync data from your customers' third-party providers into a destination database hosted in either your cloud, or Supaglue's cloud.
It's the fastest way to read data into your application and allows you to offload third-party provider authentication, APIs, rate limits, retries, and more to Supaglue.
How it works
- Use the Management Portal to create a sync configuration (SyncConfig) that defines "what" and "how" to sync from a third-party Provider to a Destination.
- Your customer connects via our Managed Authentication feature.
- Supaglue starts fetching data from your customers’ third-party Providers and landing them in a Destination database (e.g. Postgres), that you can then query from your application.
A Provider is a third-party SaaS tool that Supaglue can connect to, to sync data.
Use the Connectors -> Providers page in the Management Portal to configure your provider. For providers who support OAuth-based authentication, you'll be asked to provide your OAuth app credentials.
- Management Portal
"oauth_scopes": ["id", "api", "refresh_token"],
Use the Connectors --> Destinations page in the Management Portal to set your destination credentials:
- Management Portal
"name": "My Postgres Destination",
Before syncing data, Supaglue generates the destination tables and columns in your database if they do not already exist.
See data models for more detail on destination schemas.
A SyncConfig ties a Provider (source) to a Destination. It defines what objects to sync and how to sync them (frequency, strategy, etc).
- Management Portal
"strategy": "full then incremental",
The SyncConfig above defines references a Salesforce Provider config and a Postgres Destination config that does the following:
Supaglue offers two sync strategies that you can configure in your sync configuration:
full only: Every sync fetches all records from the Provider.
full then incremental: The initial sync performs a full sync, then incremental after using a persistent high watermark.
full_sync_every_n_incrementals: Setting this configuration allows you to perform incremental syncs and a full refresh after every n incremental syncs. This strategy is helpful to ensure the eventual consistency of records deleted on the provider side that some incremental syncs do not support.
full then incremental is used where we can, but for specific objects and Providers, we may only be able to offer
full only sync strategies. We list these in the Providers documentation.
The quickest way to query data is to query the raw JSONB data that Supaglue lands: this is well-suited if you can easily do transforms upon reading the raw data in your application using code. You may optionally add indexes to help query performance.
Logical SQL view
You can create Postgres views if you need to rename fields or run transformations easily expressed using SQL.
Postgres Generated Columns
Building on Pattern 2, if your SQL transformations are expensive to run at query time, you can use Postgres Generated Columns to speed up queries by pushing the transformation work to be maintained at write time.
If you have well-structured tables that you wish to write Supaglue-synced data into, you can paginate over the Supaglue-synced tables, run your transformations, and write the transformed data into your tables.
Read more about transformation pipelines and patterns in our transformation tutorial section.
You can decide where to host the database that Supaglue syncs to:
- Self-host your database: Supaglue writes raw data directly into your database.
- Use a Supaglue-hosted database + API: Supaglue writes to a Supaglue-hosted database and provides an API to paginate, transform, and write data into your database schema.
Option 1: Your database
Supaglue writes raw data directly into your database.
Option 2: Supaglue-hosted database + Data Listing API
Supaglue writes raw data into a Supaglue-hosted database and provides a convenient API to paginate over records, which is helpful if you don't have an existing transformation pipeline.
Choosing between the destination options
There are several differences between the two methods of hosting Destinations:
|Dimension||Your application database||Supaglue-hosted database + Data Listing API|
|Transformations||Ideal if you have existing transformation pipelines||Ideal if you don't have an existing transformation pipeline|
|Compliance (data residency)||Your customer data is never stored in Supaglue at-rest||Your customer data is stored in Supaglue at-rest|
|Security||Your database needs to be accessible from the public internet||Your final target database can be behind a VPC|
|Operational||You need to ensure your database is up||Supaglue will ensure the uptime and sync reliability|
|Performance/Load||Supaglue determines how fast to sync to your database||You want to control how fast to sync to your database|