Sync Schedules

The ingestion worker keeps your Delta Lake in sync with your Supabase project by running periodic full-table syncs.

#Default Schedule

When you first register a Supabase connector, it defaults to syncing every 300 seconds (5 minutes). The ingestion worker checks each connector's last sync time every 30 seconds and triggers a sync when current_time - last_sync_time >= sync_interval.

#How Sync Works

1. The worker's poll loop (30-second cycle) checks every enabled connector
2. When a connector is due, a sync task is dispatched:
   • A per-tenant lock ensures only one sync runs per tenant at a time
   • Multiple tenants sync concurrently
3. The connector fetches all rows from the source table via paginated HTTP requests (1000 rows at a time)
4. The entire table is written to Delta Lake in overwrite mode (the previous data is replaced atomically)

Current limitation: Syncs are full-table overwrites. Change Data Capture (CDC) for incremental syncs is on the roadmap.

#Configuring Sync Frequency

#Via API

When creating a connector, set sync_interval_seconds:

POST /api/data-platform/connectors
{
  "name": "My Supabase",
  "type": "supabase",
  "config": {
    "supabase_url": "https://abc123.supabase.co",
    "service_role_key": "<key>",
    "sync_interval_seconds": 900
  }
}

To update an existing connector's sync interval:

PATCH /api/data-platform/connectors/{connectorId}
{
  "config": {
    "sync_interval_seconds": 3600
  }
}

#Recommended intervals

| Use case | Interval | Rationale | |----------|----------|-----------| | Real-time dashboards | 60-300s | Data freshness critical | | Standard analytics | 300-900s (5-15 min) | Good balance of freshness and cost | | Daily reporting | 3600-86400s (1-24h) | Batch processing acceptable | | Historical data | 86400s (daily) | Data rarely changes |

#Triggering Manual Syncs

#Via the API

POST /api/data-platform/sync
Authorization: Bearer <your-jwt>
X-Tenant-Id: <your-tenant-id>

This triggers a full sync for all enabled connectors belonging to your tenant. The response returns a job ID you can poll:

{
  "jobId": "a1b2c3d4e5f6...",
  "status": "pending",
  "startedAt": "2026-06-22T10:30:00Z"
}

#Polling sync status

GET /api/data-platform/sync/{jobId}
Authorization: Bearer <your-jwt>
X-Tenant-Id: <your-tenant-id>

Response:

{
  "jobId": "a1b2c3d4e5f6...",
  "status": "running",
  "progress": 0.6,
  "startedAt": "2026-06-22T10:30:00Z"
}

Status transitions: pending → running → completed | failed

#Sync Behavior for Multiple Tables

The Supabase connector syncs one table at a time. When a connector is configured, the sync triggers a sync of all discovered tables sequentially. If a connector has 10 tables, each is synced in turn within the same sync job.

#Disabling a Connector

To pause syncs for a connector without deleting it:

PATCH /api/data-platform/connectors/{connectorId}
{
  "enabled": false
}

The connector configuration is preserved, but the scheduler skips it. Existing Delta tables remain queryable.

#Understanding Sync Latency

Total sync time depends on:

• Table size — number of rows and columns
• Row size — text columns (especially JSONB) increase payload per page
• Network latency — distance between the worker and your Supabase project
• Per-page timeout — each page has a 60-second timeout; if a page fetch takes longer, the sync fails and retries

Expected performance for typical tables:

| Rows | Columns | Approx. time | |------|---------|-------------| | 1,000 | 10 | 1-2 seconds | | 100,000 | 10 | 10-30 seconds | | 1,000,000 | 10 | 2-5 minutes | | 1,000,000 | 50+ (with JSONB) | 5-15 minutes |