Data Retention

Your tenant's data is stored in an isolated Delta Lake on Cloudflare R2. This page explains how data is stored, what retention guarantees exist, and how to manage your data.

#Storage Architecture

Each tenant gets an isolated R2 prefix:

s3://xen-platform/tenants/{your-tenant-id}/
  _config.json            ← Connector configurations
  events/                 ← Delta table (events)
    _delta_log/           ← Transaction log (ACID history)
      00000000000000000001.json
      00000000000000000002.json
      ...
    part-00000-xxx.parquet
    part-00001-xxx.parquet
    ...
  users/
    _delta_log/
      00000000000000000001.json
    part-00000-xxx.parquet
  ...

#Delta Lake Format

• ACID transactions — every write creates a new entry in the _delta_log/ directory
• Time-travel — you can query historical snapshots (within the retention of the transaction log)
• Schema evolution — if your source schema changes, the sync overwrites the table with the new schema
• Parquet files — data is stored in columnar Parquet format, optimised for analytical queries

#Retention Defaults

| Aspect | Setting | |--------|---------| | Data storage | Retained indefinitely — no automatic deletion | | Delta transaction log | History accumulates with each sync (overwrite creates a new log entry) | | R2 object lifecycle | No lifecycle policy by default |

The platform currently stores data indefinitely with full version history (time-travel). This means storage will grow over time as syncs accumulate. A retention policy configuration is coming soon.

#Vacuuming Old File Versions

Delta Lake's transaction log keeps a complete history of every write. Over time, old Parquet files accumulate. To reclaim storage, you can vacuum old file versions.

Vacuum is destructive — it removes unreferenced files and you lose the ability to time-travel to versions older than the retention threshold. Use with care.

Currently vacuuming is a manual operation:

from deltalake import DeltaTable

dt = DeltaTable("s3://xen-platform/tenants/{tenant-id}/{table}/")
# Dry run — list files that would be removed
files_to_delete = dt.vacuum(dry_run=True, retention_hours=168)  # 7 days

# Actual vacuum — removes files older than 7 days
dt.vacuum(dry_run=False, retention_hours=168)

Python access required — vacuum is not yet exposed via the query API. Reach out to platform engineering to schedule a vacuum for your tenant.

#Data Deletion

#Removing Delta tables

There is no self-service API to delete individual tables yet. To request table deletion, contact platform engineering with your tenant ID and table name(s).

#Removing all tenant data

When a tenant is deprovisioned:

1. Connector configs are deleted from _config.json
2. All Delta tables under the tenant's R2 prefix are removed
3. The QueryEngine instance is closed

To request deprovisioning, contact platform engineering.

#Data Isolation Guarantees

| Concern | How it's enforced | |---------|-------------------| | Cross-tenant access | Each QueryEngine instance is bound to exactly one tenant's R2 prefix. DuckDB credentials are scoped to that prefix. The API guard rejects any request where the JWT tenant ID doesn't match the route parameter. | | Data in transit | All API calls use HTTPS. The ingestion worker communicates with Supabase over HTTPS. | | Data at rest | Delta table files are stored in R2, which encrypts data at rest. | | Credential storage | Connector credentials (service_role keys) are stored in the tenant's encrypted config file on R2. |

#Storage Monitoring

Currently there is no self-service storage monitoring. The getTableStats() API provides per-table file counts and sizes:

POST /api/data-platform/query
{
  "sql": "SELECT table_name, row_count, size_in_bytes, num_files, last_modified FROM tenant_db.get_table_stats('events')"
}

This returns metadata about the Delta table's storage footprint.

#Future Retention Features

The following features are planned:

• Configurable retention policies — set TTL per connector or table
• Automated vacuum scheduling — periodic cleanup of old Delta files
• Storage usage dashboard — see per-table and total storage
• Soft delete — mark tables for deletion with a grace period