DaaS / Products / Data-Driven Vector Index Pipeline with Notification

Data-Driven Vector Index Pipeline with Notification

Supabase database changes (e.g., new document embeddings inserted) trigger Edge Functions that route events through EventBridge to invoke ECS Cloud Assistant commands, which batch-manage vector data and indexes in OSS (insert, rebuild, similarity search), then email a processing report with index statistics via Resend.

Products involved

Scenario

When a Supabase table receives new document embeddings or updates, developers need an automated pipeline to sync these vectors into Alibaba Cloud OSS for high-performance similarity search. This workflow triggers ECS-based batch processing via EventBridge and delivers a completion report with index metrics through Resend.

Integration steps

  1. Configure Supabase Edge Function: Deploy supabase/functions/sync-vectors/index.ts to listen for INSERT events on your embeddings table. Format the payload as {"action": "sync", "vectors": [...]} and POST it to the EventBridge PutEvents endpoint.
  2. Set Up EventBridge Rule: Create a rule matching {"source": ["supabase.edge"]}. Route to an ECS Cloud Assistant target using the RunCommand API. Set CommandType=RunShellScript and pass the payload via Parameters.
  3. Execute OSS Index Creation: On the target ECS instance, run a script that calls the OSS Vector REST API to initialize the index:
  4. ``bash curl -X POST "https://<bucket>.oss-<region>.aliyuncs.com/?vector&action=CreateVectorIndex" \ -H "x-oss-vector-dimension: 1536" \ -H "x-oss-vector-distance-metric: Cosine" \ -H "x-oss-vector-format: float32" ``

  5. Batch Insert & Validate: Pipe new embeddings to OSS using the InsertVectors action, ensuring each record includes a primary key. Immediately run a validation query:
  6. ``bash curl -X POST "https://<bucket>.oss-<region>.aliyuncs.com/?vector&action=QueryVectors" \ -d '{"topK": 5, "vector": [0.1, 0.2, ...]}' ``

  7. Send Report via Resend: Parse the TotalVectors and IndexSize from the OSS response. Trigger a Resend API call:
  8. ``bash curl -X POST https://api.resend.com/emails \ -H "Authorization: Bearer $RESEND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"from":"[email protected]","to":["[email protected]"],"subject":"Vector Sync Complete","html":"<p>Inserted $COUNT vectors. Index size: $SIZE MB.</p>"}' ``

Architecture

Supabase acts as the event source, pushing database changes to an Edge Function. The function publishes structured events to Alibaba Cloud EventBridge, which routes them to an ECS Cloud Assistant command. The ECS instance executes a shell script that interacts directly with the OSS Vector API to manage indexes and run similarity queries. Finally, the script calls the Resend API to deliver a formatted processing report to stakeholders.

Prerequisites

Common pitfalls

Typical questions

FAQ

Q: How does the system automatically rebuild OSS vector indexes when Supabase data changes and send a notification? A: The pipeline automatically rebuilds vector indexes in OSS whenever Supabase database changes occur. These events are routed through EventBridge to trigger ECS batch jobs, after which a processing report with index statistics is emailed via Resend.