fix(backend): prevent duplicate ingestion of events

[detj]

Summary

Double or multiple ingestion may occur when multiple event requests with the same id appears at the same time. This leads to multiple duplicate events and hence unexpected downstream results.

Revamp the ingestion pipeline to catch conditions for double/multiple ingestion and handle them gracefully.

Tasks

• Add postgres migration to add status column in event_reqs table
• Prevent duplicate ingestion of events by keeping track of a status column in event_reqs table
• Send Retry-After response header if another identical request is being processed
• Rollback gracefully to cleanup event requests in pending state if request fails at any point
• Use defer to end all OTel spans in ingestion
• 📚 Update SDK API docs

Reproduction

1. Add artificial delay of 2 seconds during ingestion
2. Ingest using sessionator with --clean flag
3. Ingest using sessionator in another window at the same time
4. Multiple duplicate events get ingested or some requests fail with 500 Internal Server Error

Proposed Fix

1. At ingestion start first check if another request with the same id is in progress
2. If yes, reply with a 429 Too Many Requests status along with a Retry-After: 60 response header
3. If no, persist event request to db with status as pending
   • If a primary key violation occurs, this means another request was started in between, reply with a 429 Too Many Requests status along with a Retry-After: 60 response header
4. Start a db transaction
5. Ingest events as usual
6. Commit the transaction with the status of event request in db as done
7. If step 6 fails, rollback by removing the pending event request from step 3

Architecture

Before

---
title: Double Ingestion
---
flowchart TB
    req1[req1, id: 100]
    req2[req2, id: 100]
    req1d{ req1 committed? }
    req2d{ req2 committed? }
    ingest(((double ingestion occurs)))
    res[Send response]

    req1 -->|occurs at same time| req1d
    req2 -->|occurs at same time| req2d
    req1d -->|No| ingest
    req2d -->|No| ingest
    ingest -->|202 Accepted| res

Loading

After

---
title: No Double Ingestion
---
flowchart TB
    req1[req1, id: 100]
    req2[req2, id: 100]
    req1d{ req1 committed? }
    req2d{ req2 committed? }
    ingest[ingest events]
    check{identical req in progress?}
    res[Send response]
    retry[retry after sometime]

    req1 -->|occurs at same time| req1d
    req2 -->|occurs at same time| req2d
    req1d -->|No| check
    req2d -->|No| check
    check -->|Yes| retry
    check -->|No| ingest
    ingest -->|202 Accepted| res
    retry -->|429 Too Many Requests| res

Loading

See also

• fixes Prevent duplicate ingestion on simultaneous identical event requests #1277

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
measure-dashboard	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 12, 2024 9:53am