Skip to content

clickhouse sync project permissions and notifications#1263

Open
BilalG1 wants to merge 1 commit intoclickhouse-sync-session-replaysfrom
clickhouse-sync-project-permission-and-notifications
Open

clickhouse sync project permissions and notifications#1263
BilalG1 wants to merge 1 commit intoclickhouse-sync-session-replaysfrom
clickhouse-sync-project-permission-and-notifications

Conversation

@BilalG1
Copy link
Collaborator

@BilalG1 BilalG1 commented Mar 18, 2026

No description provided.

@vercel
Copy link

vercel bot commented Mar 18, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
stack-auth-hosted-components Ready Ready Preview, Comment Mar 18, 2026 4:48pm
stack-backend Ready Ready Preview, Comment Mar 18, 2026 4:48pm
stack-dashboard Ready Ready Preview, Comment Mar 18, 2026 4:48pm
stack-demo Ready Ready Preview, Comment Mar 18, 2026 4:48pm
stack-docs Ready Ready Preview, Comment Mar 18, 2026 4:48pm

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 18, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 94716b7d-e33e-404c-8973-85ff768c38a6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch clickhouse-sync-project-permission-and-notifications
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 18, 2026

Greptile Summary

This PR extends the ClickHouse / external-DB sync pipeline to cover two new entity types: project-level user permissions (ProjectUserDirectPermission) and notification preferences (UserNotificationPreference). It follows the established sync pattern used by existing tables: adding sequenceId / shouldUpdateSequenceId columns to both Postgres models, registering new sequencer CTEs, wiring withExternalDbSyncUpdate into the CRUD handlers, recording soft-deletes via DeletedRow before cascade deletes, and providing full ClickHouse table + view + row-level-security definitions alongside Postgres external-DB sync mappings.

Key changes:

  • schema.prisma + migration: adds nullable sequenceId and boolean shouldUpdateSequenceId (default true) to both tables with appropriate indexes.
  • sequencer/route.ts: two new FOR UPDATE SKIP LOCKED CTEs that assign global sequence IDs to unsynced rows.
  • external-db-sync.ts: new ExternalDbSyncTarget union variants and recordExternalDbSyncDeletion handlers; new bulk-deletion helpers recordExternalDbSyncProjectPermissionDeletionsForUser and recordExternalDbSyncNotificationPreferenceDeletionsForUser.
  • permissions.tsx / crud.tsx / notification-preference/crud.tsx: sync hooks wired into grant, revoke, and user-delete paths.
  • db-sync-mappings.ts: full sync mapping definitions for both tables.
  • clickhouse-migrations.ts: new ClickHouse base tables, views, grants, and row-isolation policies.
  • E2E tests for Postgres and ClickHouse sync of both new entity types.

Issues found:

  • In db-sync-mappings.ts, the DeletedRow union branch for notification_preferences casts notificationCategoryId to ::uuid in both internalDbFetchQuery variants, while the active-rows branch returns it without a cast and the target column is text. This is inconsistent and will throw a runtime Postgres error if any category ID is not a valid UUID.
  • run-cron-jobs.ts replaces a deliberate 0–120 s randomized Vercel cron jitter with a fixed 1 s wait and removes the explanatory comment, which may cause thundering-herd pressure on the database when multiple Vercel instances fire simultaneously.
  • waitForSyncedNotificationPreferenceDeletion is defined in test utils but never invoked, leaving the notification preference deletion sync path without test coverage.

Confidence Score: 3/5

  • Mostly safe to merge, but contains a type-cast bug that will surface as a runtime error when deleted notification preferences are synced, and an undocumented aggressive polling change.
  • The overall sync pattern is solid and mirrors prior implementations, and the migration is safe on populated tables. However, the ::uuid cast for notificationCategoryId in the deleted-row fetch branches is semantically wrong (target is text) and will cause Postgres runtime errors if any category ID is not UUID-formatted. The cron wait reduction from randomized 0–120 s to 1 s removes deliberate Vercel jitter protection without documentation. The missing deletion test also leaves a code path unverified.
  • packages/stack-shared/src/config/db-sync-mappings.ts (incorrect ::uuid cast at lines ~1965 and ~2001) and apps/backend/scripts/run-cron-jobs.ts (aggressive polling change).

Important Files Changed

Filename Overview
packages/stack-shared/src/config/db-sync-mappings.ts Adds project_permissions and notification_preferences sync mappings; contains an incorrect ::uuid cast for notificationCategoryId from DeletedRow.data in both fetch query variants, while the target column is text — inconsistent with active-row branches and risky at runtime.
apps/backend/src/lib/external-db-sync.ts Adds ExternalDbSyncTarget union cases and deletion-recording logic for ProjectUserDirectPermission and UserNotificationPreference; pattern mirrors existing implementations and looks correct.
apps/backend/src/app/api/latest/internal/external-db-sync/sequencer/route.ts Adds sequencer CTEs for ProjectUserDirectPermission and UserNotificationPreference; both correctly use FOR UPDATE SKIP LOCKED and join on appropriate keys (id for the former, composite tenancyId+id for the latter).
apps/backend/scripts/run-cron-jobs.ts Removes Vercel-specific cron jitter (0–120 s) and replaces it with a fixed 1 s wait; rationale comment was also removed, which may cause thundering-herd issues on Vercel and leaves no documentation of the design decision.
apps/backend/src/lib/permissions.tsx Adds deletion tracking before deleteMany/delete calls for ProjectUserDirectPermission; correctly fetches IDs, records deletions inside the transaction, then performs the delete — no issues found.
apps/e2e/tests/backend/endpoints/api/v1/external-db-sync-basics.test.ts Adds Postgres and ClickHouse sync tests for project permissions (including grant+revoke cycle) and notification preferences (create/update only); the notification preference deletion path is not covered by any test.
apps/e2e/tests/backend/endpoints/api/v1/external-db-sync-utils.ts Adds waitForSyncedProjectPermission, waitForSyncedProjectPermissionDeletion, waitForSyncedNotificationPreference, and waitForSyncedNotificationPreferenceDeletion helpers; the deletion helper for notification preferences is defined but never called in the test suite.

Sequence Diagram

sequenceDiagram
    participant API as API Handler
    participant PG as Postgres (internal)
    participant SEQ as Sequencer (cron)
    participant SYNC as ExternalDB Sync Worker
    participant EXT as External DB (Postgres / ClickHouse)

    note over API,EXT: Grant / Update flow
    API->>PG: upsert ProjectUserDirectPermission<br/>withExternalDbSyncUpdate → shouldUpdateSequenceId=true
    API->>PG: upsert UserNotificationPreference<br/>withExternalDbSyncUpdate → shouldUpdateSequenceId=true

    SEQ->>PG: CTE: SELECT id WHERE shouldUpdateSequenceId=TRUE FOR UPDATE SKIP LOCKED
    SEQ->>PG: UPDATE SET sequenceId=nextval(), shouldUpdateSequenceId=FALSE
    SEQ->>SYNC: enqueueExternalDbSyncBatch(tenancyIds)

    SYNC->>PG: internalDbFetchQuery (active rows UNION deleted rows)<br/>WHERE sequence_id > last_synced
    SYNC->>EXT: externalDbUpdateQuery (upsert / delete row + update _stack_sync_metadata)

    note over API,EXT: Revoke / Delete flow
    API->>PG: recordExternalDbSyncDeletion → INSERT into DeletedRow<br/>(ProjectUserDirectPermission or UserNotificationPreference)
    API->>PG: DELETE from source table (cascade or explicit)

    SEQ->>PG: CTE: SELECT id FROM DeletedRow WHERE shouldUpdateSequenceId=TRUE
    SEQ->>PG: UPDATE DeletedRow SET sequenceId=nextval()
    SEQ->>SYNC: enqueueExternalDbSyncBatch(tenancyIds)

    SYNC->>PG: internalDbFetchQuery (deleted rows branch)
    SYNC->>EXT: externalDbUpdateQuery (DELETE from target table)
Loading

Comments Outside Diff (2)

  1. apps/backend/scripts/run-cron-jobs.ts, line 180 (link)

    P2 Cron jitter removal may cause thundering herd on Vercel

    The previously randomized wait (Math.random() * 120_000 ms, 0–2 min) was explicitly added to handle Vercel's minute-granularity cron scheduling — when multiple service instances fire simultaneously, the jitter prevents all of them from hitting the database in the same instant. Replacing it with a fixed 1-second wait removes this protection and also creates a near-constant polling loop.

    If this is intentional (e.g., faster sync for the new mappings), the comment explaining the reasoning should be preserved and the design decision documented. Also note the semicolon is missing:

  2. apps/e2e/tests/backend/endpoints/api/v1/external-db-sync-utils.ts, line 880-885 (link)

    P2 waitForSyncedNotificationPreferenceDeletion is defined but never called

    This utility function is exported but not used in any test case — the NotificationPreference sync (Postgres) test does not include a deletion step. The deletion sync path (triggered both via revokeProjectPermission and recordExternalDbSyncNotificationPreferenceDeletionsForUser on user delete) is therefore untested.

    Consider adding a deletion verification step to the existing Postgres notification preference test, similar to how waitForSyncedProjectPermissionDeletion is exercised in the ProjectPermission CRUD sync (Postgres) test.

Last reviewed commit: "clickhouse sync proj..."

@BilalG1 BilalG1 requested a review from N2D4 March 18, 2026 21:41
@BilalG1 BilalG1 assigned N2D4 and unassigned BilalG1 Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants