Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add partitioned tables support #2197

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

magnetised
Copy link
Contributor

@magnetised magnetised commented Dec 19, 2024

Allows for subscribing to partitions of a partitioned table (which you could already do) but also subscribing to the root, partitioned, table and receiving all updates across all partitions.

Fixes #2118

Major changes:

  • Add support for partitioned table types in the various inspector queries
  • Include partition hierarchy information in relation information from pg and in inspector relation info
  • Force reload of table information when receiving a new relation that is a partition
  • Add partition information to Filter so that it can correctly pass on a change on a partition table to a shape on the partition root.
  • For a shape subscribed to the partition root, changes coming in on partitions will be re-written so that their relation matches the partition root. The change keys are not being re-written - I don't think this is an issue as the keys are guaranteed to be unique and that's enough.
  • Relation messages notifying of a new partition table are forwarded onto root-partition consumers so the shape can be updated with knowledge of the new partition (this does not cause the shape to be deleted)
  • I've added a full-stack test to verify this stuff so that we're not making assumptions about pg's handling of partitions

Minor:

  • support querying the inspector based on the {schema, name} relation tuple, as well as the table name string because I needed to be able to look up table info based
  • Filter instances now have an inspector instance so they can do table info lookups to get partition information
  • The filter functions used by the dispatcher now return {filter, shape_ids} so that the filter's state can be updated
  • I've tweaked StubInspector so we can return richer info from the load_relation call
  • Added missing serialisation of a shape's replica setting

@magnetised magnetised force-pushed the 2118-support-shapes-on-partitioned-tables branch from 5e6a8f1 to 8aafed5 Compare December 19, 2024 18:17
and fix missing replica setting too.

also remove partitions from shape hash value much like we exclude the
table info
Copy link

netlify bot commented Dec 20, 2024

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 3c45c27
🔍 Latest deploy log https://app.netlify.com/sites/electric-next/deploys/67656c7e9cfe550008d143c8
😎 Deploy Preview https://deploy-preview-2197--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support shapes on partitioned tables
1 participant