Mirror mode: Mixpanel seems to keep staging tables in users' Redshift database

[v0rt3xh]

Our team started to explore mirror mode this November. To be more specific, we are testing this new feature on our AWS Redshift database.

A recent incident might reveal a potential flaw of Mirror mode on Redshift destination. The number of tables in our database kept increasing and reached AWS' limit yesterday, which impacted our data pipeline and downstream tasks.

We are aware of the fact that this Mirror mode is a relatively new feature and might still be under a beta testing framework. I haven't received responses from [email protected] nor has my colleague got advice from https://community.mixpanel.com/. This issue might cause a huge problem for other customers as well. Thus, I chose to post here.

Feel free to route me to the correct place or the right support team. I also noticed that @myronkaifung added a callout on this file. So probably your team has started to work on resolving this issue. Thanks.

FYI Attachment
This image shows the increasing amount of tables in our Redshift database. Periodic decrease: Amazon's scheduled maintenance. Most recent sharp drop: I manually deleted Mixpanel's staging tables

[myronkaifung]

Hi @v0rt3xh , thanks for raising the issue. My teammate Mikey just picked up the support ticket to work with you more directly on the issue. I linked this discussion to him so he has some additional context.