-
Is there an option to restore states during restarts/recovery? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments
-
Not quite sure what youre asking for here. If you mean to say can CSP recover state specifically from kafka, then the answer is yes, you can use the start_offset option and it it to START_TIME: |
Beta Was this translation helpful? Give feedback.
-
So i quess it keep state in memory?,Some other stream processors like arroyo or Bytewax are using AWS S3 resp. sqlite to keep track of state. In the case of CSP you start from a kafka offset to build up the state again and then it automatically continue when the current time or is it a 2 step process first historical and then realtime? I never heard about CSP till i saw the youtube presentation, so i start from scratch with it, so forgive me if it sounds stupid...;-) |
Beta Was this translation helpful? Give feedback.
-
CSP doesnt (currently) have any external state keeping. At the moment the only choice is to rebuild state yourself from external sources. |
Beta Was this translation helpful? Give feedback.
-
Thanks Rob, that sounds great, kafka is our main source/sink, we makes it easy for SCP to start from any streaming server and not being dependent on external db/object storage ;-) |
Beta Was this translation helpful? Give feedback.
Not quite sure what youre asking for here. If you mean to say can CSP recover state specifically from kafka, then the answer is yes, you can use the start_offset option and it it to START_TIME:
https://github.com/Point72/csp/wiki/Input-Output-Adapters-API#api
https://github.com/Point72/csp/blob/main/csp/adapters/kafka.py#L35