Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package a RisingWave image that supports HDFS after each release #12723

Open
wcy-fdu opened this issue Oct 10, 2023 · 8 comments
Open

Package a RisingWave image that supports HDFS after each release #12723

wcy-fdu opened this issue Oct 10, 2023 · 8 comments

Comments

@wcy-fdu
Copy link
Contributor

wcy-fdu commented Oct 10, 2023

Currently we support HDFS as storage backend but it's off by default, thus we should package a RisingWave image that supports HDFS after each release. We need to package manually on EC2 instead of using buildkite, I have write a guidance to explain how to package it.
https://www.notion.so/risingwave-labs/Package-RisingWave-with-HDFS-6ea9e37568bd4e1882b9217efdc5981b?pvs=4

Note that libhdfs is always upwardly compatible, so we only need to package a lower version of HDFS. Let us use HDFS 2.7.3 as the default (the same with Databend).

I have package one for v1.2 here, and we need need to do this every time we release from now on.

@github-actions github-actions bot added this to the release-1.4 milestone Oct 10, 2023
@wcy-fdu wcy-fdu removed this from the release-1.4 milestone Oct 10, 2023
Copy link
Contributor

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

@sheltonsuen
Copy link

seems like the guidance to package is a internal doc, looking forward this feature or any suggestion to mannual package with 1.5.x?

@wcy-fdu
Copy link
Contributor Author

wcy-fdu commented Dec 26, 2023

Hi @sheltonsuen , indeed it's an internal doc, and we can provide a 1.5.x image that supports HDFS if you need. Could you please let us know your hdfs version?

@sheltonsuen
Copy link

thanks, It's fine, after using the 1.2 image, working as expected

@wcy-fdu
Copy link
Contributor Author

wcy-fdu commented Dec 26, 2023

You mean you are using RisingWave 1.2 on HDFS, and everything works well?
Now you want to use RisingWave 1.5.x so that requires a new RisingWave version, right?

@sheltonsuen
Copy link

sheltonsuen commented Dec 26, 2023

sorry about the confusing, I mean this image: RW_v1.2_HDFS_v2.7-x86_64

@wcy-fdu
Copy link
Contributor Author

wcy-fdu commented Dec 26, 2023

I see. We will deliver a new version later, and will let you know here~

Copy link
Contributor

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants