Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydoop submit: allow dist cache items from HDFS #314

Open
simleo opened this issue Jun 15, 2018 · 0 comments
Open

Pydoop submit: allow dist cache items from HDFS #314

simleo opened this issue Jun 15, 2018 · 0 comments
Assignees
Milestone

Comments

@simleo
Copy link
Member

simleo commented Jun 15, 2018

pydoop submit simplifies adding items to the DC via --upload-{file,archive}-to-cache. These items are looked for in the local fs, and are automatically uploaded to temporary HDFS locations before adding them to the DC. While this is usually what the user wants, this feature should also be able to handle items already on HDFS.

Option names can change to --cache-{files,archive} (do this before Pydoop 2.0 final), since their actions won't necessarily involve an upload anymore.

@simleo simleo added this to the 2.0 milestone Jun 15, 2018
@simleo simleo self-assigned this Jun 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant