Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local env runtime variables override for local testing #690

Closed
wants to merge 5 commits into from
Closed

Add local env runtime variables override for local testing #690

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d40f160
add local teste env config override
Feb 23, 2024
41f5f25
add env as a passing parameter
Feb 27, 2024
b3dfac1
add comment
Feb 27, 2024
d4bb849
remove unused change
Feb 27, 2024
7ac3cd1
update comment
Feb 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions api/py/ai/chronon/repo/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,7 @@ def set_runtime_env(args):
- Environment variables derived from args (like app_name)
- conf.metaData.modeToEnvMap for the mode (set on config)
- team environment per context and mode set on teams.json
- production team environment per mode set on teams.json
- default team environment per context and mode set on teams.json
- Common Environment set in teams.json
"""
Expand All @@ -245,6 +246,7 @@ def set_runtime_env(args):
"conf_env": {},
"default_env": {},
"team_env": {},
"production_team_env ": {},
"cli_args": {},
}
conf_type = None
Expand All @@ -262,7 +264,7 @@ def set_runtime_env(args):
)
if args.conf and effective_mode:
try:
context, conf_type, team, _ = args.conf.split("/")[-4:]
_, conf_type, team, _ = args.conf.split("/")[-4:]
except Exception as e:
logging.error(
"Invalid conf path: {}, please ensure to supply the relative path to zipline/ folder".format(
Expand All @@ -272,6 +274,9 @@ def set_runtime_env(args):
raise e
if not team:
team = "default"
# context is the environment in which the job is running, which is provided from the args,
# default to be dev.
context = args.env
logging.info(
f"Context: {context} -- conf_type: {conf_type} -- team: {team}"
)
Expand All @@ -298,6 +303,11 @@ def set_runtime_env(args):
environment["team_env"] = (
teams_json[team].get(context, {}).get(effective_mode, {})
)
# If the job is running in dev environment but no dev environment is defined in teams.json,
# use production environment.
environment["production_team_env"] = (
teams_json[team].get("production", {}).get(effective_mode, {})
)
environment["default_env"] = (
teams_json.get("default", {})
.get(context, {})
Expand Down Expand Up @@ -326,7 +336,7 @@ def set_runtime_env(args):
environment["cli_args"]["CHRONON_DRIVER_JAR"] = args.chronon_jar
environment["cli_args"]["CHRONON_ONLINE_JAR"] = args.online_jar
environment["cli_args"]["CHRONON_ONLINE_CLASS"] = args.online_class
order = ["conf_env", "team_env", "default_env", "common_env", "cli_args"]
order = ["conf_env", "team_env", "production_team_env", "default_env", "common_env", "cli_args"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we put production_team_env before team_env？

Suggested change
order = ["conf_env", "team_env", "production_team_env", "default_env", "common_env", "cli_args"]
order = ["conf_env", "production_team_env"， "team_env", "default_env", "common_env", "cli_args"]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @better365 the PR has been moved to here #698
For this change if we put production_team_env over team_env, then even for local testing, as long as production environment is provided in team.json file it will always be picked right? We want to use dev environment if we set env=dev.

The logic is:
env=dev: production_team_env = production environment, team_env = dev team environment. and we want to use dev team environment.

print("Setting env variables:")
for key in os.environ:
if any([key in environment[set_key] for set_key in order]):
Expand Down Expand Up @@ -568,6 +578,12 @@ def set_defaults(parser):
required=False,
help="Conf param - required for every mode except fetch",
)
parser.add_argument(
"--env",
required=False,
default='dev',
help="Running environment - default to be dev"
)
parser.add_argument("--mode", choices=MODE_ARGS.keys())
parser.add_argument("--ds", help="the end partition to backfill the data")
parser.add_argument(
Expand Down