PyPI v0.16.4

Features

• Add Hydra support #219
• Add cifar ignite example #237
• Add auto extraction of tar.gz files when using StorageManager #237
• Add Task.init() argument auto_connect_streams controlling stdout/stderr/logging capture #181
• Add carriage return flush support using the sdk.development.worker.console_cr_flush_period configuration setting #181
• Add Task.create_function_task() to allow creating a new task using a function and arguments to be executed remotely #230
• Allow disabling SSL certificates verification using Task.setup_upload() argument verify or AWS S3 bucket configuration verify property #256
• Add StorageManager.get_files_server()
• Add Task.get_project_id() using project name
• Add project_name argument to Task.set_project()
• Add Task.connect() support for class / instance objects
• Add Task get_configuration_object() and Task.set_configuration_object()` for easier automation
• Improve Auto-Scaler - allow extra configurations, key name and security group are now optional, defaults using empty strings
• Use a built-in matplotlib convertor
• Add reporting text as debug sample example

Bug Fixes

• Fix Optuna HPO parameter serializing #254
• Fix connect dictionary '' cast to None #258
• Fix lightgbm binding keyword argument issue #251
• Fix artifact preview if artifact body is remote URI #239
• Fix infinite recursion in StorageManager upload #253
• Fix keras reusing model object only if the filename is the same #252
• Fix running remotely with no configuration should not crash but output a warning #243
• matplotlib
  • Fix matplotlib 3.3.3 support
  • Fix global figure enumeration
  • Fix binding without a title reported a single plot (untitled 00) instead of increasing the counter
• Fix Python 2.7/3.5 support
• Fix quote issue when reporting debug images
• Fix replace quote safe characters in upload file to include ;=@$
• Fix at_exit called from another process should be ignored
• Fix Task.set_tags() for completed / published tasks
• Fix Task.add_tags() not working when running remotely
• Fix Task.set_user_properties() docstring and interface
• Fix preview with JSON (dict) artifacts did not store the artifact
• Fix Logger.report_text() on task created using Task.create() was not supported
• Fix initialization for torch: only call torch get_worker_info if torch was loaded
• Fix flush (wait) on auxiliary task (obtained using Task.get_task()) should wait on all upload events
• Fix server was not updated with the defaults from the code when running remotely and configuration section is missing
• Fix connect dict containing None default values, blocked the remote execution from passing string instead of None
• Fix Task.upload_artifact() argument delete_after_upload=True used in conjunction with wait_for_upload=True was not supported

PyPI v0.16.3

Features

• Add LightGBM support
• Add initial Hydra support #219
• Add synchronous support for Task.upload_artifact() #231
• Add sdk.development.store_code_diff_from_remote (default false) to store diff from remote HEAD instead of local HEAD #222
• Add sdk.development.detect_with_conda_freeze (default true) for full conda freeze (requires trains-agent >= 16.2)
• Add user properties support using Task.get_user_properties, Task.set_user_properties() and Task.delete_user_properties()
• Add Logger.report_table() support for table as list of lists
• Add support to split DAG and Table in pipeline DAG plot. Pipeline DAG single nodes are now round circles below the DAG graph.
• Add Pipeline/Optimization can be attached to any Task (not just the current task)
• Add force_download flag to StorageManager.get_local_copy()
• Add control over the artifact preview using Task.upload_artifact() preview argument
• Add Logger.report_matplotlib_figure() with examples
• Add Task.set_task_type()
• AWS auto-scaler
  • Add key pair and security groups support
  • Add multi-line support for both extra bash script and extra trains.conf data
• Update examples

Bug Fixes

• Fix Task.update_output_model() wrong argument order #220
• Fix initializing task on argparse parse in remote mode. Do not call Task.init() to avoid auto connect, use Task.get_task() instead
• Fix detected task cwd outside of repository root folder
• Fix Task.connect(dict) to place non-existing entries on the section name instead of General
• Fix Task.clone() support for trains-server < 0.16
• Fix StorageManager cache extract zipped artifacts. Use modified time instead of access time for cached files
• Fix diff command output was stripped
• Make sure local packages with multi-files are marked as package
• Fix Task.set_base_docker() should be skipped when running remotely
• Fix ArgParser binding handling of string argument with boolean default value (affects Pytorch Lightning integration)
• When using detect_with_pip_freeze make sure that package @ file:// lines are replaced with package==x.y.z as local file will probably not be available
• Fix git packages to new pip standard package @ git+
• Improve conda package naming _ and - support
• Do not add specific setuptools version to requirements (pip can't install it anyway)
• Fix image URL quoting when uploading from a file path

PyPI v0.16.2

Features

• Add Task.set_resource_monitor_iteration_timeout() to set ResourceMonitor iteration wait duration timeout #208
• Add PyTorch Lightning save/restore model binding #212
• Add git diff for repository submodule (requires git 2.14 or above)
• Add TrainsJob.is_completed() and TrainsJob.is_aborted()
• Add Task.logger property
• Add Pipeline Controller automation and example (see here)
• Add improved trace filtering capabilities in trains.debugging.trace.trace_trains()
• Add default help per argument (if not provided) in ArgParser binding
• Deprecate Task.reporter
• Update PyTorch example
• Remove warning on skipped auto-magic model logging #206
• Support Keras restructuring for Network, Model and Sequential
• Update autokeras requirements according to https://github.com/keras-team/autokeras#installation

Bug Fixes

• Fix joblib auto logging models failing on compressed streams #203
• Fix sending empty reports #205
• Fix scatter2d sub-sampling and rounding
• Fix plots reporting
  • NaN representation (matplotlib conversion)
  • Limit the number of digits in a plot to reduce plot size (using sdk.metrics.plot_max_num_digits configuration value)
• Fix Task.wait_for_status() to reload after it ends
• Fix thread wait Ctrl-C interrupt did not exit process
• Improve Windows support for installed packages analysis
• Fix auto model logging using relative path
• Fix Hyper-parameter Optimization example
• Fix Task.clone() when working with TrainsServer < 0.16.0
• Fix pandas artifact handling
  • Avoid adding unnamed:0 column
  • Return original pandas object
• Fix TrainsJob hyper-params overriding order was not guaranteed
• Fix ArgParse auto-connect to support default function type

PyPI v0.16.1

Features and Bug Fixes

• Fix typing dependency for Python<3.5 #184
• Fix git+https requirements handling, resolve top_level.txt package name (kerastuner from git was not detected)
• Fix Task.get_reported_console_output() for new Trains Server API v2.9
• Fix cache handling for different partitions/drives/devices
• Disable offline mode when running remotely (i.e. executed by Trains Agent)
• Fix artifact upload to only use file stream when not uploading a locally stored file (multipart upload is not supported on stream upload) #189
• Fix double-escaped model design text when connecting OutputModel
• Enhance HyperParameter optimizer

PyPI v0.16.0

Features

• Add continuing of previously executed experiments. Add Task.init() argument continue_last_task to continue a previously used Task #160
• Allow Task editing/creation from code. Task.export_task/import_task/update_task() #128
• Add offline mode. Use Task.set_offline() and Task.import_offline_session()
  • Support setting offline mode via TRAINS_OFFLINE_MODE=1 environment variable
  • Support setting offline API version via TRAINS_OFFLINE_MODE=2.9 environment variable
• Automatically pickle all objects when uploading as artifacts, task.upload_artifact() argument auto_pickle=True #153
• Add multiple sections/groups support for Task hyper-parameters using Task.connect()
• Add multiple configurations (files) using Task.connect_configuration
• Allow enabling OS environment logging using the sdk.development.log_os_environments configuration parameter (complements the TRAINS_LOG_ENVIRONMENT environment variable)
• Add Optuna support for hyper-parameter optimization controller. OptimizerOptuna is now the default optimizer.
• Add initial Keras-Tuner support (keras-team/keras-tuner#334)
• Add automatic FastAI logging. It is disabled if tensorboard is loaded (assuming TensorBoradLogger will be used)
• Support Tensorboard text logging (add_text()) as debug samples (.txt files), instead of as console output
• Allow for more standard confusion matrix reporting. Logger.report_confusion_matrix() argument yaxis_reversed (flips the confusion matrix if True, default False) #165
• Add support for Trains Server 0.16.0 (API v2.9 support)
• Allow disabling Trains update message from the log using the TRAINS_SUPPRESS_UPDATE_MESSAGE environment variable #157
• Add AWS EC2 Auto-Scaler service wizard and Service
• Improved and updated examples
  • Add Keras Tuner CIFAR10 example
  • Add FastAI example
  • Update PyTorch Jupyter notebook examples #150
• Support global requirements detection using pip freeze (set sdk.development.detect_with_pip_freeze configuration in trains.conf)
• Add Task.get_projects() to get all projects in the system, sorted by last update time

Bug Fixes

• Fix UTC to time stamp in comment #152
• Fix and enhance GPU monitoring
  • Fix GPU stats on Windows machines #177
  • More robust GPU monitoring #170
• Fix filename too long bug allegroai/clearml-server#49
• Fix TensorFlow image logging to allow images with no width/height/color metadata #182
• Fix multiprocessing Pool throw exception in pool hangs execution. Call original signal handler and re-flush stdout
• Fix plotly support for matplotlib 3.3
• Add Python 2.7 support for get_current_thread_id()
• Update examples requirements
• Fix and improve signal handling
• Fix Tensorboard 2D convolution histogram, improve histogram accuracy on very small histograms
• Fix auto logging multiple argparse calls before Task.init()
• Limit experiment Git diff logging to 500Kb. If larger than 500Kb, diff section will contain a warning and entire diff will be uploaded as an artifact named auxiliary_git_dif
• Fix requirements detection
  • Fix Trains installed from git+
  • Fix when Trains is not directly imported
  • Fix multiple -e packages were not detected (only the first one)
  • Fix running with Trains in PYTHONPATH resulted in double entry of trains
• Fix Task.set_base_docker() on main task to do nothing when running remotely

PyPI v0.15.1

Features

• Add direct plotly figure reporting #136
• Improve matplotlib integration #140
• Add Task.set_parent() #144
• Add Task.wait_for_status() #145
• Enhance framework save/load binding callbacks pytorch/ignite#1056
• Add TrainsJob.is_failed()
• Add APIClient support using trains.backend_api.session.APIClient
• Add URL tracking when using StorageManager.get_local_copy() (log remote url instead of local file copy)
• Improve & expand examples

Bug Fixes

• Fix pytorch_tensorboard.py example #148
• Fix ssh://user@[email protected]/ in Git repository link #149
• Fix do not call make_deterministic() on import
• Fix initialization wizard (allow at most two verification retries, then print error) trains-agent #16
• Fix Google CoLab code/package detection
• Improve Pytorch-Ignite integration
• Fix TensorFlow >=2 histogram binding
• Fix Logger.tensorboard_single_series_per_graph
• Fix anonymous named models
• Fix incorrect entry point detection when called from Trains wrapper (e.g. TrainsLogger in Pytorch Ignite/Lightning)

PyPI v0.15.0

Features

• Add automation support including hyper-parameters optimization (see example here)
• Task.init() auto_connect_arg_parser argument can accept a dictionary disabling specific keys from the argparser (Trains Slack channel thread)
• Allow worker_id override using TRAINS_WORKER_NAME environment variable (Trains Slack channel thread)
• Support layout configuration for plotly objects using extra_layout argument in all Logger reporting methods #136
• Add Task.execute_remotely() to allow cloning and enqueuing a locally executed task (or stopping and re-enqueuing a remotely executed task) #128
• Add Parquet framework and model type
• Support recursive model folder packaging
• Add Task.get_reported_console_output() and Task.get_reported_scalars() to allow retrieval of reported reported output and scalar metrics
• Add Task.add_requirements() to force requirement package into "installed packages"
• Improve task reuse responsiveness
• Add raise_on_error (default False) argument to Model.get_local_copy() and Artifact.get_local_copy() allegroai/clearml-agent#17
• Support Task.get_task() without project name (i.e. all projects)
• Support using the file calling Task.init() as the task's script in case sys.argv doesn't point to a git repository
• Support detecting and remotely executing code running from a module (i.e. -m module)
• Add callback for framework save/load binding for better integration with pytorch/ignite pytorch/ignite#1056
• Support new task types provided in Trains Server v0.15.0
• Add automation and distributed examples
• Upgrade default pip version to <20.2

Bug Fixes

• Fix exact_match_regex() in case of empty pattern #138
• Address deprecation warning and newer attrs versions in MetricsEventAdapter #134
• Fix issues with plotly support (Trains Slack channel thread and thread)
• Fix default argument behavior to match argparse behavior
• Fix OutputModel with task=None should use current task, if exists
• Fix Task.get_task() to raise proper error on incorrect task_id
• Fix Task.enqueue() to use an exact queue name match
• Fix NaN, Inf and -Inf values display in reported table (not supported by JSON)
• Limit max requirement size to 0.5MB
• Fix issues with repository analysis
• Fix StorageManager should only try to extract .zip files, Model should not auto extract package allegroai/clearml-agent#17

PyPI v0.14.3

Features and Bug Fixes

• Add Logger histogram mode (stack/group/relative)
• Add PEP610 support for new pip>=20.1 git reference feature (see pypa/pip#609) #62
• Better support for PyTorch Ignite integration
• Support # and ? in uploaded image file names
• Support ArgumentParser.parse_arg() used before initializing task when auto_connect_arg_parser=False
• Fix upload_object_via_stream() in Azure storage driver
• Fix Task.get_tasks() return type

PyPI v0.14.2

Features

• Add media (audio) support for both Logger and Tensorboard binding using Logger.report_media() #120
• Add autokeras example (see here) #125
• Add caching of extracted zip artifacts
• Add Task.get_tasks() filtering support
• Add trains.storage.StorageManager
• Add nicer stdout log flush
• Add text logging to TF V1 example
• Improve SSL retries and error handling
• Improve shutdown behavior in case of a user abort (Ctrl-C)
• Improve repository and module detection
• Improve resource monitor
  • Report memory usage for process (and sub-processes)
  • Report GPU memory usage for process (and sub-processes)
  • Stability
• Improve Jupyter support
  • Make sure trains is included in Jupyter requirements
  • Ignore IPython directives in converted Python script (like % and ! lines)
• Update Pytorch/TensorboardX examples

Bug Fixes

• Fix renaming/deleting model file right after saving will break asynchronous upload #123
• Fix shutdown behavior:
  • Logger flush delay and termination
  • Repository/package detection termination
  • Threads termination
• Fix text encoding utf-8 and pr_curve broken in Tensorboard support
• Fix single log request exceeds max packet size
• Fix upload debug assets with series / title including , or / did not add iteration index to the filename
• Fix scikit-learn module detection
• Use UID in message if user name cannot be retrieved (from the Trains Slack channel)
• Fix cast in Tensorflow v2 example

PyPI v0.14.1

Features and Bug Fixes

• Add OpenMPI/Slurm support, support TRAINS_FORCE_MASTER_NODE environment variable to allow forcefully specifying a node is master (not based on the MPI Rank)
• Add Task.get_models() and Task.models for retrieving stored models for previously executed tasks
• Support multiple EventWriter in TensorFlow eager mode (TF 2.0+) #26
• Support reusing models
• Fix TensorFlow NaN/Inf values support #118
• Fix joblib binding
• Fix metrics/files/models upload on shutdown
• Fix support for Task init/close multiple times
• Update manual_reporting.py example with Logger.report_table() example
• Deprecate Task.set_model_config(), Task.get_model_config_text() and Task.get_model_config_dict()