0.4.1 Maintenance Update

As time passes, the dependencies nerd relies on continue to improve and, more importantly, introduce patches for bugs and newly discovered vulnerabilities. This release bumps the versions of Go, the build base image and the third party libraries used by this project.

Version Changes

• Go 1.16.0 to 1.17.1
• Cloudevents SDK 2.3.1 to 2.5.0
• Elasticsearch lib 7.10.0 to 7.14.0
• Gin 1.6.3 to 1.7.4
• Google UUID 1.2.0 to 1.3.0
• Redis lib 3.7.0 tp 3.7.1
• Kafka lib 0.4.9 to 0.4.17
• Test containers lib 0.9.0 to 0.11.1

0.4.0: Net Param Auto-tuning

This release introduces the long-promised feature of automatic tuning for network configuration, as well as, greater test coverage and quality gate automation.

Enhancements

Genetic algorithm

Nerd should be easy to use but having to choose an activation function, a learning rate, a number of hidden layers and a net type wasn't helping that goal. Moreover, this approach meant that each nerd instance was limited to creating networks which would only be optimal for solving a reduced set of problems. Additionally, given how weights for the connections between neurons are initialized with random numbers, creating just one meant a greater chance of getting a less accurate network when these led the algorithm to converge on a local minimum.

From now on, nerd will create multiple possible configurations when training a network and use a genetic algorithm to find the one that leads to the most accurate results within a predefined number of generations. For now, this will effectively mean trying out different values for the number of hidden layers and learning rate but when we introduce more activation functions and net types it will also help to find the optimal choice for each of them as well.

Go 1.16

Go 1.16 was released on February 16th, bringing speedier and more efficient builds, as well as, other improvements like the changes to the ParseFloat method of the strconv core library (source) which we expect will translate into faster unmarshalling times for net params and series points in nerd.

Quality Gate Automation

As this project grows, the procedures and tooling for it need to scale accordingly to ensure that quality is maintained. For this reason, we've started using Github actions for automatically running unit and functional tests whenever new code is pushed. That along with the addition of code coverage tracking for our SonarQube project, will help us deliver new features faster with greater confidence.

REST only mode

Nowadays, a lot of interesting projects are built on inexpensive hardware with limited computing resources. For these use cases, we've chosen to make the use of Kafka optional. Starting with this version, nerd can run in a "rest-only" mode when no Kafka brokers are configured, lowering the minimum required specs for a nerd setup to around 64 MB of memory.

Semantic config checks

Documentation is important, but having to go through the whole thing to figure out why your configuration isn't working is a bad user experience. That is why we've added more checks during initialization so that nerd will be more proactive and verbose when it comes to dealing with configurations that don't make sense, throwing errors explaining the issue instead of running with it and exhibiting strange behavior.

Test containers

With a service that is built for supporting more than one way of dealing with certain responsibilities like storing network parameters or series data, testing that new versions don't break compatibility with third party components can become a tedious job. To address this issue, we've started using the testcontainers-go library to automatically spin up containers for these services during testing.

Other

• Added a new query parameter for the nets endpoint to allow filtering by series ID and an alias endpoint at /api/v1/series/{seriesID}/nets
• Renamed the internal/ml package to internal/nets
• Updated the versions of the google uuid, radix and kafka-go dependencies

Fixes

• Fixed some typos

0.3.0: Usability Improvements

This release fixes/improves some of the minor annoyances that we've noticed while using the service as well as adds some documentation and tools to make the project more approachable.

Enhancements

Github issue templates

Open sourcing a project should be about more than just posting the code on Github, it should also invite collaboration either in the form of feedback or PRs. This addition is geared towards furthering the former, making it easier to write good issues. We've taken a page from the hashicorp/vault project and created templates for requesting enhancements and reporting bugs.

User friendly API responses

HTTP response codes are an important tool for providing feedback on the outcome of an API operation but they're not great for people. Tools like curl don't display it by default, which means that nerd would just reply nothing when you performed a delete or submitted a metrics update with it.

With the introduction of the SimpleRes type, we've added JSON responses for those operations that would previously give no feedback in their response body when the outcome was successful.

Nerd collectors

Collectors are lightweight components that serve to import data from other services into nerd, either as a one time operation or as a continuous service.

Starting with this version, there is a producer module (github.com/qvantel/nerd/pkg/producer) that exposes common functionality for producing metrics updates through Kafka or the REST API. Additionally, there is now a new fcollect command for reading csv-style text files and creating metrics updates with them which doubles as a basic example of how this module can be used.

As more open source nerd collectors pop up, we'll figure out the best place to list them so others can get to them faster.

Quick start guide

As previously stated, the idea behind this project is to make things easier but getting started with the existing documentation was more involved than we would like. Because of this, we've created a short guide that shows how to get up and running with basic configuration and presents a fun use case to try nerd with.

Welcome Message

The root or index of an HTTP server is an important location, among other things, because it's the easiest path to get to when you're not familiar with the API. With this thought in mind, we've added a small welcome message there with the version number and a link to the Github project.

Other

• Added missing package go doc comments

Fixes

• When listing series or nets on a fresh instance, clients will now see an empty array instead of a null response or attribute
• The error margins in the README examples now have one less 9 to ensure they don't become 0.5s
• Addressed the ineffectual assignments pointed out by go report card
• Corrected capitalization on some error messages
• Errors detected in main will now result in an error exit code (as they should)
• Fixed some typos

Breaking changes

• types. APIError has been replaced with types. SimpleRes which is now also used for success messages on some API operations

0.2.1: Open sourced!

This is the first public release of Qvantel nerd, a service that is meant to facilitate quick adoption of machine learning into processes that can benefit from it. Currently, nerd is capable of automatically training multilayer perceptron networks from patterns provided through Kafka or its REST API and has a pluggable and scalable architecture that will allow it to be easily extended to support more network types and infrastructure services for storing patterns and network parameters, as well as, scale horizontally to handle more use cases as they come.

We have a lot of ideas of what we want to do with this project and are excited to see what opening it up to the community will bring, so experiment, let us know if anything can be better and above all, do something cool with it!