Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow visibility into errors #97

Open
mrroach opened this issue Mar 18, 2020 · 3 comments
Open

Allow visibility into errors #97

mrroach opened this issue Mar 18, 2020 · 3 comments

Comments

@mrroach
Copy link

mrroach commented Mar 18, 2020

At the moment, all ingest errors are silently swallowed:

                try:
                    self._post(self._batch_data(datapoints_list),
                               '{0}/{1}'.format(
                                   self._endpoint,
                                   self._INGEST_ENDPOINT_DATAPOINT_SUFFIX))
                except:
                    _logger.exception('Posting data to SignalFx failed.')

I would like to be able to at least have a count of the types of exceptions that occur so that I can track them via other mechanisms (statsd, other internal health check code)

An approach I am using with other metric sending systems is something like this:

    ┆   # A mapping of error type to error count                                                                                                                                                                                       
    ┆   self.error_lock = threading.Lock()
    ┆   self.error_counters = collections.defaultdict(lambda: 0)

    def inc_error(self, error_type):
    ┆   """Increment internal counter of errors encountered.
    ┆   :param error_type: str, Exception class or other descriptor of error.
    ┆   """
    ┆   with self.error_lock:
    ┆   ┆   self.error_counters[error_type] += 1

    def reset_error_counters(self):
    ┆   """Reset error counters to 0 and return the previous values.
    ┆   :return: dict, Mapping of error type to count.
    ┆   """
    ┆   with self.error_lock:
    ┆   ┆   previous = self.error_counters
    ┆   ┆   self.error_counters = collections.defaultdict(lambda: 0)
    ┆   return previous

And then to use this in the sending code:

    def send(self):
    ...
    ┆   try:
    ┆   ┆   response = send_metrics(self.session, self.config, content)
    ┆   except RequestException as err:
    ┆   ┆   self.inc_error(err.__class__.__name__)
@mrroach
Copy link
Author

mrroach commented Mar 18, 2020

I'll add that if this approach is acceptable, I'd be happy to send a PR

@mmusik
Copy link

mmusik commented Mar 24, 2020

Hi (a disclaimer - I'm an employee of Splunk/SignalFx). What you propose sounds reasonable to me. I think it is a good suggestion and I'll be happy to help with reviews if you prepare a PR. Thanks!

@mrroach
Copy link
Author

mrroach commented Mar 24, 2020

Great, thank you! I'll put it together shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants