Tune New Relic Apdex Score Parameters

[btylerburton]

User Story

In order to make better use of our monitoring and alerts, datagovteam wants to tune the Apdex score to ensure we are delivering an acceptable user experience.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN our logs' response times are being used to determine Apdex user experience score
  THEN we want to set realistic upper and lower bounds to prevent always/never being in alarm
  AND instead to know when performance degrades from what is expected.

Background

After speaking with NR support on 9.27.23, we determined that our Apdex score is not tuned effectively, resulting in our site being in a constant state of alert. We want to tune that metric to more accurately reflect normal performance for our site.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Follow guidelines below to tune NR Apdex score for data.gov
  • https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measure-user-satisfaction/#:~:text=The%20Apdex%20score%20is%20a,as%20half%20a%20satisfied%20request.
  • https://newrelic.com/blog/best-practices/how-to-choose-apdex-t
  • https://www.youtube.com/watch?v=RoLIBfEpYZg
  • https://www.freshworks.com/website-monitoring/what-is-apdex/
• Observe less alerts

[Jin-Sun-tts]

how to reach target Apdex score to 90, and we need to set percentage to 80%

based on following formula, the value should be close to 0
select percentile(duration, 80) from Transaction where appID=xxxx since 24 hours ago

here is the current stats, when Apdex set to 0.5 second by default:

[Jin-Sun-tts]

select percentile(duration, 80) from Transaction where entityGuid=xxxx since 30 days ago
1.781

Also re-run above query this morning, got 2.063, so setup up this threshold number and the Apdex should reach 88, will monitor it in 24 hours.

[Jin-Sun-tts]

we may reach .92 Apdex score when do the threshold 3.087 from following query:
select apdex(duration, t: 3.087) from Transaction where entityGuid=xxxx since 30 days ago

setup this threshold for now and will continue monitoring the dashboard and message alert. close this issue.

[hkdctol]

Reopening. As discussed 12/15, we are still seeing too many alerts. We should raise parameter again, and then monitor alerts for the next few days, to get at the right level.

[Jin-Sun-tts]

Apdex scores have been configured exclusively for the production environment, and they have been performing well thus far.

However, we've been receiving Apdex score-related warning messages from the staging environment. To address this, we have decided to establish the same threshold values for Apdex scores in the staging environment and closely monitor any warning emails.

Regarding the warning emails related to the 'error percentage,' we have set up alert conditions that were initially triggered at a 1% error rate. For now, we have adjusted this threshold to 5% to assess the frequency of these email alerts.

Furthermore, we will be conducting an investigation to identify potential solutions for filtering out errors like 404 etc.

[Jin-Sun-tts]

Attempted to exclude specific HTTP response codes from the server-side configuration, but the initial approach to add them to the ignore list did not yield the desired results. Currently, all log messages appear as plain text, and it seems we lack a mechanism to filter them based on HTTP response codes.

So we decide to implement a customized query or filter within the New Relic dashboard like in #4234

Close this ticket as the warning emails (from staging) reduced after setup threshold, and error percentage adjustment for the alert condition to lead less warning emails from New Relic.