Strangely high volume_other_latency

[rodenj1]

I am trying to reconcile some odd differences between Active IQ performance Stats and Harvest collected stats. I am comparing volume_read_latency, volume_write_latency, and volume_other_latency to the "Volume / Performance - Latency" in Active IQ. I know this will not be perfect match, but I would like to assume that it would be in the ballpark. And maybe I need to be using a different set of harvest stats to compare against.

Here is what I am seeing...

"Other" ops stats seem off by quite a bit
volume_other_latency = 163ms
Active IQ Volume Other Latency = 1ms
volume_other_ops = 5IOPS
Active IQ Volume Other Ops = 1IOPS

"Write" ops stats seem close
volume_write_latency = 9.2ms
Active IQ Volume Write Latency = 9.2ms
volume_write_ops = 3791IOPS
Active IQ Volume Write Ops = 2088

"Reads" ops stats seem close
volume_read_latency = 0.6ms
Active IQ Volume Read Latency = 0.4ms
volume_read_ops = 1122IOPS
Active IQ Volume Read Ops = 1743IOPS

Here is what I am running...
Harvest: 2023.08.0
Using: zapi
Cluster Ontap: 9.11.1P8

I am thinking that volume_latency and volume_ops includes other operations that Active IQ is not. If so, is there a better volume level metric I could be using that would match closer to what Active IQ is showing?

[rahulguptajss]

@rodenj1 We have validated Harvest volume performance metrics against ONTAP. Could you try steps mentioned here and let us know if you see differences.

[rodenj1]

Thank you for providing the doc. I have verified that the general volume_avg_latency matches to the cli. It is the volume_other_latency that is way off. And there does not seem to be a CLI command that will show that too me that I can compare against. I have pulled the raw api data and followed the formula in the document of avg_latency = delta(avg latency raw)/ delta(iops raw) and came up with very similar results to what volume_other_latency shows. So it does not seem to be a Harvest math issue, it feels more like the cluster is not calculating correctly.

Here is volume_other_latency

Here is me replicating it with the raw data from the API using this formula other latency = delta(other latency raw)/ delta(other iops raw). Note, I did this with "delta" and "rate" and it was very similar

Here is what Active IQ shows for Volume Other Latency. The thing to note here is volume other latency is below 1ms for the 60 min period.

At this point I think I have verified that Harvest is correctly displaying what the cluster it telling it. It seems that the cluster might not displaying the metric correct. Would you agree with that? Can you think of any other way of verifying this stat?

[rahulguptajss]

@rodenj1 Based on the screenshots you've provided, here are the observations.

Harvest provides volume_other_latency, which is measured in microseconds, as detailed in the Harvest documentation. From the screenshot you provided, it appears that the 'volume_other_latency' reported by Harvest is approximately 300 microseconds, which is equivalent to 0.3 milliseconds.

On the other hand, the AIQ, as per your screenshot, reports a latency of 0.6 milliseconds. Although there is a slight difference between these two values, they are relatively close. This discrepancy could be attributed to factors such as variations in the collection schedule, timing, and other parameters.

Could you confirm if that understanding is correct or highlight any unclear discrepancies.

[rodenj1]

Hi Rahul, all of the latency_other screen shots all include a "/1000" which is already converting from microseconds to millisecond. So it is reporting 300 milliseconds.

[rahulguptajss]

Right. I suggest reviewing the KB article if it helps. If the issue appears to originate from ONTAP, I recommend reaching out to their support team for further assistance.

[rodenj1]

So, I know this is an old question, but I still have no answer for this. I have tried to look at this in many different ways. The KB article was not really helpful. At this point because of all the anomalies the team no longer has any faith in the numbers coming from Harvest. Which is sad because it is a great tool. But I get it. They get alerts from Harvest of high latency, they see HUGE spikes in Harvest data, but when they look at System Manager and AIQUM, these spikes don't exist and worse is the numbers just don't seem match up. I have tried averaging out the spike, changing the sample rate and looked at different latency stats. The issue is they just don't line up and not even close. Here are more examples of volume latency numbers.

Harvest volume_avg_latency stat

Here is the same stat with 30 min sample to match what I think I am getting from System Manager and Active IQ Unified Manager

Here is what system manager says

Here is what AIQUM says

Here is another volume
volume_avg_latency stat

30 min sample

System Manager

AIQUM

I have many more examples of this as well, things with longer spikes as well if that helps.

[rahulguptajss]

Thanks @rodenj1 for the detailed data. Since our discussion last time, a bug has been discovered in Harvest that is the same as the one you are reporting. It was fixed via #2544 and the fix is currently available in the nightly builds. There is another ongoing issue in this area, #2605, which should be resolved soon.

[cgrinds]

Thanks for letting us know @rodenj1 sounds promising. Please let us know if you see anymore anomalies.

[rahulguptajss]

@rodenj1 Could you please update if the latency data looks alright now with the fix?

[rodenj1]

I have been watching for a few days now and all the anomalies have subsided. It looks like the the fixes has resolved the issues I was seeing. Thank you!

[cgrinds]

Awesome! Thanks for raising this issue @rodenj1 and letting us know it is resolved

[rahulguptajss]

@rodenj1 Stable release with these fixes is now available. https://github.com/NetApp/harvest/releases/tag/v24.02.0