Harvest data collecting limit

[wooyoungAhn]

Hello everyone

have a nice day
I have 10 clusters registered to Harvest.

However, some metrics are not being collected normally.

We have around 1000 volumes per cluster.
However, only about 250 volume information is checked during volume inquiry.

Is there a limit to the maximum amount of inventory and performance data that can be collected?

[cgrinds]

hi @wooyoungAhn there is not a limit and collect 1K volumes should be fine. Can you send your poller's logs files and we'll take a look.

[rahulguptajss]

@wooyoungAhn You may be hitting issue #2109 if you are using Rest Collector. This issue is fixed in nightly builds.

[wyahn1219]

Hello All
Thank you for your help
@cgrinds I send log files.

# normal
# curl -s localhost:12901/metrics | grep -w sim_dram_al16gd4n | wc -l
46

# problem
# curl -s localhost:12901/metrics | grep -w sim_dram_cp16gmd5s  | wc -l
9

@rahulguptajss
thank you for your sharing
when exporting trace level log, I compared zapi and Rest collected.

# curl -s localhost:12901/metrics | grep volume_aggr_labels | wc -l
616
# curl -s localhost:12901/metrics | grep volume_labels | wc -l
237

I will try to upgrade.

[wooyoungAhn]

Hello

Today I updated Nightly version and the problem was solved.
Thank you for your help.

[LaurentB-14]

Hello,

One of my customer faces the same issue from release 22.11 and ZAPI to Rest API conversion.
He has 2 Netapp clusters with around 3000 and 1500 volumes.
He collects 300 volumes for the first one and 500 for the second one.
He tried with different Harvest releases 22.11, 23.02 and the lastest one 23.11.
He tried with different setting : time_out and batch_size without success.

His clusters are Ontap version 9.12.X.
For small clusters, data collect works.

Thank you for your help.

[rahulguptajss]

@LaurentB-14

1. Could you please send your harvest logs to [email protected]?. You can find instructions on how to collect these logs here.
2. Are are experiencing issues with both the Zapi and Rest Collector?
3. It seems like you might be referring to the client_timeout parameter. Could you please share the value you have set for this parameter? You can find more information about client_timeout here.
4. Are there any customizations to Volume template? If yes, Please share the template.

[LaurentB-14]

Hello Rahul,

Thank you for your answer.
In fact my customer wasn't use harvest and grafana contenairs with nabox environment.
He tested nabox with default contenairs successfully.

Thank you again.
We can close this discussion for the moment.
Have a nice day.

BR