Large changes

[johnseekins]

1. Move to python3 (since python2 is considered un-supported).
2. Make a lot of the .value metrics actually show the proper metric
3. make better use of influx format to reduce printed lines (this improves telegraf's performance with larger amounts of data)
4. flake8 passes
5. example input config file
6. more restrictive telegraf sudo commands (so we don't give the telegraf user the ability to delete/modify btrfs volumes)
7. patch in the other MR in your repo
8. add scrub stats (need to work on the matching to get start/running/end times)

Example new output:

btrfs_stat,pool=/,drive=/dev/md125 write_io_errs=0,read_io_errs=0,flush_io_errs=0,corruption_errs=0,generation_errs=0
btrfs_df,type=Data,raidType=single,pool=/ total=13967032320,used=10596356096,free=3370676224
btrfs_df,type=System,raidType=single,pool=/ total=33554432,used=16384,free=33538048
btrfs_df,type=Metadata,raidType=single,pool=/ total=2155872256,used=163217408,free=1992654848
btrfs_df,type=GlobalReserve,raidType=single,pool=/ total=26869760,used=0,free=26869760
btrfs_usage,pool=/ device_size=245823963136,device_allocated=16156459008,device_unallocated=229667504128,device_missing=0,used=10759589888,free=233038180352,data_ratio=1.00,metadata_ratio=1.00,global_reserve=26869760
btrfs_usage,pool=/,drive=/dev/md125 Data=13967032320,Metadata=2155872256,System=33554432,Unallocated=229667504128
btrfs_scrub,pool=/ data_extents_scrubbed=275641,tree_extents_scrubbed=9965,data_bytes_scrubbed=10596188160,tree_bytes_scrubbed=163266560,read_errors=0,csum_errors=0,verify_errors=0,no_csum=1958,csum_discards=0,super_errors=0,malloc_errors=0,uncorrectable_errors=0,unverified_errors=0,corrected_errors=0,last_physical=17235443712
btrfs_stat,pool=/data,drive=/dev/sda write_io_errs=0,read_io_errs=0,flush_io_errs=0,corruption_errs=0,generation_errs=0
btrfs_stat,pool=/data,drive=/dev/sdb write_io_errs=0,read_io_errs=0,flush_io_errs=0,corruption_errs=0,generation_errs=0
btrfs_stat,pool=/data,drive=/dev/sdc write_io_errs=0,read_io_errs=0,flush_io_errs=0,corruption_errs=0,generation_errs=0
btrfs_stat,pool=/data,drive=/dev/sdd write_io_errs=0,read_io_errs=0,flush_io_errs=0,corruption_errs=0,generation_errs=0
btrfs_df,type=Data,raidType=RAID1,pool=/data total=4169339502592,used=4132889722880,free=36449779712
btrfs_df,type=System,raidType=RAID1,pool=/data total=33554432,used=573440,free=32980992
btrfs_df,type=Metadata,raidType=RAID1,pool=/data total=7516192768,used=5457084416,free=2059108352
btrfs_df,type=GlobalReserve,raidType=single,pool=/data total=536870912,used=0,free=536870912
btrfs_usage,pool=/data device_size=32006252888064,device_allocated=8353778499584,device_unallocated=23652474388480,device_missing=0,used=8276694761472,free=11862686973952,data_ratio=2.00,metadata_ratio=2.00,global_reserve=536870912
btrfs_usage,pool=/data,drive=/dev/sda Data=2081985396736,Metadata=6442450944,Unallocated=5913135374336
btrfs_usage,pool=/data,drive=/dev/sdb Data=2083059138560,Metadata=5368709120,Unallocated=5913135374336
btrfs_usage,pool=/data,drive=/dev/sdc Data=2087354105856,Metadata=1073741824,System=33554432,Unallocated=5913101819904
btrfs_usage,pool=/data,drive=/dev/sdd Data=2086280364032,Metadata=2147483648,System=33554432,Unallocated=5913101819904
btrfs_scrub,pool=/data data_extents_scrubbed=127438186,tree_extents_scrubbed=666190,data_bytes_scrubbed=8265474105344,tree_bytes_scrubbed=10914856960,read_errors=0,csum_errors=0,verify_errors=0,no_csum=497792,csum_discards=0,super_errors=0,malloc_errors=0,uncorrectable_errors=0,unverified_errors=0,corrected_errors=0,last_physical=8394756849664

[1n5aN1aC]

Thanks for this, looking great!

Just an FYI, BTRFS 5.10 adds another line to be parsed like 5.7 did, causing it to break it again. (Unless you've handled it differently)

Since it looks like you haven't changed how that section is handled, the quick and dirty fix, though there's gotta be a better solution:

if "statfs" in metric:
                    continue

[johnseekins]

I don't have any boxes running 5.10 yet, so I didn't realize that was a thing...what command produces the statfs line?

[1n5aN1aC]

I don't have any boxes running 5.10 yet, so I didn't realize that was a thing...what command produces the statfs line?

btrfs fi usage. It's the same exact problem/solution you worked around HERE by using my solution.

In my hack-y solution, (still using a modification of the original script, not yours yet) I just added another line to that:

if "Multiple_profiles" in metric:
    continue
if "statfs" in metric:
    continue

Here's my BTRFS output:

root@HOSTNAME:~# btrfs --version
btrfs-progs v5.10
root@HOSTNAME:~# btrfs fi usage /mnt/data
Overall:
    Device size:                  92.78TiB
    Device allocated:             84.01TiB
    Device unallocated:            8.77TiB
    Device missing:                  0.00B
    Used:                         83.95TiB
    Free (estimated):              4.40TiB      (min: 2.94TiB)
    Free (statfs, df):             3.32TiB
    Data ratio:                       2.00
    Metadata ratio:                   3.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

[johnseekins]

So this should work then:

                if any(k in j for k in ["Multiple_profiles", "statfs"]):
                    continue

[1n5aN1aC]

Yup, that should do it!

If I was doing it, I'd prefer to iterate through and only parse lines I knew how to parse, rather than excluding lines I didn't know how to parse, which would make it much more future-proof, but that's a lot more work and potential code bloat, so /shrug.

At any rate, that should work, as it's functionally equivalent to what I'm running on a 5.10 box fight now.

[johnseekins]

It's really unfortunate that, generally, btrfs (and zfs) have terrible reporting tools. Having to do all this string parsing just to get useful data is a real bummer. (Can you imagine if there was a --json switch for all these commands? ❤️ )

You're right to want to switch from exclusive to inclusive filtering (and I kinda did that with the scrub parsing), but that's a lot of extra work that I don't really have time for right now.