Extracts, transforms and loads accounting data fed by PowerMTA pipe into Redis.
./acct_etl -h
Extracts, transforms and loads accounting data fed by PowerMTA pipe into Redis
Usage of ./acct_etl:
-infile string
Input file (omit to read from stdin)
-logfile string
File written with message logs
Here is an example PowerMTA config file showing "accounting pipe" setup. The pipe carries message attributes that "feeder" uses to augment the open and click event data.
| PowerMTA accounting file config | SparkPost term / project usage |
|---|---|
| type | d=delivery |
| rcpt | Recipient (To: address) |
| header_x-sp-message-id | Message ID (added by wrapper) |
| header_x-sp-subaccount-id | Optional subaccount ID. Place in injected message if you wish to use |
You can test without PowerMTA using the included example file:
./acct_etl -infile example.csv
Starting acct_etl, logging to
2020/01/10 19:02:32 PowerMTA accounting headers: [type rcpt header_x-sp-message-id header_x-sp-subaccount-id]
2020/01/10 19:02:32 Loaded acct_headers -> {"header_x-sp-message-id":2,"header_x-sp-subaccount-id":3,"rcpt":1,"type":0} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef0 -> {"header_x-sp-subaccount-id":"0","rcpt":"[email protected]"} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef1 -> {"header_x-sp-subaccount-id":"1","rcpt":"[email protected]"} into Redis
2020/01/10 19:02:32 Loaded msgID_0000123456789abcdef2 -> {"header_x-sp-subaccount-id":"2","rcpt":"[email protected]"} into Redis
The start.sh file copies the acct_etl executable to a place where PowerMTA runs it, and sets owner. It temporarily stops and restarts PowerMTA.
Logfile default location for this process is /opt/pmta/acct_etl.log.
Redis key/value pairs hold data for each message ID, with a configured time-to-live (matching SparkPost's event retention).
You can list these keys with redis-cli keys msgID*.
Redis is used to persist the PowerMTA accounting record header field names and positions. You can see the current value with:
redis-cli get acct_headers
"{\"header_x-sp-message-id\":2,\"header_x-sp-subaccount-id\":3,\"rcpt\":1,\"type\":0}"