-
Notifications
You must be signed in to change notification settings - Fork 556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] ApproximateCreationDateTime from DynamoDBStreamRecord in milliseconds when originating from Kinesis, not seconds. #478
Comments
It may not crash, but Unmarshaling and Marshaling a value will cause an overflow for most Unix timestamps due to calling package main
import (
"encoding/json"
"fmt"
"time"
)
// SecondsEpochTime serializes a time.Time in JSON as a UNIX epoch time in seconds
type SecondsEpochTime struct {
time.Time
}
const secondsToNanoSecondsFactor = 1000000000
const milliSecondsToNanoSecondsFactor = 1000000
func TestUnmarshalJSON(epoch float64) time.Time {
epochSec := int64(epoch)
epochNano := int64((epoch - float64(epochSec)) * float64(secondsToNanoSecondsFactor))
return time.Unix(epochSec, epochNano)
}
func TestMarshalJSON(t time.Time) ([]byte, error) {
// UnixNano() returns the epoch in nanoseconds
unixTime := float64(t.UnixNano()) / float64(secondsToNanoSecondsFactor)
return json.Marshal(unixTime)
}
func testVal(test float64) time.Time {
convertedTime := TestUnmarshalJSON(test)
convertBack, _ := TestMarshalJSON(convertedTime)
fmt.Printf("%f\t%s\t%s\n", test, convertedTime.String(), convertBack)
return convertedTime
}
func main() {
testVal(1669739327580.0) // Millisecond
testVal(1669739327.0) // Second
}
|
aws/aws-lambda-dotnet#839 (comment) claims that this only occurs when the dynamo event is first passed through kinesis. Do I understand that correctly? If so, I'm not sure if this is something that's supportable, and the function should operate on the Kinesis event rather than the Dynamo event |
I missed the reply on this issue from last year, but it seems to still be relevant.
Technically, yes, the pipeline is DynamoDB table -> DynamoDB stream -> Kinesis Data stream -> Firehose (where a transformation Lambda is called) -> Firehose destination That said, the events passed into the transformation lambda are Kinesis Firehose Events which contain Kinesis Firehose Event Records, which in turn have a When configured as above, the So There is the
Lastly, the original issue that was referenced (aws/aws-lambda-dotnet#839) appears to have been closed, using the same workaround the we implemented back in 2023: Check if the timestamp is more than 5,000 years in the future, and convert to milliseconds if so:
I'm not sure if that same fix should be incorporated here into the library, but the above has been working reasonably well for us over the last 18 months. |
Confirming that I am experiencing the same error, while using kinesis dynamodb stream with firehose/lambda. There is a {
"dynamodb": {
"ApproximateCreationDateTime": 1731101300058336,
"ApproximateCreationDateTimePrecision": "MICROSECOND",
}
} |
This is essentially the same issue as aws/aws-lambda-dotnet#839, but without crashing deserialization, which I'm guessing is due to the use of
float64
that avoids overflowing with the larger value to deserialize. The relevant points of discussion are:It seems that the value of ApproximateCreationDateTime will be in seconds when coming from a DynamoDB Stream, but in milliseconds when coming from a Kinesis Stream:
There appears to be an internal ticket that's being tracked, so I wanted to open an issue here as well for AWS to monitor and hopefully resolve in the near future. Thanks!
The text was updated successfully, but these errors were encountered: