S3.SelectObjectContent throws SignatureDoesNotMatch exception

[nitinprakash96]

Trying out the s3 select feature by amazon using SelectObjectContent API by amazonka results in SignatureDoesNotMatch despite correct credentials. The following is the only function that fails out of all that are listed in examples/src/S3.hs.

I'm not sure if I am doing something wrong here or there's something wrong with the API itself.

Here's a small function that can be plugged into example/src/S3.hs module:

defaultRegion :: Region
defaultRegion = "us-east-1"

-- >>> selectObjectContent (BucketName "nitin-testing") (ObjectKey "airtravel.csv") "select * from S3Object limit 5"
selectObjectContent :: BucketName -> ObjectKey -> Text -> IO ()
selectObjectContent bucketName s3Key query = do
    lgr <- newLogger Debug stdout
    env <- newEnv discover <&> set #logger lgr . set #region defaultRegion

    let inputSerializationConfig = newInputSerialization
            & inputSerialization_csv
                ?~ ( newCSVInput
                        & cSVInput_fieldDelimiter ?~ ","
                        & cSVInput_fileHeaderInfo ?~ FileHeaderInfo_USE
                    )
        outputSerializationConfig = newOutputSerialization
            & outputSerialization_csv
                ?~ ( newCSVOutput
                        & cSVOutput_fieldDelimiter ?~ ","
                        & cSVOutput_recordDelimiter ?~ "\n"
                    )
        request =
            newSelectObjectContent bucketName s3Key query ExpressionType_SQL inputSerializationConfig outputSerializationConfig

    void . runResourceT $ send env request

which results in he following error:

[ServiceError] {
  service    = S3
  status     = 403 Forbidden
  code       = SignatureDoesNotMatch
  message    = Just The request signature we calculated does not match the signature you provided. Check your key and signing method.
  request-id = Just AD3M8H0XG2Y351WD
}

Also, I'm using AWS_PROFILE for calling these APIs.

[endgame]

In which region did you create the bucket? Although S3 is a global service, buckets are attached to regions and you might be signing your request for the wrong region.

[nitinprakash96]

@endgame us-east-1 (same as defaultRegion in the above snippet)

[endgame]

Weird. Here are some things I'd like to see to help debug this:

1. The request dumped by Amazonka debug loggin.
2. The equivalent request made via the AWS CLI, using --debug true
3. If you're testing with open data, a description of how you set up and populated the bucket so I can try stuff locally.

[nitinprakash96]

Request dump:

[Client Request] {
  host      = nitin-testing.s3.us-east-1.amazonaws.com:443
  secure    = True
  method    = POST
  target    = Nothing
  timeout   = ResponseTimeoutMicro 70000000
  redirects = 0
  path      = /airtravel.csv
  query     = ?select-type=2&select=
  headers   = x-amz-content-sha256: REDACTED; x-amz-date: 20231118T040907Z; host: nitin-testing.s3.us-east-1.amazonaws.com; authorization: AWS4-HMAC-SHA256 Credential=REDACTED20231118/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
  body      = <?xml version="1.0" encoding="UTF-8"?><SelectObjectContentRequest xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Expression>Select * from S3Object limit 5</Expression><ExpressionType>SQL</ExpressionType><InputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><FileHeaderInfo>USE</FileHeaderInfo></CSV></InputSerialization><OutputSerialization><CSV><FieldDelimiter>,</FieldDelimiter><RecordDelimiter>
</RecordDelimiter></CSV></OutputSerialization></SelectObjectContentRequest>
}
[Client Response] {
  status  = 403 Forbidden
  headers = <REDACTED>
}

Via AWS cli:

➜  ~ aws s3api select-object-content --debug --bucket nitin-testing --key "airtravel.csv" --expression "select * from s3object limit 5" --expression-type 'SQL' \
    --input-serialization '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}' \
    --output-serialization '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}' "output.csv" --profile nitin-dev --region us-east-1

output

2023-11-18 11:33:38,953 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.11.4 Darwin/22.6.0 exe/x86_64
2023-11-18 11:33:38,953 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3api', 'select-object-content', '--debug', '--bucket', 'nitin-testing', '--key', 'airtravel.csv', '--expression', 'select * from s3object limit 5', '--expression-type', 'SQL', '--input-serialization', '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}', '--output-serialization', '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}', 'output.csv', '--profile', 'nitin-dev', '--region', 'us-east-1']
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x10c574180>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x10c3a1a80>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10c307b00>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x10c319120>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x10c576c00>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x10c3fc4a0>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x10c576ac0>
2023-11-18 11:33:38,977 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10c639110>>
2023-11-18 11:33:38,978 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2023-11-18 11:33:38,979 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x10c4a9da0>
2023-11-18 11:33:38,979 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x10c4aa0c0>
2023-11-18 11:33:38,994 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x10c4aa020>
2023-11-18 11:33:38,994 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x10c4aa200>
2023-11-18 11:33:38,994 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x10c4aa160>
2023-11-18 11:33:38,994 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x10c62e280>
2023-11-18 11:33:38,995 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'nitin-dev'
2023-11-18 11:33:38,995 - MainThread - botocore.session - DEBUG - Setting config variable for region to 'us-east-1'
2023-11-18 11:33:38,995 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.11.4 Darwin/22.6.0 exe/x86_64 prompt/off
2023-11-18 11:33:38,995 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3api', 'select-object-content', '--debug', '--bucket', 'nitin-testing', '--key', 'airtravel.csv', '--expression', 'select * from s3object limit 5', '--expression-type', 'SQL', '--input-serialization', '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}', '--output-serialization', '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}', 'output.csv', '--profile', 'nitin-dev', '--region', 'us-east-1']
2023-11-18 11:33:38,995 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x10c574ae0>
2023-11-18 11:33:38,995 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x10bea2660>
2023-11-18 11:33:38,995 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x10c5f0040>
2023-11-18 11:33:38,995 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x10bd9c900>
2023-11-18 11:33:38,995 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x10bebdf80>
2023-11-18 11:33:38,997 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-11-18 11:33:39,001 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2023-11-18 11:33:39,001 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x10c3d0c20>
2023-11-18 11:33:39,001 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x10c37dd00>
2023-11-18 11:33:39,026 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/service-2.json
2023-11-18 11:33:39,037 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function add_waiters at 0x10c576ac0>
2023-11-18 11:33:39,064 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/waiters-2.json
2023-11-18 11:33:39,064 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10c639110>>
2023-11-18 11:33:39,065 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('bucket', <awscli.arguments.CLIArgument object at 0x10c8ff790>), ('key', <awscli.arguments.CLIArgument object at 0x10c8ff950>), ('sse-customer-algorithm', <awscli.arguments.CLIArgument object at 0x10c8ffa90>), ('sse-customer-key', <awscli.arguments.CLIArgument object at 0x10c8ffa50>), ('sse-customer-key-md5', <awscli.arguments.CLIArgument object at 0x10c8ec0d0>), ('expression', <awscli.arguments.CLIArgument object at 0x10c8ec250>), ('expression-type', <awscli.arguments.CLIArgument object at 0x10c8ec410>), ('request-progress', <awscli.arguments.CLIArgument object at 0x10c8ec510>), ('input-serialization', <awscli.arguments.CLIArgument object at 0x10c8ec6d0>), ('output-serialization', <awscli.arguments.CLIArgument object at 0x10c8ec790>), ('scan-range', <awscli.arguments.CLIArgument object at 0x10c8eca10>), ('expected-bucket-owner', <awscli.arguments.CLIArgument object at 0x10c8ecb90>)])
2023-11-18 11:33:39,065 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function add_streaming_output_arg at 0x10c574fe0>
2023-11-18 11:33:39,065 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function add_event_stream_output_arg at 0x10c594720>
2023-11-18 11:33:39,065 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function add_cli_input_json at 0x10bebe8e0>
2023-11-18 11:33:39,065 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function add_cli_input_yaml at 0x10bebe980>
2023-11-18 11:33:39,065 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function unify_paging_params at 0x10c3a20c0>
2023-11-18 11:33:39,096 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2023-11-18 11:33:39,096 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/paginators-1.sdk-extras.json
2023-11-18 11:33:39,096 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.select-object-content: calling handler <function add_generate_skeleton at 0x10c4a85e0>
2023-11-18 11:33:39,097 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api_select-object-content: calling handler <function add_waiters at 0x10c576ac0>
2023-11-18 11:33:39,097 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api_select-object-content: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x10c639110>>
2023-11-18 11:33:39,097 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.bucket: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,097 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,097 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'nitin-testing' for parameter "bucket": 'nitin-testing'
2023-11-18 11:33:39,098 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,098 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,098 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'airtravel.csv' for parameter "key": 'airtravel.csv'
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.sse-customer-algorithm: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.sse-customer-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.sse-customer-key-md5: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.expression: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,140 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'select * from s3object limit 5' for parameter "expression": 'select * from s3object limit 5'
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.expression-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,140 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,141 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'SQL' for parameter "expression_type": 'SQL'
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.request-progress: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.input-serialization: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,141 - MainThread - awscli.argprocess - DEBUG - Param input_serialization looks like JSON, not considered for param shorthand.
2023-11-18 11:33:39,141 - MainThread - awscli.arguments - DEBUG - Unpacked value of '{"CSV": {"FieldDelimiter": ",", "FileHeaderInfo": "USE"}, "CompressionType": "NONE"}' for parameter "input_serialization": OrderedDict([('CSV', OrderedDict([('FieldDelimiter', ','), ('FileHeaderInfo', 'USE')])), ('CompressionType', 'NONE')])
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.output-serialization: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.select-object-content: calling handler <awscli.argprocess.ParamShorthandParser object at 0x10bdc4d90>
2023-11-18 11:33:39,141 - MainThread - awscli.argprocess - DEBUG - Param output_serialization looks like JSON, not considered for param shorthand.
2023-11-18 11:33:39,141 - MainThread - awscli.arguments - DEBUG - Unpacked value of '{"CSV": {"FieldDelimiter": ",", "RecordDelimiter": "\n"}}' for parameter "output_serialization": OrderedDict([('CSV', OrderedDict([('FieldDelimiter', ','), ('RecordDelimiter', '\n')]))])
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.scan-range: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.expected-bucket-owner: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.select-object-content.outfile: calling handler <awscli.paramfile.URIArgumentHandler object at 0x10c66dad0>
2023-11-18 11:33:39,142 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-11-18 11:33:39,142 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-11-18 11:33:39,142 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-11-18 11:33:39,142 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-11-18 11:33:39,142 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2023-11-18 11:33:39,143 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2023-11-18 11:33:39,156 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x10ae9e480>
2023-11-18 11:33:39,186 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json
2023-11-18 11:33:39,189 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/partitions.json
2023-11-18 11:33:39,192 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x10adf2c00>
2023-11-18 11:33:39,192 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x10adf2980>
2023-11-18 11:33:39,193 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_service
2023-11-18 11:33:39,193 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: environment_global
2023-11-18 11:33:39,193 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_service
2023-11-18 11:33:39,193 - MainThread - botocore.configprovider - DEBUG - Looking for endpoint for s3 via: config_global
2023-11-18 11:33:39,193 - MainThread - botocore.configprovider - DEBUG - No configured endpoint found.
2023-11-18 11:33:39,246 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2023-11-18 11:33:39,247 - MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler
2023-11-18 11:33:39,247 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x10aebeb60>
2023-11-18 11:33:39,247 - MainThread - botocore.hooks - DEBUG - Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x10ce73d90>>
2023-11-18 11:33:39,247 - MainThread - botocore.regions - DEBUG - Calling endpoint provider with parameters: {'Bucket': 'nitin-testing', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-11-18 11:33:39,248 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://nitin-testing.s3.us-east-1.amazonaws.com
2023-11-18 11:33:39,248 - MainThread - botocore.regions - DEBUG - Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-11-18 11:33:39,248 - MainThread - botocore.regions - DEBUG - Selected auth type "v4" as "v4" with signing context params: {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}
2023-11-18 11:33:39,248 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.SelectObjectContent: calling handler <function base64_decode_input_blobs at 0x10c5f00e0>
2023-11-18 11:33:39,248 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.SelectObjectContent: calling handler <function validate_bucket_name at 0x10aebc900>
2023-11-18 11:33:39,248 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.SelectObjectContent: calling handler <function remove_bucket_from_url_paths_from_model at 0x10aebe980>
2023-11-18 11:33:39,248 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.SelectObjectContent: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x10ce73d90>>
2023-11-18 11:33:39,267 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.SelectObjectContent: calling handler <function generate_idempotent_uuid at 0x10aebc720>
2023-11-18 11:33:39,268 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.SelectObjectContent: calling handler <function add_expect_header at 0x10aebccc0>
2023-11-18 11:33:39,268 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.SelectObjectContent: calling handler <function inject_api_version_header_if_needed at 0x10aebe200>
2023-11-18 11:33:39,268 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=SelectObjectContent) with params: {'url_path': '/airtravel.csv?select&select-type=2', 'query_string': {}, 'method': 'POST', 'headers': {'User-Agent': 'aws-cli/2.13.7 Python/3.11.4 Darwin/22.6.0 exe/x86_64 prompt/off command/s3api.select-object-content'}, 'body': b'select * from s3object limit 5SQL,USENONE,\n', 'auth_path': '/nitin-testing/airtravel.csv?select&select-type=2', 'url': 'https://nitin-testing.s3.us-east-1.amazonaws.com/airtravel.csv?select&select-type=2', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x10ce61610>, 'has_streaming_input': False, 'auth_type': 'v4', 'signing': {'region': 'us-east-1', 'signing_name': 's3', 'disableDoubleEncoding': True}, 's3_redirect': {'redirected': False, 'bucket': 'nitin-testing', 'params': {'Bucket': 'nitin-testing', 'Key': 'airtravel.csv', 'Expression': 'select * from s3object limit 5', 'ExpressionType': 'SQL', 'InputSerialization': OrderedDict([('CSV', OrderedDict([('FieldDelimiter', ','), ('FileHeaderInfo', 'USE')])), ('CompressionType', 'NONE')]), 'OutputSerialization': OrderedDict([('CSV', OrderedDict([('FieldDelimiter', ','), ('RecordDelimiter', '\n')]))])}}}}
2023-11-18 11:33:39,268 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.SelectObjectContent: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10ce0ba90>>
2023-11-18 11:33:39,268 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.SelectObjectContent: calling handler <function set_operation_specific_signer at 0x10aebc5e0>
2023-11-18 11:33:39,268 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.SelectObjectContent: calling handler <function remove_arn_from_signing_path at 0x10aebeac0>
2023-11-18 11:33:39,269 - MainThread - botocore.credentials - DEBUG - Credentials for role retrieved from cache.
2023-11-18 11:33:39,269 - MainThread - botocore.credentials - DEBUG - Retrieved credentials will expire at: 2023-11-18 06:44:30+00:00
2023-11-18 11:33:39,269 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-11-18 11:33:39,269 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/airtravel.csv
select=&select-type=2
host:nitin-testing.s3.us-east-1.amazonaws.com
x-amz-content-sha256:REDACTED
x-amz-date:20231118T060339Z
x-amz-security-token:REDACTED

host;x-amz-content-sha256;x-amz-date;x-amz-security-token

2023-11-18 11:33:39,269 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20231118T060339Z
20231118/us-east-1/s3/aws4_request
REDACTED
2023-11-18 11:33:39,269 - MainThread - botocore.auth - DEBUG - Signature: REDACTED
2023-11-18 11:33:39,269 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=POST, url=https://nitin-testing.s3.us-east-1.amazonaws.com/airtravel.csv?select&select-type=2, headers={'User-Agent': b'aws-cli/2.13.7 Python/3.11.4 Darwin/22.6.0 exe/x86_64 prompt/off command/s3api.select-object-content', 'X-Amz-Date': b'20231118T060339Z', 'X-Amz-Security-Token': b'REDACTED', 'X-Amz-Content-SHA256': b'REDACTED', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20231118/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED', 'Content-Length': '481'}>
2023-11-18 11:33:39,270 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2023-11-18 11:33:39,270 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): nitin-testing.s3.us-east-1.amazonaws.com:443
2023-11-18 11:33:40,335 - MainThread - urllib3.connectionpool - DEBUG - https://nitin-testing.s3.us-east-1.amazonaws.com:443 "POST /airtravel.csv?select&select-type=2 HTTP/1.1" 200 None
2023-11-18 11:33:40,335 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '', 'x-amz-request-id': '1XJJ0VPCX66GP3V7', 'Date': 'Sat, 18 Nov 2023 06:03:41 GMT', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2023-11-18 11:33:40,335 - MainThread - botocore.parsers - DEBUG - Response body:
<urllib3.response.HTTPResponse object at 0x10ce1b670>
2023-11-18 11:33:40,335 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.SelectObjectContent: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x10ce74ad0>>
2023-11-18 11:33:40,336 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-11-18 11:33:40,336 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.SelectObjectContent: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x10ce73d90>>
2023-11-18 11:33:40,336 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.SelectObjectContent: calling handler <bound method S3SelectStreamOutputArgument.save_file of <awscli.customizations.s3events.S3SelectStreamOutputArgument object at 0x10c8ecf90>>
2023-11-18 11:33:40,337 - MainThread - botocore.parsers - DEBUG - Response headers: {':message-type': 'event', ':event-type': 'Records', ':content-type': 'application/octet-stream'}
2023-11-18 11:33:40,337 - MainThread - botocore.parsers - DEBUG - Response body:
b'Jack,McGinnis,220 hobo Av.,Phila, PA,09119\nJohn "Da Man",Repici,120 Jefferson St.,Riverside, NJ,08075\nStephen,Tyler,7452 Terrace "At the Plaza" road,SomeTown,SD, 91234\n,Blankman,,SomeTown, SD, 00298\n"Joan ""the bone"", Anne",Jet,"9th, at Terrace plc",Desert City,CO,00123\n'
2023-11-18 11:33:40,337 - MainThread - botocore.parsers - DEBUG - Response headers: {':message-type': 'event', ':event-type': 'Stats', ':content-type': 'text/xml'}
2023-11-18 11:33:40,337 - MainThread - botocore.parsers - DEBUG - Response body:
b'328328272'
2023-11-18 11:33:40,338 - MainThread - botocore.parsers - DEBUG - Response headers: {':message-type': 'event', ':event-type': 'End'}
2023-11-18 11:33:40,338 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2023-11-18 11:33:40,338 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.SelectObjectContent: calling handler <function enhance_error_msg at 0x10c574720>
2023-11-18 11:33:40,338 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.SelectObjectContent: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x10cdffad0>>
2023-11-18 11:33:40,339 - MainThread - awscli.formatter - DEBUG - RequestId: 1XJJ0VPCX66GP3V7

If you're testing with open data, a description of how you set up and populated the bucket so I can try stuff locally.

Well, my bucket is a private bucket. All I did was uploaded a small csv file and tried running s3 select API from a haskell script.

One difference I note between the request made by amazonka and aws cli is that aws s3api has Content-Lenght in the headers whereas amazonka does not. Could it be that it is used for calculating the signature?

[chreekat]

I'm also currently investigating a SignatureDoesNotMatch error, except I'm using Cloudflare's R2 and the request is a PutObject. The same code and command works fine with AWS S3!

I'll add a bit more data when I can.

[chreekat]

My problem is probably something altogether different, so I opened a different issue: #975