Opensearch 1.3 error "invalid character 'U' looking for beginning of value"

[asepulcre]

Hi,

Recently we've updated our AWS Opensearch cluster from 1.2 to 1.3, we've been using your provider to manage users, policies, destinations and monitors, but with the update an error started to show up whenever we do a terraform plan or apply:

│ Error: invalid character 'U' looking for beginning of value
│ 
│   with elasticsearch_opensearch_ism_policy.policies["delete_3d_policy"],
│   on policies.tf line 1, in resource "elasticsearch_opensearch_ism_policy" "policies":
│    1: resource "elasticsearch_opensearch_ism_policy" "policies" {

This is the resource configuration :

resource "elasticsearch_opensearch_ism_policy" "policies" {
  for_each  = fileset("${path.module}/policies", "*")
  policy_id = each.key
  body      = file("${path.module}/policies/${each.key}")
}

and an example of policy:

{
    "policy": {
        "description": "Delete index after 3 days.",
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template":
        [
            {
                "index_patterns":
                [
                    "metricbeat-*",
                    "filebeat-*"
                ],
                "priority": 100
            }
        ]
    }
}

terraform config:

terraform {

  # providers
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.2.0"
    }
    elasticsearch = {
      source  = "phillbaker/elasticsearch"
      version = "2.0.4"
    }

    sops = {
      source  = "carlpett/sops"
      version = "~> 0.5"
    }
  }

  # terraform version
  required_version = ">= 0.15.5"
}

This error seems to occur whenever a json is involved, managing users works fine with version 1.3, but destinations, policies and monitors do not.

Any help is greatly appreciated, thanks in advance.

[raphael-caylent]

I'm getting the same error with the elasticsearch_opensearch_roles_mapping resource. I'm running OpenSearch version 1.1 and the provider version is 2.0.4.

This is an example resource definition:

resource "elasticsearch_opensearch_roles_mapping" "all_access" {
  provider    = elasticsearch.main
  role_name   = "all_access"
  description = "Mapping XXX to OS role"
  backend_roles = [
    "V4 UUID STRING",
    "arn:aws:iam::ACCOUNT_ID:role/aws-reserved/sso.amazonaws.com/REGION/AWSReservedSSO_XXX-XXXXXX-XXXX_random",
    "arn:aws:iam::ACCOUNT_ID:role/foo-bar",
  ]
  users = ["admin"]
}

Error:

[INFO]  provider.terraform-provider-elasticsearch_v2.0.4: [INFO] Failed to put role mapping: invalid character '<' looking for beginning of value:
[TRACE] provider.terraform-provider-elasticsearch_v2.0.4: Called downstream: tf_provider_addr=provider tf_req_id=suppressed tf_rpc=ApplyResourceChange @module=sdk.proto tf_proto_version=5 tf_resource_type=elasticsearch_opensearch_roles_mapping @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:608
[TRACE] provider.terraform-provider-elasticsearch_v2.0.4: Served request: @module=sdk.proto tf_proto_version=5 tf_req_id=suppressed tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:614 tf_provider_addr=provider tf_resource_type=elasticsearch_opensearch_roles_mapping
╷
│ Error: invalid character '<' looking for beginning of value
│ 
│   with elasticsearch_opensearch_roles_mapping.all_access,
│   on opensearch.tf line 164, in resource "elasticsearch_opensearch_roles_mapping" "all_access":
│  164: resource "elasticsearch_opensearch_roles_mapping" "all_access" {
│

[raphael-caylent]

I ran the provider in debug mode and was able to find the issue (kinda). From my tests, the issue seems to be because it is only able to handle responses in JSON. In my case, I have a proxy in front of OpenSearch which was down, so making calls to OpenSearch (proxy) was returning a 404 from Nginx (which returns a text/html) this way broking the JSON parse (hence why my error message contains a mention to a < character).
In your case @asepulcre I think your error might be an issue with credentials. I say this because when I was trying to configure the debug mode I got an Unauthorized from OpenSearch which in terraform was shown exactly like the error you got: Error: invalid character 'U' looking for beginning of value
If that's not the problem, I suggest you run the provider locally in debug mode and try to find the full error message.

[asepulcre]

Connection and credentials with the opensearch cluster work fine, because when we manage users using the provider it works without issue.

I've tried to do a terraform plan with a clean tfstate and the plan works, it is when we try to do an apply that it still breaks with the aforementioned error.

Sadly, running the provider in debug mode doesn't provide a more specific error:

{"@level":"info","@message":"2022/09/15 09:58:49 [INFO] Failed to create OpenDistroPolicy: invalid character 'U' looking for beginning of value","@timestamp":"2022-09-15T09:58:49.708740+02:00"}

[raphael-caylent]

@asepulcre you will need to run the provider in debug mode and attach a debugger to the process in order to use some breakpoints and check the variables (I know it's not an easy process but I think it's the only one in this case). What you want to find is the HTTP response that is breaking the JSON parse and thus throwing this error you see.

[psarun27]

Anyone able to find resolution for this issue? I am unable to run my build in debug mode. Below is my code. I was able to do the same when I use internal master user as the admin user but getting this error when iam role is the master user.

provider "elasticsearch" {
url = "https://xxx.region.es.amazonaws.com/"
healthcheck = false
#sniff = false
sign_aws_requests = false
aws_assume_role_arn = "arn:aws:iam::account#:role/rolename"
}

resource "elasticsearch_opensearch_roles_mapping" "mapper" {
role_name = "readall"
description = "Mapping AWS IAM roles to ES role"
backend_roles = [
"arn:aws:iam::account#:role/rolename1"
]
}

"Error: invalid character 'A' looking for beginning of value" - Error in the role mapping block.

[raphael-caylent]

@psarun27 I think the overall issue which we all faced is that there's an error happening (a different one for each of us) and this error is not being returned in JSON, which is breaking the parsing from the provider and thus returning these generic errors. Unless there's a change on this provider to handle these cases (@phillbaker is that possible? I'm available to help if so) the only way to identify what is the real error you are facing is to run the provider in debug mode and find the HTTP response yourself. Note that I'm saying running the provider in debug mode, not terraform.

[psarun27]

Thank you, @raphael-caylent. Has anyone been able to create a user, role or do role mapping using this provider with an iam role as the master user. I am able to create user, role, and do role mapping if the master user is a internal database user but no luck if the master user is an iam role. My provider block is below. Am I missing anything here? Or there special permissions or configuration needed for the iam role?

provider "elasticsearch" {
url = "https://xxx.region.es.amazonaws.com/"
healthcheck = false #(tried with true as well with no luck)
sign_aws_requests = false #(tried with true as well with no luck)
aws_assume_role_arn = "arn:aws:iam::account#:role/rolename"
}

Another question (somewhat related to the above question), please. I can run a below CURL command using a username/password (shown in bold below). How do I run the same command if the master user is an iam role?

curl -XPUT "https://xxx.region.es.amazonaws.com/_plugins/_security/api/roles/newrole" -u "un:pwd" -k -H "Content-Type: application/json" -d {JSON}

[psarun27]

Looks like there is a newer version (2.0.5) which I used and got a detailed error message (helpful, thanks to all who contributed). Below is the recent error. Thoughts? @phillbaker @raphael-caylent
User: arn:aws:sts::accountnumber:assumed-role/rolename/324234254525 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::accountnumber:role/rolename/Message>

Error: NoCredentialProviders: no valid providers in chain. Deprecated.
��� For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[raphael-caylent]

@psarun27

Has anyone been able to create a user, role or do role mapping using this provider with an iam role as the master user

I think in the beginning when I was trying to use this provider I tried to use an IAM Role but didn't work out so I switched to an internal user.

How do I run the same command if the master user is an iam role?

That's a good question, I need to think a bit on this one. But I guess what you could do is use a programming language that will give you a library to sign the request. Check this example. In this case, you could run your code in a Lambda that has the master role attached to it.

Thoughts?

Maybe your TF role doesn't have permission to assume the master role? I would need more context on what arn:aws:sts::accountnumber:assumed-role/rolename/ is to have a better idea of what could be the error.

[isarns]

Hey, I got the same Error NoCredentialProviders: no valid providers in chain. Deprecated.
I was assuming the master role arn:aws:iam::12345666:role/master while running terragrunt apply
then got the error I mention above.

After some careful log reading, I noticed that User: arn:aws:iam::12345666:assumed-role/master/me is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::12345666:role/master

So I just went and added to arn:aws:iam::12345666:role/master another trust relation for itself.

Fixed the issue for me.