Incorrect Handling of Permissions Can Result in Data Loss

[asmith-elastic]

Description

Due to a misconfiguration of Elasticsearch role permissions, an incorrect index pattern can lead to data loss during ingestion, with ESF erroneously logging a successful HTTP 200 status instead of the actual failure.

Version: 1.13.0

Steps to reproduce

Configure permissions allowing write access to a specific index pattern (e.g., logs-app1*) with privileges such as create_doc and auto_configure.

• Incorrectly modify the permissions by changing the index pattern to an invalid one (e.g., logs-app2*), which does not correspond to an existing index.
• Send bulk ingest requests to ESF, targeting documents at the original (correct) index pattern.
• Review the logs which show an HTTP 200 status followed by a message indicating a high count of failed documents.

Expected Behavior

When sending ingest requests to an index that does not match the configured index pattern, an appropriate error response (e.g., HTTP 403) is expected, and documents should be routed to a replay queue rather than being lost.

Actual Behavior

Despite modifying permissions to an incorrect index pattern, ESF logs report an HTTP 200 status after ingest requests. In reality, the documents fail to index, and subsequent logs report the number of ingestion failures. These documents do not enter a replay queue and are effectively lost.

Initial logs showing a 200 status for the bulk request:

{
  "@timestamp": "2024-03-08T12:11:11.120Z",
  "log.level": "info",
  "message": "POST .../_bulk [status:200 request:0.026s]",
  ...
}

Logs directly following the above 200 status, showing a warning for ingestion failures:

{
  "@timestamp": "2024-03-08T12:11:12.120Z",
  "log.level": "warning",
  "failed": 110,
  "success": 0,
  ...
}

Additional notes

This issue does not appear to occur when removing permissions e.g., removing the create_doc permission. When the create_doc permission is removed from the user's role it results in an expected 403 error and routes failed documents to a replay queue. This behavior differs from changing the index pattern where documents are lost, suggesting an issue in handling the latter scenario.

{
    "@timestamp": "2024-03-07T18:35:01.683Z",
    "log.level": "warning",
    "_id": "1234",
    "error": "AuthorizationException(403, 'security_exception', 'action [indices:data/write/bulk] is unauthorized for API key id [xxxx] of user [xxx], this action is granted by the index privileges [create_doc,create,delete,index,write,all]')",
}