Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Living off the Land Detection - Update supported data [Investigation] #9359

Closed
1 of 10 tasks
susan-shu-c opened this issue Mar 13, 2024 · 11 comments
Closed
1 of 10 tasks
Assignees

Comments

@susan-shu-c
Copy link
Member

susan-shu-c commented Mar 13, 2024

Overview

For the ProblemChild (Living off the land detection) integration package, look into support for Elastic agent - Windows, aka Windows agent integration.

Currently the package explicitly support winlogbeat and Elastic Defend.

  • Elastic Defend user.id is inconsistent between string and int, while the package expects string, leading to errors when the user.id type is int (see discussion below)
  • [Edit: infeasible at this time due to observed differences in packages] Windows agent integration coverage is same if not more than winlogbeat, and we should consider testing the indices the package looks at to support integrations including (but not limited to) agent. This includes testing, updating documentation wording, and editing the package if needed.

[Update] Found differences in featurization and PowerShell records (see more info); so will not claim to support Windows Agent until we understand more.

Integration release checklist

All changes

  • Change follows the contributing guidelines
  • Supported versions of the monitoring target are documented
  • Supported operating systems are documented (if applicable)
  • Integration or System tests exist
  • Documentation exists, useful guidelines to follow
  • Fields follow ECS and naming conventions
  • At least a manual test with ES / Kibana / Agent has been performed.
  • Required Kibana version set to:
@jmcarlock
Copy link
Contributor

jmcarlock commented May 15, 2024

Elastic Defend user.id is inconsistent between string and int, while the package expects string, leading to errors when the user.id type is int

After following up with @susan-shu-c and @sodhikirti07 it turns out this isn't an issue. Elastic Defend provides user.id of int on POSIX and user.id of keyword on Windows. The failed simulation was due to the ingest pipeline filtering this out as a POSIX record, as the ingest pipeline for LotL/ProblemChild (shown below unrolled for clarity) only supports Windows records.

description: "Pipelines for ProblemChild detection"
processors:
  - pipeline:
      if: ctx.containsKey('event') && 
      ctx['event'].containsKey('kind') && 
      ctx['event'].containsKey('category') && 
      ctx['event']['kind'] == 'event' && 
      ctx['event']['category'].contains('process') && 
      ctx.containsKey('host') &&
      ctx['host'].containsKey('os') &&
    (ctx['host']['os'].containsKey('type') || 
        ctx['host']['os'].containsKey('family') ||
        ctx['host']['os'].containsKey('platform')) && 
    (ctx['host']['os']['type'] == 'windows' || 
        ctx['host']['os']['type'] == 'Windows' ||
        ctx['host']['os']['family'] == 'windows' || 
        ctx['host']['os']['family'] == 'Windows' || 
        ctx['host']['os']['platform'] == 'windows' || 
        ctx['host']['os']['platform'] == 'Windows')

@sodhikirti07 verified that Elastic Defend was working with the integration.

Windows agent integration coverage is same if not more than winlogbeat, and we should consider testing the indices the package looks at to support integrations including (but not limited to) agent. This includes testing, updating documentation wording, and editing the package if needed.

The testing pipeline portion of this task would be covered by:
* https://github.com/elastic/security-team/issues/8774

I would suggest closing this issue. (Edits: see below)

@susan-shu-c
Copy link
Member Author

Thanks for testing!

As for updating documentation wording, it seems that we can say that we support Windows Agent then as part of this ticket. Or, do you mean you want to install Windows Agent separately and test end to end as part of https://github.com/elastic/security-team/issues/8774 ? (Regardless, saying we support Windows Agent should be outside of https://github.com/elastic/security-team/issues/8774

@jmcarlock
Copy link
Contributor

@susan-shu-c That's a good point, the agent integrations aren't mentioned in https://github.com/elastic/security-team/issues/8774, only checking if a package conforms to ECS. So you're right, it wouldn't fall under #8774.

This ticket is a good place to start for testing agent integrations, along the way I would suggest writing some team documentation on this process for repeatablilty.

After this we could also check how Windows Agent affects any other EA integration packages. (It provides DNS records so it would probably support DGA, for instance.)

Lastly, we could consider providing system tests for the common data stream providers (IE Windows Agent, Elastic Defend, winlogbeat) for all of the EA packages.

@susan-shu-c susan-shu-c changed the title Living off the Land Detection - Update supported data Living off the Land Detection - Update supported data (documentation) Jun 6, 2024
@jmcarlock
Copy link
Contributor

jmcarlock commented Jun 24, 2024

It was difficult to get the Elastic Windows Agent up and running. Thank you @sodhikirti07 for the help today and Friday! A couple of the older guides no longer work for this purpose. Here is a list of everything I tried:

  1. Install an Agent and Endpoint using Parallels - In this guide you setup a Kibana dev environment with the latest snapshot on your local machine and connect the Parallels Windows VM to that. It no longer works on step 12.
  2. Tried substituting the Kibana dev environment for elastic-package using Creating, testing and publishing Fleet integration packages. It no longer works on 3a, and there is an open issue add ability to set publish ports to config.yml elastic-package#1262. I was able to get the agent port open using Add support to define expose ports for independent agents elastic-package#1795, but not the one needed for fleet.
  3. I looked into running the Kibana dev environment on the Windows using Parallels. However, this needed to use another layer of virtualization and seemed overly complicated for what I was trying to do.
  4. I was able to get Windows Agent in a ARM VM sending some data to Elastic Cloud. I am looking into if the correct fields are being sent.

If the approach in 4 send the correct data there are two other things I can try:

  1. If the Windows VM doesn't work for the Windows Agent, I will look at starting a Windows instance in the cloud and connecting it to Elastic Cloud.
  2. I believe it is possible to use number 2 above to install an agent without Fleet.

@susan-shu-c
Copy link
Member Author

susan-shu-c commented Jun 25, 2024

Thanks for sharing these points!

Notes from meeting:

@jmcarlock
Copy link
Contributor

jmcarlock commented Jun 26, 2024

I was able to gather some sample logs from the Windows integration, along with a log to compare with from Elastic Defend.

I confirmed that neither sysmon or PowerShell in the Windows Integration work with the inference pipeline in LotL as is, but I believe they have the appropriate data. I will verify this and investigate what changes need to be made to support them.

@jmcarlock
Copy link
Contributor

jmcarlock commented Jun 27, 2024

The inference pipeline of Living off the Land works fine with the Windows integration using sysmon. Looking into updates to the documentation.

@jmcarlock
Copy link
Contributor

jmcarlock commented Jun 27, 2024

@susan-shu-c For one last sanity check, I compared the three results from the different records above and now I'm not so sure this is a good idea without further investigation. In all three tests I launched powershell.exe Start-Process notepad.exe from cmd.exe. I believe change may cause an increase in false positives, but I'm not sure what the underlying cause would be. Elastic Defend properly said this was not a positive prediction, while the Windows Integration logs were predicted positive.

Elastic Defend
"prediction": "0",
"prediction_probability": 0.9968744314457992

Windows Integration - Sysmon
"prediction": "1",
"prediction_probability": 0.9862823685565043

Windows Integration - PowerShell
"prediction": "1",
"prediction_probability": 0.4493628952893507

Edit: see below

@jmcarlock
Copy link
Contributor

jmcarlock commented Jul 1, 2024

Added a more rigorous test (under "Test on live data from Elastic integrations") and I was able to collect the results from the same run of powershell.exe Start-Process notepad.exe.

index feature_command_line feature_elevated feature_elevation_type feature_ends_with_exe feature_event_action feature_filename_mismatch feature_integrity_level feature_normalized_ppath feature_process_executable feature_process_name feature_process_parent_executable feature_process_parent_name feature_process_pe_original_file_name feature_process_signed feature_process_signer_trusted feature_running_as_system feature_user_id prediction prediction_score model_id prediction_probability query index agent.type timestamp
0 powershell.exe start-process notepad.exe FALSE unknown TRUE Process creation TRUE High win_system_dir c:\win_system_dir\windowspowershell\v1.0\power... powershell.exe c:\win_system_dir\cmd.exe cmd.exe PowerShell.EXE FALSE FALSE TRUE S-1-5-18 0 0.057775 problemchild_20210526_1.0 0.996874 logs-windows.sysmon* .ds-logs-windows.sysmon_operational-default-20... filebeat 2024-07-01T17:43:05.499Z
1 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.643Z
2 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.645Z
3 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.645Z
4 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.649Z
5 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.653Z
6 powershell.exe start-process notepad.exe FALSE unknown FALSE Provider Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.670Z
7 powershell.exe start-process notepad.exe FALSE unknown FALSE Engine Lifecycle FALSE unknown unknown unknown unknown unknown unknown unknown FALSE FALSE FALSE unknown 0 0.054858 problemchild_20210526_1.0 0.946549 logs-windows.powershell* .ds-logs-windows.powershell-default-2024.06.24... filebeat 2024-07-01T17:43:09.709Z
8 powershell.exe start-process notepad.exe FALSE unknown TRUE start TRUE high win_system_dir c:\win_system_dir\windowspowershell\v1.0\power... powershell.exe c:\win_system_dir\cmd.exe cmd.exe PowerShell.EXE TRUE TRUE FALSE S-1-5-21-3261892665-114283024-1478164545-1000 0 0.057775 problemchild_20210526_1.0 0.996874 logs-endpoint* .ds-logs-endpoint.events.process-default-2024.... endpoint 2024-07-01T17:43:05.4994991Z

Now the predictions are returning similar results, however there are still two differences:

  • Featurization is different for the Windows Integration with sysmon and PowerShell.
  • The PowerShell part of the integration is significantly different and returns multiple records when running a command.

@susan-shu-c
Copy link
Member Author

Hi Gus, so it seems like we can't simply claim that it'll work with Windows Agent (Elastic Agent) due to some of these differences. Thanks a lot for finding them!

I think we can close this ticket for now.

Next steps: I'm currently reaching out to people working on Windows Agent to see who I can track down to answer if these differences are intentional; and what we need to keep in mind if we go down the route of supporting Windows Agent.

I'll keep you in the loop once I get any info.

@susan-shu-c susan-shu-c changed the title Living off the Land Detection - Update supported data (documentation) Living off the Land Detection - Update supported data [Investigation] Jul 2, 2024
@jmcarlock
Copy link
Contributor

Thanks @susan-shu-c !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants