Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing filemods from processes with over 10,000 filemods in Cb Response #145

Open
nphull opened this issue Mar 29, 2019 · 3 comments
Open

Comments

@nphull
Copy link

nphull commented Mar 29, 2019

If I pull up a process in the Cb Response web GUI and it has over 10,000 filemods, and then I pull up the same process using cbapi, the cbapi Process claims it only has 10,000 filemods, and if I try enumerating them, I only get about 9,920 of them. What happened to all of the other filemods? How do I get them? Is this a limitation of the Response REST API?

Example: the web GUI shows process 0000cbf9-0000-1728-01d4-e63e96539a3a had 29969 filemods. So I pull it up with cbapi:

c = CbEnterpriseResponseAPI()
process = c.select(Process, '0000cbf9-0000-1728-01d4-e63e96539a3a')
print(process.filemod_count)
# 10000
fms = []
for fm in process.filemods:
    fms.append(fm)

print(len(fms))
# 9917

Why does it claim the process only had 10000 filemods? Why am I only able to get 9917 of those 10000? How do I get the other filemods?

The REST endpoint used to retrieve the filemods is /api/v4/process/0000cbf9-0000-1728-01d4-e63e96539a3a/0/event. What endpoint would I use to get the rest of the filemods? Changing the 0 at the end to a 1 gets a 404 Not Found.

Cb Response version: 6.2.4.190118.1044

@jgarman
Copy link
Contributor

jgarman commented May 7, 2019

What happens if you try using the all_filemods() method on the process? This may be an edge case that needs addressing

@nphull
Copy link
Author

nphull commented May 7, 2019

I get the exact same result using all_filemods() as iterating over process.filemods. For a process with 29,968 filemods, both approaches give me the same 9,465 filemods in the same order.

@mtscher
Copy link

mtscher commented Oct 9, 2021

I am curious to know if there was ever an update on this issue. I can consistently reproduce it on processes that contain more than 10k events. It seems to be avoidable by not selecting the process "by object" but rather using a direct query ("process_id: ") and then looping the returned processes and accumulating the events.

It looks to me like the issue is in the code for Process.require_all_events() method which (for recent versions) uses a single call to get the events with the default (0) process segment. I suspect this call is subject to the documented 10k limit indicated here where it indicates "There is a hard limit of 5k rows on the /devices API per call, and a hard limit of 10k rows on the /events API at maximum"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants