Hybrid Analysis - error handling around max API hits #285

MarcOverIP · 2022-12-05T12:09:54Z

Error handling with Hybrid Analysis fails in some cases:

HTTP error code 525 as reported by daemon.log. This means a TLS error. I believe this is reported by Cloudflare. See also the full HTTP webpage that is reported in the daemon.log as stated below
HTTP error code 429 as reported by daemon.log. This means Too many requests, rate limited. Im not sure if the quota checking actually understands and adjusts accordingly. Given the amount of errors on this in my test setup, I tend to believe that ioc_ybridanalysis.py just yolo checks any amount of hashes and hopes for the best.

1. Output from daemon.log on SSL issue:

<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>www.hybrid-analysis.com | 525: SSL handshake failed</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" />


</head>
<body>
<div id="cf-wrapper">



    <div id="cf-error-details" class="p-0">
        <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">
            <h1 class="inline-block sm:block sm:mb-2 font-light text-60 lg:text-4xl text-black-dark leading-tight mr-2">

              <span class="inline-block">SSL handshake failed</span>
              <span class="code-label">Error code 525</span>
            </h1>
            <div>
               Visit <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">cloudflare.com</a> for more information.
            </div>
            <div class="mt-3">2022-12-05 12:00:16 UTC</div>
        </header>

        <div class="my-8 bg-gradient-gray">
            <div class="w-240 lg:w-full mx-auto">
                <div class="clearfix md:px-8">

<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-browser block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">You</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Browser

  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">
    <span class="cf-icon-cloud block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
    </a>
  </div>
  <span class="md:block w-full truncate">Amsterdam</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
    <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" target="_blank" rel="noopener noreferrer">
    Cloudflare
    </a>
  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-server block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">www.hybrid-analysis.com</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Host

  </h3>
  <span class="leading-1.3 text-2xl text-red-error">Error</span>
</div>

                </div>

            </div>
        </div>

        <div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
            <div class="clearfix">
                <div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What happened?</h2>
                    <p>Cloudflare is unable to establish an SSL connection to the origin server.</p>
                </div>

                <div class="w-1/2 md:w-full float-left leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What can I do?</h2>
                          <h3 class="text-15 font-semibold mb-2">If you're a visitor of this website:</h3>
      <p class="mb-6">Please try again in a few minutes.</p>

      <h3 class="text-15 font-semibold mb-2">If you're the owner of this website:</h3>
      <p><span>It appears that the SSL configuration used is not compatible with Cloudflare. This could happen for a several reasons, including no shared cipher suites.</span> <a rel="noopener noreferrer" href="https://support.cloudflare.com/hc/en-us/articles/200278659">Additional troubleshooting information here.</a></p>
                </div>
            </div>

        </div>

        <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">774c9dcd9b650ae0</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">188.166.47.245</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_525&utm_campaign=www.hybrid-analysis.com" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->


    </div>
</div>
</body>
</html>```

The text was updated successfully, but these errors were encountered:

fastlorenzo · 2022-12-05T19:41:46Z

Normally this is tackled by first calling HA to request the remaining quota amount:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_hybridanalysis.py

Lines 33 to 73 in 5f50fc6

    
           def get_remaining_quota(self): 
        
               """Returns the number of hashes that could be queried within this run""" 
        
               url = "https://www.hybrid-analysis.com/api/v2/key/current" 
        
               headers = { 
        
                   "Accept": "application/json", 
        
                   "User-Agent": "RedELK", 
        
                   "api-key": self.api_key, 
        
               } 
        
               # Get the quotas, if response code != 200, return 0 so we don't query further 
        
               response = requests.get(url, headers=headers) 
        
               if response.status_code != 200: 
        
                   self.logger.warning( 
        
                       "Error retrieving Hybrid Analysis Quota (HTTP Status code: %d)", 
        
                       response.status_code, 
        
                   ) 
        
                   return 0 
        
               api_limits_json = response.headers.get("api-limits") 
        
               api_limits = json.loads(api_limits_json) 
        
               # First check if the limit has been reached 
        
               limit_reached = get_value("limit_reached", api_limits, False) 
        
               if limit_reached: 
        
                   return 0 
        
               # Extract the limits and usage 
        
               limits_minute = get_value("limits.minute", api_limits, 0) 
        
               limits_hour = get_value("limits.hour", api_limits, 0) 
        
               used_minute = get_value("used.minute", api_limits, 0) 
        
               used_hour = get_value("used.hour", api_limits, 0) 
        
               remaining_minute = limits_minute - used_minute 
        
               remaining_hour = limits_hour - used_hour 
        
               self.logger.debug( 
        
                   "Remaining quotas: hour(%d) / minute(%d)", remaining_hour, remaining_minute 
        
               ) 
        
               # Return the remaining quota per minute 
        
               return remaining_minute

It should still check some of the hashes and stop when the limit is reached:

RedELK/elkserver/docker/redelk-base/redelkinstalldata/scripts/modules/alarm_filehash/ioc_hybridanalysis.py

Lines 104 to 113 in 5f50fc6

    
           # Get the remaining quota for this run 
        
           remaining_quota = self.get_remaining_quota() 
        
           ha_results = {} 
        
           # Query HA API for file hashes 
        
           count = 0 
        
           for md5 in hash_list: 
        
               if count < remaining_quota: 
        
                   # Within quota, let's check the file hash with HA 
        
                   ha_result = self.get_ha_file_results(md5)

MarcOverIP assigned fastlorenzo Dec 5, 2022

MarcOverIP changed the title ~~Hybrid Analysis - daemon.py crashes when API max rate was hit~~ Hybrid Analysis - error handling around max API hits Dec 5, 2022

MarcOverIP mentioned this issue Dec 5, 2022

Hybrid Anlaysis - SSL handshake failure #286

Closed

fastlorenzo added this to the v2.0.0-beta.6 milestone Feb 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hybrid Analysis - error handling around max API hits #285

Hybrid Analysis - error handling around max API hits #285

MarcOverIP commented Dec 5, 2022 •

edited

Loading

fastlorenzo commented Dec 5, 2022

Hybrid Analysis - error handling around max API hits #285

Hybrid Analysis - error handling around max API hits #285

Comments

MarcOverIP commented Dec 5, 2022 • edited Loading

fastlorenzo commented Dec 5, 2022

MarcOverIP commented Dec 5, 2022 •

edited

Loading