Skip to content

Home

Jump to bottom
Marcin Kozlowski edited this page Jul 1, 2024 · 25 revisions

Welcome to the betterscan-ce wiki!

What it does

It is a Code and Infrastructure (IaC) and Cloud-native Scanning/SAST/Static Analysis/Linting solution using many tools/Scanners with One Report. You can also add any tool to it. Currently, it supports many languages and tech stacks.

betterscan-concept

Fig. 1 Betterscan concept diagram

If you like it, please give it a GitHub star/fork/watch/contribute. This will ensure continuous development ⭐

Run this command in your code directory (checkout from Git - .git folder needs to be there, if you work with the normal directory, make the Git repo out of it git init && git add . && git commit etc):

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli.sh)

or for an HTML report:

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli-html.sh)

That's it. You just checked your code for 2,300+ Checks (Defects, Vulnerabilities, Best Practices, Secrets - 166+ secret types) and 4,000+ YARA rules for Antidebug, Antivm, Crypto, CVE, Exploits Kits, Malware, Web shells, and APTs.

FYI Above will maintain state via (.checkmate folder). Only new commits will be checked.

Use cases for you

This solutions should be part of DevOps/DevSecOps/Platform Engineering efforts. You can "shift left" your security efforts using below and providing tools to developers.

Asynchronous installation of the whole platform (Web Interface, workers etc)

That way you can setup asynchronous DevSecOps feedback for the repositories you set up to scan (daily, on commit etc.)

git clone [email protected]:tcosolutions/betterscan-ce.git
cd betterscan-ce/dockerhub
docker compose up

Local installation on your laptop

Just run for CLI:

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli.sh)

or for an HTML report:

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli-html.sh)

It will maintain state in .checkmate folder. First run can be longer, next should be very fast. Just re-run the same command on the next iteration.

Many other possibilities

Let me know your setup.

If you want to use the Platform, follow below:

Installation

Local

To install it, first:

Install docker and docker-compose

then:

2 options:

  1. Fastest method (use DockerHub built images). If unsure, use this.
git clone [email protected]:tcosolutions/betterscan-ce.git
cd betterscan-ce/dockerhub
docker compose up
  1. Slower method (build everything)
git clone [email protected]:tcosolutions/betterscan-ce.git
cd betterscan-ce/docker
docker compose up

Open up the Browser to:

http://localhost:5000

Sign up locally (and login in when needed)

GitLab Integration

It is possible to integrate results also in GitLab's Security & Compliance Dashboad. We can convert SARIF to GitLab format. GitLab Ultimate is required.

DefectDojo Integration

You can import via SARIF vulnerabilities and setup a pipeline in DefectDojo to manage vulnerabilities.

Azure DevOps Integration

To integrate BetterScan with Azure DevOps, you can do the following:

  1. Install the Azure DevOps SARIF SAST Scans Tab extension.
  2. Add this job to your azure-pipelines.yml:
  - job: SAST
    displayName: Static Application Security Test (SAST)
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
    pool:
      vmImage: 'ubuntu-latest'
    container: 'tcosolutions/betterscan-ce:worker-cli'
    steps:
    - script: |
        sudo apt-get update
        sudo apt-get install git-lfs
      displayName: Install git LFS
      
    - checkout: self
      persistCredentials: true

    - script: |
        set -e
        git config --global --add safe.directory $(Build.SourcesDirectory)
        git config --global user.email "[email protected]"
        git config --global user.name "Azure DevOps" 
        git checkout -b $(Build.SourceBranchName)
        sudo CODE_DIR=$(Build.SourcesDirectory) checkmate init
        sudo CODE_DIR=$(Build.SourcesDirectory) checkmate git init
        sudo CODE_DIR=$(Build.SourcesDirectory) checkmate git analyze --branch $(Build.SourceBranchName)
        sudo CODE_DIR=$(Build.SourcesDirectory) checkmate issues html
      displayName: Static Application Security Test (SAST)
      env:
        CODE_DIR: '$(Build.SourcesDirectory)'

    - task: PublishBuildArtifacts@1
      displayName: Publish SAST report
      inputs:
        PathtoPublish: $(Build.SourcesDirectory)
        ArtifactName: CodeAnalysisLogs

    - script: |
        git add .checkmate/db.sqlite
        git add report.html
        git commit -m '[ci skip] update checkmate db'
        git push origin $(Build.SourceBranchName):$(Build.SourceBranch)
      displayName: Commit and Push checkmate db

Platforms & OS'es

It is platform independent (Python). The "main" branch is for Linux x86_64. Some checkers are only available for x86_64

Linux (amd64)

Yes, by default

Usage

More info in the Wiki:

https://github.com/tcosolutions/betterscan-ce/wiki

Advanced Usage (CLI Mode only)

You can plug it anywhere on your CI/CD pipeline as a command.

With Betterscan one command, you add 6,300+ checks using different scanners (Betterscan is a "Meta" scanner in that sense, with supporting smart snapshots and other goodies. It is not just running tools always on full code)

Betterscan supports also CLI only mode, no Web Interface, worker etc. Run a binary in Docker in your own CI/CD pipeline (whatever it is) in Quality Gates that will output line by line (scanner and findings) on checkout code from Git (folder)

Quick Install

Plain CLI output

Just run this command (it will take care of everything):

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli.sh)

Corresponds to running these:

export CODE_DIR=${PWD}
cd $CODE_DIR
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git analyze --branch `git rev-parse --abbrev-ref HEAD`'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues'

HTML CLI output

Just run this command (it will take care of everything):

sh <(curl https://raw.githubusercontent.com/tcosolutions/betterscan-ce/main/cli-html.sh)

report will be in the directory under report.html

Corresponds to running these:

export CODE_DIR=${PWD}
cd $CODE_DIR
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate git analyze --branch `git rev-parse --abbrev-ref HEAD`'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v ${PWD}:${PWD}  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd $CODE_DIR && checkmate issues html'

Detailed explanation

Build Docker image Worker-CLI and run checkmate from there. Below is a sample flow:

CLI Mode only

  1. Clone the repo i.e into /tmp/samplecode
  2. set env var CODE_DIR i.e export CODE_DIR=/tmp/samplecode. This env var should point to your Git cloned repo dir. Set LIC env var.
  3. Cd into it (this is important!)
  4. Run checkmate init
  5. Run checkmate git init
  6. Run checkmate git analyze

Run checkmate issues

This will be shown

Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: checkov
Loading plugin: tfsec
Loading plugin: kubescape
Loading plugin: insidersecswift
Loading plugin: insiderseckotlin
Loading plugin: insiderseccsharp
Loading plugin: pmdapex
Loading plugin: semgrepccpp
Loading plugin: semgrepjava
Loading plugin: semgrepeslint
Loading plugin: graudit
Loading plugin: text4shell
Loading plugin: yara
Loading plugin: fluidattacksscanner
Loading plugin: gostaticcheck

There is a DockerHub image also for it ready.

docker pull tcosolutions/betterscan-ce:worker-cli

You can run the Commands with docker as below:

$ docker run -ti  tcosolutions/betterscan-ce:worker-cli checkmate
Loading plugin: git
Loading plugin: trufflehog3
Loading plugin: trojansource
Loading plugin: metrics
Loading plugin: bandit
Loading plugin: brakeman
Loading plugin: phpanalyzer
Loading plugin: pmd
Loading plugin: semgrep
Loading plugin: checkov
Loading plugin: tfsec
Loading plugin: kubescape
Loading plugin: insidersecswift
Loading plugin: insiderseckotlin
Loading plugin: insiderseccsharp
Loading plugin: pmdapex
Loading plugin: semgrepccpp
Loading plugin: semgrepjava
Loading plugin: semgrepeslint
Loading plugin: graudit
Loading plugin: text4shell
Loading plugin: yara
Loading plugin: fluidattacksscanner
Loading plugin: gostaticcheck
Usage: checkmate [command] [command] [...] [args]

Type "checkmate help" for help

Same workflow as above, but using Docker binary:

export CODE_DIR=/tmp/samplecode
cd /tmp/samplecode
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v /tmp/samplecode:/tmp/samplecode  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd /tmp/samplecode && checkmate init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v /tmp/samplecode:/tmp/samplecode  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd /tmp/samplecode && checkmate git init'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v /tmp/samplecode:/tmp/samplecode  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd /tmp/samplecode && checkmate git analyze'
docker run -e CODE_DIR -e LIC -e OPENAI_GPT_API -v /tmp/samplecode:/tmp/samplecode  -ti  tcosolutions/betterscan-ce:worker-cli /bin/sh -c 'cd /tmp/samplecode && checkmate issues'

Under the hood

Progpilot, PMD, Bandit, Brakeman, gostaticcheck, semgrep, trufflehog3, jshint, njsscan, log4shell via custom semgrep rule, checkov, kubescape, graudit, insidersec, flawfinder, find sec bugs, eslint, YARA, fluidattacks scanner, tfsec and other(s). Some were modified. See LICENSE for licensing and acknoweledgements.

Major features

Betterscan uses static analysis (semantic and graph analysis) to find bugs and defects.

  • Betterscan supports Cloud-native and Infrastructure Scanning
  • Betterscan supports secrets Scanning (166+ secret types)
  • Betterscan scans for 4,000 rules for Antidebug, Antivm, Crypto, CVE, Exploits Kits, Malware and Web shells, APTs
  • Betterscan can be extended with any tool producing JSON output (any binary, in any technology/language/product)
  • Betterscan is open Source
  • Betterscan supports scanning only changed files (differential analysis). You can store state in Database (PostgreSQL, MySQL/MariaDB, Oracle, Microsoft SQL Server) or in you Git repo.
  • Outputs in CLI, HTML, SARIF, JSON
  • Betterscan uses many tools adding up to 6,300+ checks which also semgrep as one of the tools

Welcome to Betterscan CE (Community Edition)!

Betterscan is based on QuantifedCode. QuantifiedCode is a code analysis & automation platform. It helps you to keep track of issues and metrics in your software projects, and can be easily extended to support new types of analyses.

The application consists of several parts:

  • A frontend, realized as a React.js app
  • A backend, realized as a Flask app
  • A background worker, realized using Celery, that performs the code analysis

Currently supports: PHP, Java, Scala, Python, PERL, Ruby, C, C++, Swift, Kotlin, Apex (Salesforce), GO, Infrastructure as a Code (IaC) Security and Best Practices (Docker, Kubernetes (k8s), Terraform AWS, GCP, Azure), Secret Scanning (166+ secret types), Trojan Source, Open Source and Proprietary Checks (total ca. 6,000+ checks). Checks for misconfigurations across all major (and some minor) cloud providers (AWS Checks, Azure Checks, GCP Checks, CloudStack Checks, DigitalOcean Checks, GitHub Checks, Kubernetes Checks, OpenStack Checks, Oracle Checks)

Advantages:

  • Many tools, one report (unification)
  • Dismiss, and collaborate on findings. Mark false-positives
  • Enable/disable each individual check in Checkers
  • ca. 6,300+ checks now (Linters, Static Code Analysis/Code Scanning, YARA ca. 4000 YARA binary matching/textual matching rules for Antidebug, Antivm, Crypto, CVE, Exploits Kits, Malware and Webshells, APTs )
  • any tool outputting JSON can be added
  • fast (checks only new code on recheck)
  • you can store state in Database (PostgreSQL, MySQL/MariaDB, Oracle, Microsoft SQL Server) or in you Git repo.
  • Outputs in CLI, HTML, SARIF, JSON.
  • Git support (HTTPS/TLS and SSH). For private repositories only SSH.
  • Swiss army knife tool/SIEM for Code Scanning
  • 100% Code transparency & full control of your code

Contribute

Contributing

Feel free to use, recommend improvements, or contribute to new implementations.

Check out our contributing guide to learn about our development process, how to suggest bugfixes and improvements.

Licensing

Betterscan's QuantifiedCode parts remain released under BSD-3 Clause License. However, modifications are released under AGPL-3.0 (see LICENSE in the repo for more details)

Installation

We provide several options for installing Betterscan. Which one is the right one for you depends on your use case.

  • The manual installation is best if you want to modify or change Betterscan
  • The Docker-based installation is probably the easiest way to try Betterscan without much work

The following section will only discuss the manual installation process. for the other options, please check their corresponding repositories.

Manual Installation

The installation consists of three parts:

  • Install the dependencies required to run Betterscan
  • Download the required source code
  • Set up the configuration

Installing Dependencies

Betterscan requires the following external dependencies:

  • A message broker (required for the background tasks message queue). We recommend either RabbitMQ or Redis.
  • A database (required for the core application). We recommend PostgreSQL, but SQLite is supported as well. Other database systems might work too (e.g. MySQL), but are currently not officially supported.

Download the Betterscan CE source code

Now with the dependencies installed, we can go ahead and download Betterscan:

    git clone [email protected]:tcosolutions/betterscan-ce.git

Install the required Python packages

Betterscan CE manages dependencies via the Python package manager, pip.

Edit Settings

Betterscan gets configured via YAML settings files. When starting up the application, it incrementally loads settings from several files, recursively updating the settings object. First, it will load default settings from quantifiedcode/settings/default.yml. Then, it will check if a QC_SETTINGS environment variable is defined and points to a valid file, and if so it will load settings from it (possibly overwriting default settings). If not, it will look for a settings.yml file in the current working directory and load settings from there. Additionally, it will check if a QC_SECRETS environment variable is defined and points to a valid file, and also load settings from there

There is a sample settings.yml file in the root of the repository that you can start from.

Running the Setup

After editing your settings, run the setup command via

    #run from the root directory of the repository
    python manage.py setup

The setup assistant will iteratively walk you through the setup, and when finished you should have a working instance of Betterscan!

Running the web application

To run the web application, simply run

    python manage.py runserver

Running the background worker

To run the background worker, simply run

    python manage.py runworker

Docker-Based Installation

See docker folder

© Copyright 2024 Betterscan.io All rights reserved.

Try it yourself!

Clone this wiki locally