Unperformant test execution after plugin implementation

[bdimitrovski]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I acknowledge that I will attach a full debug log, otherwise the issue will be closed with no response.

Environment information

System:
    OS: macOS 12.6.1
    CPU: (8) arm64 Apple M1
    Memory: 91.06 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.15.1 - ~/.nvm/versions/node/v16.15.1/bin/node
    npm: 8.3.0 - ~/.nvm/versions/node/v16.15.1/bin/npm
    pnpm: 8.7.6 - ~/.nvm/versions/node/v16.15.1/bin/pnpm
  Browsers:
    Chrome: 119.0.6045.159
    Edge: 111.0.1661.62
    Safari: 15.6.1
  npmPackages:
    @cypress/angular:  0.0.0-development
    @cypress/grep: ^4.0.0 => 4.0.0
    @cypress/mount-utils:  0.0.0-development
    @cypress/react:  0.0.0-development
    @cypress/react18:  0.0.0-development
    @cypress/svelte:  0.0.0-development
    @cypress/vue:  0.0.0-development
    @cypress/vue2:  0.0.0-development
    @faker-js/faker: ^8.1.0 => 8.1.0
    @types/node: ^20.7.0 => 20.7.0 (18.18.6)
    @typescript-eslint/eslint-plugin: ^6.7.3 => 6.7.3
    @typescript-eslint/parser: ^6.7.3 => 6.7.3
    cypress: ^13.6.0 => 13.6.0
    cypress-debugger: ^1.0.9 => 1.0.9
    cypress-mochawesome-reporter: ^3.6.0 => 3.6.0
    cypress-multi-reporters: ^1.6.3 => 1.6.3
    cypress-terminal-report: ^5.3.6 => 5.3.6
    dayjs: ^1.11.10 => 1.11.10
    eslint: ^8.50.0 => 8.50.0
    faker: ^6.6.6 => 6.6.6
    mocha-junit-reporter: ^2.2.1 => 2.2.1
    rimraf: ^5.0.5 => 5.0.5 (3.0.2)
  npmGlobalPackages:
    corepack: 0.10.0
    json: 11.0.0
    npm: 8.3.0
    pnpm: 8.7.6

Describe the bug

I encounter almost crippling performance issues while using this plugin. We are using Nuxt version 3.7.3 for the frontend and Vite as the build system.

There is nothing special from plugins which I use, in fact, I have tried disabling all of them except cypress-debugger, yet still I have seen the tests execute at least 3x longer than they usually would.

I have tried, apart from the above, to upgrade Cypress to latest version (13.6.0), and to run the tests in headless Edge browser too.

Expected behavior

The tests should execute in a reasonable timeframe of about 6-7 minutes, which they usually do.

With the plugin enabled, this is the execution time:

Command and Setup

Command (share the exact cypress or cypress-cloud command you're running)

npx cypress run --headless --browser chrome --reporter cypress-multi-reporters

Setup files cypress.config.j|ts|json

cypress.config.js

const { defineConfig } = require('cypress');
const { debuggerPlugin } = require('cypress-debugger');

module.exports = defineConfig({
  reporter: 'cypress-multi-reporters',
  reporterOptions: {
    configFile: 'reporter-config.json'
  },
  e2e: {
    responseTimeout: 30000,
    requestTimeout: 30000,
    viewportWidth: 1920,
    viewportHeight: 1080,
    baseUrl: "***",
    supportFile: "cypress/support/e2e.js",
    video: false,
    trashAssetsBeforeRuns: true,
    retries: {
      runMode: 1
    },
    env: {
      username: "***",
      password: "***",
      apiBaseUrl: "***"
    },
    specPattern: "cypress/e2e/**/*.spec.ts",
    setupNodeEvents(on, config) {
      debuggerPlugin(on, config, {
        meta: {
          key: 'value',
        },
        failedTestsOnly: true,
        callback: (path, data) => {
          console.log({
            path,
            data,
          });
        },
      })

      const loggerOptions = {
        outputRoot: `${config.projectRoot}/cypress/reports/logs`,
        outputTarget: {
          'verbose.log': 'txt',
          'verbose.json': 'json'
        }
      }

      require('cypress-mochawesome-reporter/plugin')(on);
      require('cypress-terminal-report/src/installLogsPrinter')(on, loggerOptions);

      on('task', {
        getDownload: () => {
          const downloadsFolder = config['downloadsFolder'];

          if (!fs.existsSync(downloadsFolder)) {
            fs.mkdirSync(downloadsFolder);
          }

          return new Promise((resolve, reject) => {
            const watcher = fs.watch(downloadsFolder, (eventType, filename) => {
              if (eventType === 'rename' && !filename.endsWith('.crdownload') && !filename.endsWith('.part')) {
                resolve(filename);
                watcher.close();
              }
            });

            setTimeout(reject, config.taskTimeout);
          });
        }
      });

      return config;
    }
  }
});

Full log and debug output

There are no particular errors.

[agoldis]

Would be great to be able to reproduce

[bdimitrovski]

@agoldis Your comment is quite interesting, I must admit - what better reproduction could you expect than this? I obviously cannot hand you over my source code, so what are the other options?

[bdimitrovski]

@agoldis Here's a repo for you: https://github.com/bdimitrovski/cypress-example-kitchensink - please try to clone it, and run :

1. npm install
2. npx cypress run --browser chrome

Without the plugin:

With the plugin:

This is a 28s difference for very trivial, FE heavy tests. Now, imagine much more complex tests like I have presented you with initially - it's quite obvious to me that the plugin becomes almost exponentially underperforming as the complexity and the scope of the tests rise.

[bdimitrovski]

@agoldis Any updates on this?

[bdimitrovski]

This is utterly disappointing. I feel that I've provided you with more than enough data and context, as well as a repro repo which very well demonstrates the issues we are facing, however, you did not even make an effort to reply.

[bdimitrovski]

@agoldis Could you please let me know how long you think it might take to fix this? It's been months without a single reply, can you at least confirm if there are performance issues on your end as well? Is it a OS specific or environment issue, unoptimised code, something completely different...

[agoldis]

@bdimitrovski Hi Bojan!
You won't be able to use this plugin with cypress version 13+, that's why it's getting not much attention (because of recent Cypress blocking) - that's why the issues for this plugins are getting neglected.
When we have a good solution for running cypress tests without being prone to cypress.io shenanigans, this plugin will get more love.

As for the issue itself - I will try taking a look at the repo you've shared.

Thanks for your understanding.

[Hypenate]

We are also facing this performance hit but I just read the article @agoldis so yeah...