Performance

[tangkhaiphuong]

Question

I concern about the performance of stream sheets. For example, My plant has 300 standalone machines, each machine needs collection sensor data for (vibration/temperature/speed) of many motors and has 100 signals per machine. So if apply full plant, we have 300 x 100 = 300.000 signal/seconds.

My sheet simply just consumes from MQTT consumer and use Rest for storing in InfluxDB only.
So is streamsheets is suitable for this case?

Context

I look in deep repos and found that streamsheets is developed by Node.js. And Node.js only have 1 JS thread event-loop for handler code logic. So node.js JS thread event-loop will busy to process sheet-logic so for a same time have large signal can be slow?

We can accept for slow, but how can scale out?

[tangkhaiphuong-colpal]

After looking in deep process of streamsheets. I found

Mem: 1908288K used, 104032K free, 1228K shrd, 335416K buff, 381084K cached
CPU:  23% usr   9% sys   0% nic  60% idle   3% io   0% irq   2% sirq
Load average: 1.40 1.50 1.79 6/645 330
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  169    28 root     R     304m  15%   1   9% /usr/local/bin/node /streamsheets/packages/machine-core/src/ipc/MachineTask.js --owner anon
   29    14 root     S    1359m  69%   1   4% node /streamsheets/packages/service-streams/start.js
   21    14 root     R     375m  19%   1   3% {@cedalo/gateway} @cedalo/gateway
   14     1 root     S    19384   1%   0   2% {supervisord} /usr/bin/python2 /usr/bin/supervisord -c /etc/supervisord.conf
  183    28 root     S     304m  15%   1   2% /usr/local/bin/node /streamsheets/packages/machine-core/src/ipc/MachineTask.js --owner anon
   20    14 mosquitt S     4808   0%   1   1% /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
   17    14 root     S    19988   1%   1   1% /usr/bin/redis-server /etc/redis.conf --bind 0.0.0.0 --notify-keyspace-events Kls
   16    14 root     S     803m  41%   1   1% /usr/bin/mongod --config /etc/mongod.conf --bind_ip 0.0.0.0
   28    14 root     S     317m  16%   1   0% node /streamsheets/packages/service-machines/start.js
   27    19 nginx    S     6496   0%   0   0% nginx: worker process
   23    14 root     S     328m  17%   1   0% node /streamsheets/packages/service-graphs/start.js
   22    14 root     S     277m  14%   1   0% node /usr/local/bin/pushstate-server .
   19    14 root     S     5804   0%   1   0% nginx: master process /usr/sbin/nginx -g pid /tmp/nginx.pid; daemon off;
   18    14 mosquitt S     4516   0%   0   0% /usr/sbin/mosquitto -c /etc/mosquitto-default/mosquitto.conf
  307     0 root     S     1600   0%   0   0% sh
    1     0 root     S     1596   0%   1   0% sh streamsheets.sh
  314   307 root     R     1528   0%   1   0% top

Each Stream Machine is handled by
usr/local/bin/node /streamsheets/packages/machine-core/src/ipc/MachineTask.js --owner anon

So if I have 10 stream machine. I have 10 process run independent so the it's no problem for many stream machine.

[evlach]

Apart from the IPC implementation that allows what you describe above, Streamsheets' microservice architecture allows for horizontal scaling. Of course there is always room for improvements.