iCubGenova11 S/N:050 – Temporary loss of ETH link

[pattacini]

Robot Name 🤖

iCubGenova11 S/N:050

Request/Failure description

Today, while @mfussi66 and I were trying to run the red-ball demo, we spotted that the robot got stuck in a nondeterministic manner.

Detailed context

From the log posted in the next section, we believe that the problem may be due to the ETH cable. The link seems to have been lost for a while before being restored.

These failures happen quite often on the robot during the red ball demo and, in particular, when iCub reaches for specific configurations of the ensemble torso + arm.

Likely, a mechanical inspection is required to find the interruption and rewire the ETH.

Additional context

Here's below the yarprobotinterface log when the problem occurs:

[ERROR] from BOARD 10.0.1.27 (right_arm-eb27-j4_7) time=4837s 609m 121u :  ETH monitor: link goes down  in port ETH output (P3/P12/J5). Application state is unknown.
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
...
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
[ERROR] |yarp.devices.controlBoard_nws_yarp| Encoder Timestamps are not consistent! Data will not be published.
[ERROR] [h, o, s, t, T, r, a, n, s, c, e, i, v, e, r, (, ), :, :, p, a, r, s, e, (, ),  , d, e, t, e, c, t, e, d,  , a, n,  , E, R, R, O, R,  , i, n,  , s, e, q, u, e, n, c, e,  , n, u, m, b, e, r,  , f, r, o, m,  , I, P,  , =,  , 1, 0, ., 0, ., 1, ., 5, .,  , E, x, p, e, c, t, e, d, :,  , 8, 6, 2, 1, 5, 6, ,,  , R, e, c, e, i, v, e, d, :,  , 8, 6, 2, 5, 3, 7, ,,  , M, i, s, s, i, n, g, :,  , 3, 8, 1, ,,  , P, r, e, v,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 7, 1, 8, 3, 4, 8, 4,  , u, s, ,,  , T, h, i, s,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 9, 0, 9, 2, 4, 7, 5,  , u, s, 
[ERROR] from BOARD 10.0.1.5 (torso-eb5-j0_2) time=4839s 91m 876u :  SYS: the board has detected a hole in rx ropframe sequence number. Expected number is 12759, received number is 12948 
[ERROR] [h, o, s, t, T, r, a, n, s, c, e, i, v, e, r, (, ), :, :, p, a, r, s, e, (, ),  , d, e, t, e, c, t, e, d,  , a, n,  , E, R, R, O, R,  , i, n,  , s, e, q, u, e, n, c, e,  , n, u, m, b, e, r,  , f, r, o, m,  , I, P,  , =,  , 1, 0, ., 0, ., 1, ., 2, 8, .,  , E, x, p, e, c, t, e, d, :,  , 8, 6, 0, 4, 6, 2, ,,  , R, e, c, e, i, v, e, d, :,  , 8, 6, 0, 8, 4, 4, ,,  , M, i, s, s, i, n, g, :,  , 3, 8, 2, ,,  , P, r, e, v,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 7, 4, 9, 6, 7, 6, 2,  , u, s, ,,  , T, h, i, s,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 9, 4, 0, 6, 7, 6, 3,  , u, s, 
[ERROR] [h, o, s, t, T, r, a, n, s, c, e, i, v, e, r, (, ), :, :, p, a, r, s, e, (, ),  , d, e, t, e, c, t, e, d,  , a, n,  , E, R, R, O, R,  , i, n,  , s, e, q, u, e, n, c, e,  , n, u, m, b, e, r,  , f, r, o, m,  , I, P,  , =,  , 1, 0, ., 0, ., 1, ., 2, 9, .,  , E, x, p, e, c, t, e, d, :,  , 8, 6, 0, 0, 8, 0, ,,  , R, e, c, e, i, v, e, d, :,  , 8, 6, 0, 4, 6, 1, ,,  , M, i, s, s, i, n, g, :,  , 3, 8, 1, ,,  , P, r, e, v,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 7, 4, 5, 6, 0, 5, 2,  , u, s, ,,  , T, h, i, s,  , F, r, a, m, e,  , T, X,  , a, t,  , 4, 8, 3, 9, 3, 6, 6, 0, 5, 2,  , u, s, 
[ERROR] from BOARD 10.0.1.28 (right_arm-eb28-j8_11) time=4839s 407m 853u :  ETH monitor: link goes up. in port ETH input (P2/P13/J4). Application state is unknown.
[DEBUG] eth::EthMonitorPresence: BOARD 10.0.1.5 (torso-eb5-j0_2) has shown after being lost for 1.91301 sec
[DEBUG] eth::EthMonitorPresence: BOARD 10.0.1.28 (right_arm-eb28-j8_11) has shown after being lost for 1.91305 sec
[DEBUG] eth::EthMonitorPresence: BOARD 10.0.1.29 (right_arm-eb29-j12_15) has shown after being lost for 1.91302 sec
[ERROR] from BOARD 10.0.1.27 (right_arm-eb27-j4_7) time=4839s 510m 123u :  ETH monitor: link goes up. in port ETH output (P3/P12/J5). Application state is unknown.

How does it affect you?

We have a demo in the early afternoon, but we can get away with what we have to show.
No problem.

[pattacini]

cc @martinaxgloria @xEnVrE

[AntonioAzocar]

Hi @pattacini , I found the E27 cable with a damaged crimp, so I fixed it.
We tested the robot and the problem of this issue has been solved.
Thank you @martinaxgloria for your help during the tests!

cc @sgiraz @xEnVrE

[martinaxgloria]

Just to be more precise, we checked that running the yarprobotinterface the error was not displayed anymore. We didn't try running the red ball demo for example, so we are not sure that the problem is fixed also for those particular configuration mentioned by @pattacini.

[pattacini]

Super, thanks heaps @AntonioAzocar 👍🏻

Yep, @martinaxgloria is right. We may give the red ball demo a try.

[AntonioConsilvio]

Hi @pattacini @mfussi66, the demo red ball has not been tested, but we are aware that the robot has been used many times in the last days without showing any ETH issues. If you are agree I proceed closing this issue.

cc @sgiraz @martinaxgloria @AntonioAzocar

[pattacini]

Ok!