Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chassis] Image upgrade on the 7808 chassis wipes out the configuration on the Linecards #105

Open
arlakshm opened this issue Nov 2, 2024 · 6 comments
Open

[chassis] Image upgrade on the 7808 chassis wipes out the configuration on the Linecards #105

arlakshm opened this issue Nov 2, 2024 · 6 comments

Comments

@arlakshm
Copy link

arlakshm commented Nov 2, 2024

The 7808 chassis in the lab is being upgrading using the command

./testbed-cli.sh install-image vms26-t2-7800-1 str2 http://10.201.148.43/pipelines/Networking-acs-buildimage-Official/broadcom/internal-202405/sonic-aboot-broadcom-dnx.swi

vms26-t2-7800-1 is the T2 testbed with one supervisor and 3 Linecards (2 Clearwater2 cards and one wolverine card)

str2 is the inventory file name

This command upgrades the supervisor and all the linecards and waits for Linecards and supervisor to be up.

This commands errors out because the chassis is not accessible after 15 mins

On logging in from the serial port, I see these errors on the screen for long time

[  930.802723] pcieport 0000:06:05.0: DPC: unmasked uncorrectable error detected
[  930.802798] pcieport 0000:06:05.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  930.802800] pcieport 0000:06:05.0:   device [11f8:8533] error status/mask=00000020/04400000
[  930.802802] pcieport 0000:06:05.0:    [ 5] SDES
[  930.802951] pcieport 0000:06:05.0: AER: device recovery successful
[  930.853928] pcieport 0000:06:03.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[  930.953965] pcieport 0000:06:03.0:    [ 5] SDES
[  930.954113] pcieport 0000:06:03.0: AER: device recovery successful
[  931.026989] pcieport 0000:06:03.0:   device [11f8:8533] error status/mask=00000001/0000e000
[  931.432374] pcieport 0000:06:03.0: pciehp: Slot(4): No link
[  931.456587] pcieport 0000:06:03.0:    [ 0] RxErr                  (First)
[  931.556821] pcieport 0000:06:03.0: DPC: containment event, status:0x0001 source:0x0000
[  931.629785] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:05.0
[  931.757968] pcieport 0000:06:03.0: DPC: unmasked uncorrectable error detected
[  931.758044] pcieport 0000:06:03.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  931.892045] pcieport 0000:06:05.0: DPC: containment event, status:0x0001 source:0x0000
[  931.931107] pcieport 0000:06:03.0:   device [11f8:8533] error status/mask=00000020/04400000
[  931.931109] pcieport 0000:06:03.0:    [ 5] SDES
[  931.931303] pcieport 0000:06:03.0: AER: device recovery successful
[  932.014670] pcieport 0000:06:05.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[  932.014673] pcieport 0000:06:05.0:   device [11f8:8533] error status/mask=00000001/0000e000
[  932.014676] pcieport 0000:06:05.0:    [ 0] RxErr                  (First)
[  932.014740] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:03.0
[  932.014791] pcieport 0000:00:03.0: AER: Multiple Corrected error message received from 0000:06:05.0
[  932.015414] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:03.0
[  932.015465] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:05.0
[  932.015516] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:03.0
[  932.015567] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:05.0
[  932.059440] pcieport 0000:06:05.0: DPC: unmasked uncorrectable error detected
[  932.584515] pcieport 0000:00:03.0: AER: Corrected error message received from 0000:06:03.0
[  932.584758] pcieport 0000:06:03.0: DPC: containment event, status:0x0001 source:0x0000
[  932.584760] pcieport 0000:06:03.0: DPC: unmasked uncorrectable error detected
[  932.584836] pcieport 0000:06:03.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  932.584838] pcieport 0000:06:03.0:   device [11f8:8533] error status/mask=00000020/04400000
[  932.584841] pcieport 0000:06:03.0:    [ 5] SDES
[  932.584990] pcieport 0000:06:03.0: AER: device recovery successful
[  932.621666] pcieport 0000:06:05.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[  932.749805] pcieport 0000:06:03.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[  932.849887] pcieport 0000:06:05.0:   device [11f8:8533] error status/mask=00040020/04400000
[  932.849891] pcieport 0000:06:05.0:    [ 5] SDES
[  932.849893] pcieport 0000:06:05.0:    [18] MalfTLP                (First)
[  932.922960] pcieport 0000:06:03.0:   device [11f8:8533] error status/mask=00000001/0000e000

Eventually these backtraces are seen supervisor bootups

[  939.096369] INFO: task systemd:1 blocked for more than 311 seconds.
[  939.096373]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.096375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.096376] task:systemd         state:D stack:0     pid:1     ppid:0      flags:0x00000002
[  939.096380] Call Trace:
[  939.096382]  <TASK>
[  939.096385]  __schedule+0x34d/0x9e0
[  939.096392]  schedule+0x5a/0xd0
[  939.096394]  schedule_timeout+0x118/0x150
[  939.096398]  __down_common+0x11a/0x230
[  939.096402]  down+0x43/0x60
[  939.096404]  console_lock+0x21/0x80
[  939.096409]  show_cons_active+0x3f/0x190
[  939.096415]  dev_attr_show+0x18/0x40
[  939.096419]  sysfs_kf_seq_show+0xa3/0xe0
[  939.096424]  seq_read_iter+0x122/0x450
[  939.096430]  vfs_read+0x23c/0x310
[  939.096434]  ksys_read+0x6b/0xf0
[  939.096437]  do_syscall_64+0x55/0xb0
[  939.096443]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096446]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096449]  ? do_syscall_64+0x61/0xb0
[  939.096452]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096454]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096457]  ? do_syscall_64+0x61/0xb0
[  939.096460]  ? do_syscall_64+0x61/0xb0
[  939.096463]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096465]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096468]  ? do_syscall_64+0x61/0xb0
[  939.096471]  ? do_syscall_64+0x61/0xb0
[  939.096474]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096476]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096479]  ? do_syscall_64+0x61/0xb0
[  939.096482]  ? do_syscall_64+0x61/0xb0
[  939.096485]  ? do_syscall_64+0x61/0xb0
[  939.096488]  ? do_syscall_64+0x61/0xb0
[  939.096492]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.096497] RIP: 0033:0x7f14f656f1dc
[  939.096500] RSP: 002b:00007ffc865e21b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  939.096502] RAX: ffffffffffffffda RBX: 000055d769862b60 RCX: 00007f14f656f1dc
[  939.096504] RDX: 0000000000001000 RSI: 000055d7699b93c0 RDI: 0000000000000011
[  939.096506] RBP: 00007f14f66465e0 R08: 0000000000000000 R09: 00007f14f6649d30
[  939.096507] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f14f5a0d7f0
[  939.096508] R13: 0000000000000d68 R14: 00007f14f66459e0 R15: 0000000000000d68
[  939.096511]  </TASK>
[  939.096578] INFO: task systemd-logind:10117 blocked for more than 311 seconds.
[  939.096580]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.096581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.096582] task:systemd-logind  state:D stack:0     pid:10117 ppid:1      flags:0x00000002
[  939.096585] Call Trace:
[  939.096586]  <TASK>
[  939.096587]  __schedule+0x34d/0x9e0
[  939.096590]  schedule+0x5a/0xd0
[  939.096592]  schedule_timeout+0x118/0x150
[  939.096595]  __down_common+0x11a/0x230
[  939.096598]  ? device_match_of_node+0x20/0x20
[  939.096601]  down+0x43/0x60
[  939.096603]  console_lock+0x21/0x80
[  939.096606]  con_install+0x1b/0x140
[  939.096609]  tty_init_dev.part.0+0x45/0x220
[  939.096613]  tty_open+0x477/0x690
[  939.096615]  ? reconfigure_single+0x60/0x60
[  939.096619]  ? kobj_lookup+0xf1/0x160
[  939.096623]  chrdev_open+0xc4/0x250
[  939.096626]  ? __unregister_chrdev+0x50/0x50
[  939.096630]  do_dentry_open+0x1e5/0x410
[  939.096635]  path_openat+0xb7d/0x1260
[  939.096638]  ? memcg_list_lru_alloc+0xac/0x3d0
[  939.096643]  ? memcg_slab_post_alloc_hook+0x160/0x220
[  939.096647]  do_filp_open+0xaf/0x160
[  939.096653]  do_sys_openat2+0xaf/0x170
[  939.096655]  __x64_sys_openat+0x6a/0xa0
[  939.096657]  do_syscall_64+0x55/0xb0
[  939.096661]  ? audit_filter_inodes.part.0+0x2e/0x120
[  939.096665]  ? audit_reset_context+0x232/0x300
[  939.096667]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096670]  ? memcg_slab_post_alloc_hook+0x160/0x220
[  939.096672]  ? ep_ptable_queue_proc+0x2b/0x90
[  939.096677]  ? __seccomp_filter+0x32a/0x4e0
[  939.096682]  ? audit_filter_inodes.part.0+0x2e/0x120
[  939.096684]  ? audit_reset_context+0x232/0x300
[  939.096686]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096688]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096691]  ? do_syscall_64+0x61/0xb0
[  939.096695]  ? audit_filter_inodes.part.0+0x2e/0x120
[  939.096697]  ? audit_reset_context+0x232/0x300
[  939.096699]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096701]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096703]  ? do_syscall_64+0x61/0xb0
[  939.096707]  ? handle_mm_fault+0xdb/0x2d0
[  939.096711]  ? do_user_addr_fault+0x1b0/0x550
[  939.096714]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096716]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.096719] RIP: 0033:0x7ff95ca0ff01
[  939.096721] RSP: 002b:00007fff8acd3a10 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[  939.096723] RAX: ffffffffffffffda RBX: 0000000000080902 RCX: 00007ff95ca0ff01
[  939.096725] RDX: 0000000000080902 RSI: 000056199b48d0f0 RDI: 00000000ffffff9c
[  939.096726] RBP: 000056199b48d0f0 R08: 0000000000000007 R09: 000056199b48c3e0
[  939.096727] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[  939.096728] R13: 0000000000000000 R14: 0000000000000000 R15: 000056199b475990
[  939.096731]  </TASK>
[  939.096733] INFO: task runc:[2:INIT]:10258 blocked for more than 311 seconds.
[  939.096735]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.096736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.096736] task:runc:[2:INIT]   state:D stack:0     pid:10258 ppid:10243  flags:0x00000002
[  939.096739] Call Trace:
[  939.096740]  <TASK>
[  939.096741]  __schedule+0x34d/0x9e0
[  939.096743]  schedule+0x5a/0xd0
[  939.096745]  schedule_preempt_disabled+0x11/0x20
[  939.096748]  __mutex_lock.constprop.0+0x399/0x700
[  939.096751]  ptmx_open+0x8b/0x190
[  939.096754]  chrdev_open+0xc4/0x250
[  939.096758]  ? __unregister_chrdev+0x50/0x50
[  939.096761]  do_dentry_open+0x1e5/0x410
[  939.096765]  path_openat+0xb7d/0x1260
[  939.096768]  ? audit_reset_context+0x232/0x300
[  939.096771]  do_filp_open+0xaf/0x160
[  939.096776]  do_sys_openat2+0xaf/0x170
[  939.096778]  __x64_sys_openat+0x6a/0xa0
[  939.096780]  do_syscall_64+0x55/0xb0
[  939.096784]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096786]  ? do_syscall_64+0x61/0xb0
[  939.096790]  ? __rseq_handle_notify_resume+0xa9/0x4a0
[  939.096795]  ? call_rcu+0xde/0x6b0
[  939.096800]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096802]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096804]  ? do_syscall_64+0x61/0xb0
[  939.096808]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096810]  ? do_syscall_64+0x61/0xb0
[  939.096813]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096816]  ? do_syscall_64+0x61/0xb0
[  939.096819]  ? do_syscall_64+0x61/0xb0
[  939.096822]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096824]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.096828] RIP: 0033:0x5570a91c42ce
[  939.096829] RSP: 002b:000000c00018f5f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[  939.096831] RAX: ffffffffffffffda RBX: ffffffffffffff9c RCX: 00005570a91c42ce
[  939.096832] RDX: 0000000000080102 RSI: 000000c00020ad60 RDI: ffffffffffffff9c
[  939.096834] RBP: 000000c00018f630 R08: 0000000000000000 R09: 0000000000000000
[  939.096835] R10: 0000000000000000 R11: 0000000000000202 R12: 000000c00020ad60
[  939.096836] R13: 0000000000000000 R14: 000000c0000061a0 R15: ffffffffffffffff
[  939.096838]  </TASK>
[  939.096844] INFO: task systemd-getty-g:11267 blocked for more than 311 seconds.
[  939.096845]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.096846] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.096847] task:systemd-getty-g state:D stack:0     pid:11267 ppid:1      flags:0x00000006
[  939.096849] Call Trace:
[  939.096850]  <TASK>
[  939.096851]  __schedule+0x34d/0x9e0
[  939.096853]  schedule+0x5a/0xd0
[  939.096855]  schedule_timeout+0x118/0x150
[  939.096858]  __down_common+0x11a/0x230
[  939.096861]  down+0x43/0x60
[  939.096864]  console_lock+0x21/0x80
[  939.096866]  show_cons_active+0x3f/0x190
[  939.096870]  dev_attr_show+0x18/0x40
[  939.096873]  sysfs_kf_seq_show+0xa3/0xe0
[  939.096875]  seq_read_iter+0x122/0x450
[  939.096880]  vfs_read+0x23c/0x310
[  939.096883]  ksys_read+0x6b/0xf0
[  939.096885]  do_syscall_64+0x55/0xb0
[  939.096889]  ? __vfs_getxattr+0x53/0x70
[  939.096893]  ? get_vfs_caps_from_disk+0x8a/0x220
[  939.096897]  ? audit_copy_inode+0xb3/0xf0
[  939.096899]  ? filename_lookup+0x18f/0x1f0
[  939.096904]  ? _copy_to_user+0x21/0x30
[  939.096908]  ? cp_new_stat+0x135/0x170
[  939.096913]  ? __do_sys_newfstatat+0x4e/0x80
[  939.096917]  ? mntput_no_expire+0x4a/0x250
[  939.096920]  ? audit_reset_context+0x232/0x300
[  939.096923]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096924]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096927]  ? do_syscall_64+0x61/0xb0
[  939.096931]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096933]  ? do_syscall_64+0x61/0xb0
[  939.096937]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.096938]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.096941]  ? do_syscall_64+0x61/0xb0
[  939.096944]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.096948] RIP: 0033:0x7f23d7f6e19d
[  939.096949] RSP: 002b:00007ffc80af97c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  939.096951] RAX: ffffffffffffffda RBX: 0000555df4e1b2d0 RCX: 00007f23d7f6e19d
[  939.096952] RDX: 0000000000001000 RSI: 0000555df4e1ba20 RDI: 0000000000000004
[  939.096953] RBP: 00007f23d80455e0 R08: 0000000000000008 R09: 0000000000000001
[  939.096954] R10: 0000000000001000 R11: 0000000000000246 R12: 00007f23d74252f0
[  939.096955] R13: 0000000000000d68 R14: 00007f23d80449e0 R15: 0000000000000d68
[  939.096958]  </TASK>
[  939.096962] INFO: task runc:[2:INIT]:11435 blocked for more than 311 seconds.
[  939.096963]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.096964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.096965] task:runc:[2:INIT]   state:D stack:0     pid:11435 ppid:11389  flags:0x00000002
[  939.096967] Call Trace:
[  939.096968]  <TASK>
[  939.096969]  __schedule+0x34d/0x9e0
[  939.096972]  schedule+0x5a/0xd0
[  939.096974]  schedule_preempt_disabled+0x11/0x20
[  939.096976]  __mutex_lock.constprop.0+0x399/0x700
[  939.096979]  ptmx_open+0x8b/0x190
[  939.096981]  chrdev_open+0xc4/0x250
[  939.096985]  ? __unregister_chrdev+0x50/0x50
[  939.096988]  do_dentry_open+0x1e5/0x410
[  939.096992]  path_openat+0xb7d/0x1260
[  939.096996]  do_filp_open+0xaf/0x160
[  939.097001]  do_sys_openat2+0xaf/0x170
[  939.097003]  __x64_sys_openat+0x6a/0xa0
[  939.097005]  do_syscall_64+0x55/0xb0
[  939.097009]  ? audit_filter_inodes.part.0+0x2e/0x120
[  939.097011]  ? audit_reset_context+0x232/0x300
[  939.097013]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.097015]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.097018]  ? do_syscall_64+0x61/0xb0
[  939.097021]  ? __do_sys_newfstat+0x6b/0x80
[  939.097025]  ? audit_filter_inodes.part.0+0x2e/0x120
[  939.097027]  ? audit_reset_context+0x232/0x300
[  939.097029]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.097031]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.097033]  ? do_syscall_64+0x61/0xb0
[  939.097037]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.097039]  ? do_syscall_64+0x61/0xb0
[  939.097043]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.097046] RIP: 0033:0x55e98173d2ce
[  939.097047] RSP: 002b:000000c00018f5f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[  939.097049] RAX: ffffffffffffffda RBX: ffffffffffffff9c RCX: 000055e98173d2ce
[  939.097050] RDX: 0000000000080102 RSI: 000000c000208da0 RDI: ffffffffffffff9c
[  939.097052] RBP: 000000c00018f630 R08: 0000000000000000 R09: 0000000000000000
[  939.097053] R10: 0000000000000000 R11: 0000000000000202 R12: 000000c000208da0
[  939.097054] R13: 0000000000000000 R14: 000000c0000061a0 R15: ffffffffffffffff
[  939.097056]  </TASK>
[  939.097057] INFO: task runc:[2:INIT]:11436 blocked for more than 311 seconds.
[  939.097058]       Tainted: G           OE      6.1.0-22-2-amd64 #1 Debian 6.1.94-1
[  939.097059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  939.097060] task:runc:[2:INIT]   state:D stack:0     pid:11436 ppid:11426  flags:0x00000002
[  939.097062] Call Trace:
[  939.097063]  <TASK>
[  939.097064]  __schedule+0x34d/0x9e0
[  939.097066]  schedule+0x5a/0xd0
[  939.097068]  schedule_preempt_disabled+0x11/0x20
[  939.097070]  __mutex_lock.constprop.0+0x399/0x700
[  939.097073]  ptmx_open+0x8b/0x190
[  939.097075]  chrdev_open+0xc4/0x250
[  939.097079]  ? __unregister_chrdev+0x50/0x50
[  939.097082]  do_dentry_open+0x1e5/0x410
[  939.097085]  path_openat+0xb7d/0x1260
[  939.097089]  do_filp_open+0xaf/0x160
[  939.097094]  do_sys_openat2+0xaf/0x170
[  939.097096]  __x64_sys_openat+0x6a/0xa0
[  939.097098]  do_syscall_64+0x55/0xb0
[  939.097102]  ? get_vfs_caps_from_disk+0x8a/0x220
[  939.097107]  ? ext4_getattr+0x98/0x160 [ext4]
[  939.097145]  ? ovl_getattr+0xa1/0x3d0 [overlay]
[  939.097157]  ? _copy_to_user+0x21/0x30
[  939.097161]  ? cp_new_stat+0x135/0x170
[  939.097166]  ? __do_sys_newfstatat+0x4e/0x80
[  939.097170]  ? mntput_no_expire+0x4a/0x250
[  939.097173]  ? audit_reset_context+0x232/0x300
[  939.097175]  ? exit_to_user_mode_prepare+0x44/0x1f0
[  939.097178]  ? syscall_exit_to_user_mode+0x1e/0x40
[  939.097181]  ? do_syscall_64+0x61/0xb0
[  939.097184]  ? do_syscall_64+0x61/0xb0
[  939.097188]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  939.097192] RIP: 0033:0x5613ec4ec2ce
[  939.097193] RSP: 002b:000000c0001af5f0 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[  939.097195] RAX: ffffffffffffffda RBX: ffffffffffffff9c RCX: 00005613ec4ec2ce
[  939.097197] RDX: 0000000000080102 RSI: 000000c000206aa0 RDI: ffffffffffffff9c
[  939.097198] RBP: 000000c0001af630 R08: 0000000000000000 R09: 0000000000000000
[  939.097200] R10: 0000000000000000 R11: 0000000000000202 R12: 000000c000206aa0
[  939.097201] R13: 0000000000000000 R14: 000000c0000061a0 R15: ffffffffffffffff
[  939.097204]  </TASK>
[  939.097207] Kernel panic - not syncing: hung_task: blocked tasks
[  939.097209] CPU: 11 PID: 89 Comm: khungtaskd Tainted: G           OE      6.1.0-22-2-amd64 #1  Debian 6.1.94-1
[  939.097212] Hardware name: Intel Camelback Mountain CRB/Camelback Mountain CRB, BIOS Aboot-norcal7-7.1.4-14169220 11/09/2019
[  939.097213] Call Trace:
[  939.097214]  <TASK>
[  939.097215]  dump_stack_lvl+0x44/0x5c
[  939.097219]  panic+0x118/0x2f4
[  939.097223]  watchdog.cold+0xc/0xbb
[  939.097228]  ? proc_dohung_task_timeout_secs+0x30/0x30
[  939.097233]  kthread+0xda/0x100
[  939.097236]  ? kthread_complete_and_exit+0x20/0x20
[  939.097239]  ret_from_fork+0x22/0x30
[  939.097245]  </TASK>
[  940.130309] Shutting down cpus with NMI
[  940.130315] Kernel Offset: 0xa200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)


Aboot 7.1.4-14169220


Press Control-C now to enter Aboot shell
Booting flash:image-20240531.10/.sonic-boot.swi
Secure Boot disabled, skipping check
TPM: failed to raise GPIO line
SPI flash hardware write protection disabled
19.46: Using previously installed image
19.52: Loading extra kernel parameters from kernel-params
19.58: Next reboot will use flash:image-20240531.10/.sonic-boot.swi
20.16: Kexecing...
[   20.335936] Starting new kernel
[    1.091971] tpm tpm0: [Firmware Bug]: TPM interrupt not working, polling instead
setfacl: /mnt/root-convfs/aquota.user: Operation not permitted
tune2fs 1.47.0 (5-Feb-2023)
Setting reserved blocks percentage to 0% (0 blocks)
Setting reserved blocks count to 0

after this the linecards come up with the new image but the minigraph/config is not retained from the previous image

admin@str2-7804-sup-1:~$ ssh 127.100.5.1
The authenticity of host '127.100.5.1 (127.100.5.1)' can't be established.
RSA key fingerprint is SHA256:qPr4CpwSSNyWk3aLhJxGXBQ2683BT0xWilzHnH+veE4.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '127.100.5.1' (RSA) to the list of known hosts.
Debian GNU/Linux 12 \n \l

[email protected]'s password:
Linux sonic 6.1.0-22-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64
You are on
  ____   ___  _   _ _  ____
 / ___| / _ \| \ | (_)/ ___|
 \___ \| | | |  \| | | |
  ___) | |_| | |\  | | |___
 |____/ \___/|_| \_|_|\____|

-- Software for Open Networking in the Cloud --

Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.

Help:      https://sonic-net.github.io/SONiC/
Wiki:      https://microsoft.sharepoint.com/teams/WAG/AzureNetworking/Wiki/SONiC.aspx
On-Call:   https://portal.microsofticm.com/imp/v3/oncall/current?serviceId=10045&teamIds=26162
Dashboard: https://aka.ms/sonic-dri
Contact:   [email protected]

Last login: Sat Nov  2 01:34:30 2024 from 10.1.84.57
admin@sonic:~$ show chassis module status
Key * not found in CHASSIS_MODULE_TABLE table
admin@sonic:~$ ls /etc/sonic/
asic_config_checksum   fast-reboot_order        snmp.yml
constants.yml          frr                      sonic-environment
copp_cfg.json          generated_services.conf  sonic_release
core_analyzer.rc.json  grpc_secrets.json        sonic_version.yml
credentials            init_cfg.json            swss_dependent
default_users.json     macsec_reconcile         warm-reboot_order
dhcp_relay_reconcile   old_config
dhcp_server_reconcile  remote_ctr.config.json
admin@sonic:~$ show vers

SONiC Software Version: SONiC.20240531.10
SONiC OS Version: 12
Distribution: Debian 12.6
Kernel: 6.1.0-22-2-amd64
Build commit: cf7d8848af
Build date: Fri Nov  1 09:34:37 UTC 2024
Built by: azureuser@53ad391cc000000

Platform: x86_64-arista_7800r3a_36dm2_lc
HwSKU: None
ASIC: broadcom
ASIC Count: 2
Serial Number: SGD21190878
Model Number: 7800R3A-36DM2-LC
Hardware Revision: 2a.00
Uptime: 02:16:07 up 5 min,  1 user,  load average: 0.63, 1.32, 0.71
Date: Sat 02 Nov 2024 02:16:07

Looks like the linecards are provisioned again.
@arlakshm
Copy link
Author

arlakshm commented Nov 2, 2024

add arista_logs
arista_logs.tar.gz

@arlakshm
Copy link
Author

arlakshm commented Nov 2, 2024

add
dmesg.log

@patrickmacarthur
Copy link
Contributor

I ran this on one of our DUTs and was unable to reproduce the issue. Is this issue consistently reproducible on your end or has it only happened this once?

@kenneth-arista
Copy link

As discussed offline the clearing of configuration files may a consequence of the PCIe errors seen on this particular supervisor.

@arlakshm
Copy link
Author

arlakshm commented Dec 9, 2024

@kenneth-arista, can you please provide a root-cause why the linecard went to provision mode.

@rlhui
Copy link

rlhui commented Dec 12, 2024

@kenneth-arista - it seems due to this issue, this testbed in our lab cannot be used. What should be the fix for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@patrickmacarthur @rlhui @arlakshm @kenneth-arista