Can not bao run the linux with rt_patch

[LSVulnerable]

Hello,

I want to run linux with rt patches on bao. The previous build process was OK. The built image worked with qemu, but it didn't work in bao. I want to know why, pls.

Thank you

[LSVulnerable]

i followed this guide and still can not login. i found that when a pl011 interrupt inject, maintenance interrupt occur and then guestos have no more response

                        .base = 0x60000000,
                        .size = 0x40000000,
                        .place_phys = true,
                        .phys = 0x60000000

#46 (comment)

[josecm]

I have filed an issue (#162) and will try to replicate the issue and debug it on our side. Hopefully I'll come back to you soon

[josecm]

@LSVulnerable can you provide more information on what kernel version and other build settings you are using? As well as about your bao config?

[josecm]

@zlaazlaa thanks for this insight! I'll find the time this week to look deeper into it

[josecm]

@LSVulnerable could you also confirm which Bao version are you using? Are you using it straight from the demo (version 1) or a more recent commit?

[josecm]

@LSVulnerable Unfortunately I am not being able to reproduce this issue with neither Bao version. I am using the linux version and patch you indicated and a configuration similar to the demos.

Maybe you could post here your configuration and the final guest binary you are passing to it.

Also, what is the qemu version you are using, and what command are you executing?

[josecm]

@LSVulnerable forget what I said. I was missing enabling PREEMPT_RT in Kconfig. I am seeing the same issue on my side. Hopefully I'll comeback with some news soon.

[zlaazlaa]

@josecm @LSVulnerable I'm replicate the issue, I found that the console only responds to a keyboard interrupt once. I copy a string form other place

root

and I paste it to the console, and the console changed to let me to enter password. Then it still stuck.

Hope, this could help.

[josecm]

@LSVulnerable @zlaazlaa

I have found the issue. What happens is that linux-rt seems to disable an interrupt as soon as it receives it. Bao reacted to it by removing it from the list registers, but failed to keep information around that this interrupt was still active (in the spilled active list). The guest would then eoir the interrupt without enabling it, triggering a maintenance interrupt since it was not on the list registers. However, since Bao wasn't keeping this info around, it failed to eoir the hardware interrupt that was kept always active.

I've managed a hot fix that simply keeps the active interrupt active in the list registers. Attached you will find the patch for it. However, I will have to look deeper into it to understand further implications of this issue and maybe come up with a more structured fix. Any ideas are welcomed.

From 9791459a615df9dac05ce8d98c388fee1a0d2843 Mon Sep 17 00:00:00 2001
From: Jose Martins <[email protected]>
Date: Wed, 24 Jul 2024 12:55:16 +0100
Subject: [PATCH] fix(arm/vgic): keep disable active irqs in lrs

Signed-off-by: Jose Martins <[email protected]>
---
 src/arch/armv8/vgic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/arch/armv8/vgic.c b/src/arch/armv8/vgic.c
index 3f362da1..a117ab4d 100644
--- a/src/arch/armv8/vgic.c
+++ b/src/arch/armv8/vgic.c
@@ -146,7 +146,7 @@ void vgic_send_sgi_msg(struct vcpu* vcpu, cpumap_t pcpu_mask, irqid_t int_id)
 
 static void vgic_route(struct vcpu* vcpu, struct vgic_int* interrupt)
 {
-    if ((interrupt->state == INV) || !interrupt->enabled) {
+    if ((interrupt->state == INV) || ((!interrupt->enabled) && (interrupt->state != ACT))) {
         return;
     }
 
@@ -315,7 +315,7 @@ bool vgic_add_lr(struct vcpu* vcpu, struct vgic_int* interrupt)
 {
     bool ret = false;
 
-    if (!interrupt->enabled || interrupt->in_lr) {
+    if (interrupt->state == INV || interrupt->in_lr) {
         return ret;
     }
 
-- 
2.34.1

[LSVulnerable]

Please let me know if this patch is working for you.

It works! Thank you very much for your great help, which has benefited me a lot! Have a nice day!

[zlaazlaa]

@LSVulnerable @zlaazlaa

I have found the issue. What happens is that linux-rt seems to disable an interrupt as soon as it receives it. Bao reacted to it by removing it from the list registers, but failed to keep information around that this interrupt was still active (in the spilled active list). The guest would then eoir the interrupt without enabling it, triggering a maintenance interrupt since it was not on the list registers. However, since Bao wasn't keeping this info around, it failed to eoir the hardware interrupt that was kept always active.

I've managed a hot fix that simply keeps the active interrupt active in the list registers. Attached you will find the patch for it. However, I will have to look deeper into it to understand further implications of this issue and maybe come up with a more structured fix. Any ideas are welcomed.

From 9791459a615df9dac05ce8d98c388fee1a0d2843 Mon Sep 17 00:00:00 2001
From: Jose Martins <[email protected]>
Date: Wed, 24 Jul 2024 12:55:16 +0100
Subject: [PATCH] fix(arm/vgic): keep disable active irqs in lrs

Signed-off-by: Jose Martins <[email protected]>
---
 src/arch/armv8/vgic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/arch/armv8/vgic.c b/src/arch/armv8/vgic.c
index 3f362da1..a117ab4d 100644
--- a/src/arch/armv8/vgic.c
+++ b/src/arch/armv8/vgic.c
@@ -146,7 +146,7 @@ void vgic_send_sgi_msg(struct vcpu* vcpu, cpumap_t pcpu_mask, irqid_t int_id)
 
 static void vgic_route(struct vcpu* vcpu, struct vgic_int* interrupt)
 {
-    if ((interrupt->state == INV) || !interrupt->enabled) {
+    if ((interrupt->state == INV) || ((!interrupt->enabled) && (interrupt->state != ACT))) {
         return;
     }
 
@@ -315,7 +315,7 @@ bool vgic_add_lr(struct vcpu* vcpu, struct vgic_int* interrupt)
 {
     bool ret = false;
 
-    if (!interrupt->enabled || interrupt->in_lr) {
+    if (interrupt->state == INV || interrupt->in_lr) {
         return ret;
     }
 
-- 
2.34.1

Unbelievable! It works for me too, Thanks! ❤️

[LSVulnerable]

@josecm i have more questions about this, whether the condition '!= ACT' is absolute or not and the case 'PENDACT' needs to be considered. And whether the change in 'vgic_add_lr' is not enough restrictive, when we need to eoir a PENDACT interrupt which is actually disabled in the spilled list, the maintenance interrupt make it into the lr because of the condition changed in 'vgic_add_lr'.
I would appreciate it if you could help me.

[josecm]

I think I get your point and you are probably right. This kind of thing I had in mind when I said "I will have to look deeper into". However, I haven't found the time yet, but this is on my todo list and logged as an issue #164.

For that specific issue, maybe we could extend that check to see if the interrupt is still enabled and only add it to the LR in that case.

[zlaazlaa]

I think I get your point and you are probably right. This kind of thing I had in mind when I said "I will have to look deeper into". However, I haven't found the time yet, but this is on my todo list and logged as an issue #164.

For that specific issue, maybe we could extend that check to see if the interrupt is still enabled and only add it to the LR in that case.

I have been looking at this issue recently as well, while looking code of IRQ, I have another question.
While giving the same interrupts to two different VM, why IRQ of arch timer work well, and IRQ of pl011 work failed.
IRQ of arch timer:
https://github.com/bao-project/bao-demos/blob/426a74912f340302e5baf975b63876ea1958703c/demos/linux%2Bfreertos/configs/qemu-aarch64-virt.c#L52-L56

https://github.com/bao-project/bao-demos/blob/426a74912f340302e5baf975b63876ea1958703c/demos/linux%2Bfreertos/configs/qemu-aarch64-virt.c#L116-L119
And the two VM work well.

But if we give the IRQ of pl011 to two VM, which is
https://github.com/bao-project/bao-demos/blob/426a74912f340302e5baf975b63876ea1958703c/demos/linux%2Bfreertos/configs/qemu-aarch64-virt.c#L108-L115
It just can't work, I think if work well, all the two VM's output will display in pl011 device.