Libinput and fbdev improvements for unl0kr #280

calebccff · 2023-06-04T20:58:54Z

Add a framebuffer quirk to force the display to refresh after every draw, this is needed for some quirky panels and DRM/fbdev emulation setups. It requires that the display buffer is set to the same size as the whole framebuffer so that the entire display can be flushed at once if needed, otherwise there is tearing.

Also use pthreads and a small circular queue to stop the libinput driver from dropping inputs. The LVGL tickrate can't always keep up and this ensures that inputs aren't dropped.

The keypad support needs to be tested.

Cc: @Johennes

Johennes

Pretty amazing stuff! Thanks @calebccff!

I've only given it a quick look this morning. Will try to do some testing against https://gitlab.com/cherrypicker/unl0kr/-/merge_requests/21 this week.

@kisvegabor, I know, lv_drivers is supposed to be deprecated in future. It'll be very helpful to have these changes merged though because https://gitlab.com/cherrypicker/unl0kr still depends on it.

We already started migrating to lvgl master in https://gitlab.com/cherrypicker/unl0kr/-/merge_requests/20 and we'll port the changes to the fbdev driver and the libinput driver, too, as part of this.

kisvegabor

Really nice, thank you!

We can merge it here, but yeah... it'd be great to add these updates to master as well. 🙂

display/fbdev.h

indev/libinput.c

calebccff · 2023-06-24T22:14:59Z

Totally forgot to push 😅 sorry for the wait. The touch/pointer handling was actually pretty broken... Anyways, hopefully this is good to go now!

Johennes · 2023-06-30T05:16:59Z

Thanks for updating the PR @calebccff and sorry for the delay. I will take a look at this and test it within unl0kr but I won't manage before next week, unfortunately.

Johennes

Sorry again for the delay. I reviewed the code and tested it with https://gitlab.com/cherrypicker/unl0kr/-/merge_requests/21 in unl0kr.

Pointer devices don't currently work correctly because their delta motion logic needs special handling with the new event buffer.

There is possibly also a problem with the mutex. I didn't notice any issues related to that when testing but I think the mutex initialization is missing and might just work accidentally in unl0kr.

indev/libinput.c

This is needed specifically for Qualcomm devices using DRM fbdev emulation, it is probably useful for other platforms using fbdev emulation.

Libinput doesn't buffer events, so hook up a dedicated thread per input device and a small circular queue to buffer input events. This fixes the dropped input issue so you can type much faster. If there are too many events, the oldest events in the queue will be dropped.

When typing fast with thumbs, you might hit a key while your other thumb is already pressing a key, however this confuses indev quite a lot. For example, you'll press a key (say 'h') with your right thumb, then before releasing your thumb you'll press the 'e' key with your left thumb, and then finally release the 'h' key followed by the 'e' key. This should result in "he" being typed, but usually results in an "e". Indev assumes that a pointer release will always be for whatever the last position was. This is reasonable for mice but not for multitouch displays. This workaround detects this scenario and inserts two dummy events, first a "pressed" state with the coordinates of the first finger (the 'h' key), then a release event, triggering the keypress, then it sends another pressed events with the position of the second finger (the 'e' key) so that if the next event is that finger releasing, the position will be correct.

calebccff · 2023-07-11T14:50:56Z

Rebased on master and picked up Johannes' fixes (Thanks a lot for that!). Should be good to go! Tested again with unl0kr.

I added a new commit to properly handle typing with two thumbs by injecting some fake events, with this you can type as fast as you want with your thumbs and not miss an input which is really nice for unl0kr. See the commit description for more details.

I get that this is a bit of a hack, I'm happy to drop it if it blocks the rest of the series.

indev/libinput.c

Johennes · 2023-07-14T19:49:58Z

Rebased on master and picked up Johannes' fixes (Thanks a lot for that!). Should be good to go! Tested again with unl0kr.

I added a new commit to properly handle typing with two thumbs by injecting some fake events, with this you can type as fast as you want with your thumbs and not miss an input which is really nice for unl0kr. See the commit description for more details.

I get that this is a bit of a hack, I'm happy to drop it if it blocks the rest of the series.

The slot stuff is clever!

One possible issue I'm seeing is that you're writing state->slots from both the polling thread and the LVGL input handler. The locking is avoiding concurrent writes. But depending on how fast events enter and leave the queue, I think it's possible for later multi-touch events to already have overridden the slots array filled by earlier ones before you could use it?

I guess generally it'd be nice if LVGL itself supported multi-touch but the event injection strikes me as an acceptable workaround until then (especially given that we already do some event rewriting with is_relative).

Johennes · 2023-07-23T20:09:45Z

indev/libinput.c

+      evt->point.y += libinput_event_pointer_get_dy(pointer_event);
+      evt->point.x = LV_CLAMP(0, evt->point.x, drv->hor_res - 1);
+      evt->point.y = LV_CLAMP(0, evt->point.y, drv->ver_res - 1);
+      evt->is_relative = true;


One more issue that I just found is that the code currently doesn't support dragging anymore. E.g. if you click and hold but even just slightly move the cursor, that triggers a release because we're not setting evt->pressed here.

Unfortunately, libinput_event_pointer_get_button_state(..) doesn't seem to work for motion events. So we cannot use it here.

We could maybe define two relativity flags, is_relative_position (which would mean add up x and y to the latest state) and is_relative_button (which would mean inherit the pressed value from the latest state). LIBINPUT_EVENT_POINTER_MOTION events would then require both is_relative_position and is_relative_button while LIBINPUT_EVENT_POINTER_BUTTON events would only require is_relative_position.

Does that make sense?

I haven't tested it but I just added a pointer_button_down property to libinput_drv_state_t, had that attribute be updated on LIBINPUT_EVENT_POINTER_BUTTON and set evt->pressed based on that. I also made LIBINPUT_EVENT_POINTER_BUTTON fallthrough to LIBINPUT_EVENT_POINTER_MOTION so that evt->point is set.

I think that should work?

Nice! Yes, that seems to work. This now actually makes me think we could do the same for the delta movements. Instead of doing the weird is_relative thing when consuming events, just store last position we've seen in state and add the delta from libinput to it.

Have made it so. This thread can be resolved now.

Johennes · 2023-07-23T20:30:13Z

One possible issue I'm seeing is that you're writing state->slots from both the polling thread and the LVGL input handler. The locking is avoiding concurrent writes. But depending on how fast events enter and leave the queue, I think it's possible for later multi-touch events to already have overridden the slots array filled by earlier ones before you could use it?

Actually, I think you can work around that by doing the event injection when reading from libinput as opposed to when consuming the cached events. The slots array is guaranteed to only be filled by events preceding the current one at that point.

… make it thread-safe

Johennes · 2023-07-27T19:18:20Z

Actually, I think you can work around that by doing the event injection when reading from libinput as opposed to when consuming the cached events. The slots array is guaranteed to only be filled by events preceding the current one at that point.

This has been implemented, too, now.

Johennes · 2023-07-27T19:30:12Z

@kisvegabor this is now done from our end and ready for your review. I've tested it in unl0kr with touch, pointer and keyboard devices. Once we make another unl0kr release, I'll start working on porting this into lvgl master.

This enables fast typing on hardware keyboards as keys are no longer forgotten.

hack for hardware keyboard

Johennes reviewed Jun 5, 2023

View reviewed changes

calebccff force-pushed the caleb/unl0kr-disp-input-fixes branch from 131c16c to 0cfcdbb Compare June 5, 2023 13:32

kisvegabor reviewed Jun 5, 2023

View reviewed changes

display/fbdev.h Outdated Show resolved Hide resolved

Johennes reviewed Jun 5, 2023

View reviewed changes

indev/libinput.c Show resolved Hide resolved

indev/libinput.c Show resolved Hide resolved

indev/libinput.c Outdated Show resolved Hide resolved

calebccff force-pushed the caleb/unl0kr-disp-input-fixes branch from 0cfcdbb to f0f053a Compare June 24, 2023 22:10

Johennes suggested changes Jul 7, 2023

View reviewed changes

indev/libinput.c Outdated Show resolved Hide resolved

indev/libinput.c Outdated Show resolved Hide resolved

indev/libinput.c Outdated Show resolved Hide resolved

indev/libinput.c Outdated Show resolved Hide resolved

This was referenced Jul 8, 2023

Only call libinput_event_get_touch_event on touch events calebccff/lv_drivers#1

Open

Initialise mutex in libinput_init_state calebccff/lv_drivers#2

Open

Handle pointer click and motion events using delta logic calebccff/lv_drivers#3

Open

calebccff and others added 6 commits July 11, 2023 15:33

display: fbdev: add force_refresh quirk

6f4d2de

This is needed specifically for Qualcomm devices using DRM fbdev emulation, it is probably useful for other platforms using fbdev emulation.

Only call libinput_event_get_touch_event on touch events

4b61d7c

Initialise mutex in libinput_init_state

30519f0

Handle pointer click and motion events using delta logic

af1c040

calebccff force-pushed the caleb/unl0kr-disp-input-fixes branch from 7a1f200 to 878388c Compare July 11, 2023 14:38

Johennes reviewed Jul 14, 2023

View reviewed changes

indev/libinput.c Outdated Show resolved Hide resolved

Johennes reviewed Jul 14, 2023

View reviewed changes

indev/libinput.c Outdated Show resolved Hide resolved

Johennes mentioned this pull request Jul 23, 2023

Re-fix delta logic for pointer devices calebccff/lv_drivers#4

Open

Johennes reviewed Jul 23, 2023

View reviewed changes

calebccff and others added 5 commits July 24, 2023 21:28

libinput: fix pointer always releasing on motion event

5a70147

Eliminate is_relative in favor of storing last known position

181f593

Also set pressed on absolute pointer motion events

953c664

Move dummy event logic for multi-touch into the libinput read loop to…

1c4d998

… make it thread-safe

Handle the case where the second finger is released before the first

805a7f5

Johennes mentioned this pull request Sep 30, 2023

lv_dropdown sends event for the wrong item id lvgl/lvgl#4605

Closed

4 tasks

Johennes and others added 3 commits September 30, 2023 20:28

Set point on touch release events

7e3135b

libinput: release keyboard keys in lvgl directly after pressing

36d981a

This enables fast typing on hardware keyboards as keys are no longer forgotten.

Merge pull request #5 from jane400/jane/unl0kr-keyboard-hack

71dbdf0

hack for hardware keyboard

Johennes mentioned this pull request Jan 25, 2024

feat(drivers): Add libinput/xkb driver lvgl/lvgl#5486

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Libinput and fbdev improvements for unl0kr #280

Libinput and fbdev improvements for unl0kr #280

calebccff commented Jun 4, 2023

Johennes left a comment

kisvegabor left a comment

calebccff commented Jun 24, 2023

Johennes commented Jun 30, 2023

Johennes left a comment

calebccff commented Jul 11, 2023

Johennes commented Jul 14, 2023 •

edited

Loading

Johennes Jul 23, 2023

calebccff Jul 24, 2023

Johennes Jul 26, 2023

Johennes Jul 26, 2023 •

edited

Loading

Johennes commented Jul 23, 2023

Johennes commented Jul 27, 2023

Johennes commented Jul 27, 2023 •

edited

Loading

Libinput and fbdev improvements for unl0kr #280

Are you sure you want to change the base?

Libinput and fbdev improvements for unl0kr #280

Conversation

calebccff commented Jun 4, 2023

Johennes left a comment

Choose a reason for hiding this comment

kisvegabor left a comment

Choose a reason for hiding this comment

calebccff commented Jun 24, 2023

Johennes commented Jun 30, 2023

Johennes left a comment

Choose a reason for hiding this comment

calebccff commented Jul 11, 2023

Johennes commented Jul 14, 2023 • edited Loading

Johennes Jul 23, 2023

Choose a reason for hiding this comment

calebccff Jul 24, 2023

Choose a reason for hiding this comment

Johennes Jul 26, 2023

Choose a reason for hiding this comment

Johennes Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

Johennes commented Jul 23, 2023

Johennes commented Jul 27, 2023

Johennes commented Jul 27, 2023 • edited Loading

Johennes commented Jul 14, 2023 •

edited

Loading

Johennes Jul 26, 2023 •

edited

Loading

Johennes commented Jul 27, 2023 •

edited

Loading