Allow user to submit user-space callback function into io_uring worker thread.

[YoSTEALTH]

iovec   iov
timeout ts

int my_callback(iovec):
    // do something with iovec data here

    // either job is done, error or timeout.
    return 123

// first read data
sqe = io_uring_get_sqe(&ring)
io_uring_prep_readv(sqe, &iov, ...)


// let `io_uring` crunch the data using multiple worker thread.

sqe = io_uring_get_sqe(&ring)
io_uring_submit_callback(sqe, my_callback, &iov[0], &ts)

sqe = io_uring_get_sqe(&ring)
io_uring_submit_callback(sqe, my_callback, &iov[1], &ts)
// note: `ts` will be handled by `io_uring_submit_callback` function

// submit and wait
cqe.res  // 123

This is just a ruff sketch. Allowing user to submit a callback function into io_uring will solve a lot of issues currently users are having. This literally makes kernel side utilize most of the cpu and make user-space more of a managerial area!. Now we don't have to split resources into kernel and user-side workers, only kernel side. Possibilities becomes endless...

[axboe]

I think you'd need the BPF io_uring patches for this, don't see how else you could do something like this.

[YoSTEALTH]

I think you'd need the BPF io_uring patches for this, don't see how else you could do something like this.

I am not really sure what that would entail.

The way I was thinking about it was that io_uring should already have all the feature required, its only matter of adding a io_uring_prep_callback (or w/e name) and have it execute my_callback function inside it in worker thread. Allowing user to use their function as async + await feature.

Lets take io_uring_prep_read for example... when user submits an sqe for it, io_uring already has personality/limits set for it so only that privileged user can read from it. This could be same for my_callback function as well, only that users privilege is applied for it. Also other personality can be set using sqe->personality = 123.

@axboe I just hope you can see how this feature can change the way computing will work in the future. If you think about it... we could have multiple user-space software running and submitting work into same io_uring workers

If there are some kind of technical limitation, do mention it here, so we can further flush out the idea.

[isilence]

@YoSTEALTH, the problem here is that io-wq runs in the kernel space and obviously it can't execute arbitrary userspace code.

One way is to make the thread to switch into the userspace and execute it, but the overhead on switching will most probably be intolerable. Another way, mentioned by Jens, is to execute it in the kernel space but limit what programs can be run, take a look what eBPF is. It was actually tried out, see [1], though it doesn't bring as much performance as we'd wish, and so got frozen until a better moment.

[1] https://lore.kernel.org/io-uring/[email protected]/

[YoSTEALTH]

Aww, I should have guessed you guys would have already thought of such ideas ;)

YoSTEALTH, the problem here is that io-wq runs in the kernel space and obviously it can't execute arbitrary userspace code.

Right

One way is to make the thread to switch into the userspace and execute it, but the overhead on switching will most probably be intolerable.

Good to see you have already tested this.

Another way, mentioned by Jens, is to execute it in the kernel space but limit what programs can be run

Yes, this is more or less what I am suggesting as well. Make a callback function so limited that it can either only be sent data previously processed using other io_uring_prep_* function and only make it manipulate data, no calling anything external, maybe limited to string*.h or str* functions. That said limiting really tend to be a pain since someone will come around and say they want to have time.h in their callback and so on it goes...

Another approach would be something like io_uring_prep_regex (though not a callback). If we look at this in action:

// in liburing

sqe = io_uring_get_sqe(&ring)
io_uring_prep_recv(sqe, sock_fd, buffer, ...)

sqe = io_uring_get_sqe(&ring)
io_uring_prep_regex(sqe, 'my-regex-syntax', buffer, ...)
// here we could be parsing http header info received.

// in io_uring - run this code in its own thread till the task is done.

int io_uring_prep_regex(regex, buffer):
    // here we would be manipulate buffer with regex
    return 123

Most of the time when people are trying to use threads its normally to crunch data. With such limited type of workload passed to kernel side, there shouldn't be a problem?

take a look what eBPF is.

Wrapping a eBPF would be way too much work at this point, since I am pretty much close to done with uring project.

though it doesn't bring as much performance as we'd wish, and so got frozen until a better moment.

I see, seems like you did a lot of work on it! Sorry, I can't comment much on eBPF project.