DiscoBSD on Teensy 3.5 development board (NXP Kinetis K)

[onre]

I've got one of these:

https://www.pjrc.com/store/teensy35.html

I've been writing a toy OS for it but then I came across RetroBSD and then DiscoBSD. I figured that as the hard work here has already been done by others, it'd probably be pretty trivial to get it going on another Cortex-M based board, even if the peripherals differ a bit.

So, now I spent a long weekend mashing bits and pieces together, and got it to actually show some pretty strong signs of life. Thanks to both the Teensy support libraries and the stm32 port of DiscoBSD being well structured and easy to read, this was actually rather easy to achieve.

20241119 02:16:14    connected to /dev/ttyACM0    press C-z q to exit

!
DiscoBSD 2.3-current (TEENSY35) #130 558: Tue Nov 19 02:16:08 2024
     esp@tieteislaskin:/sys/mk64/teensy35
cpu: MK64FX512, 120 MHz, bus 60 MHz
oscillator: unknown
panic: No root filesystem found!
no fs on dev (0,1)
syncing disks... done
halted

The exclamation mark is a debug print from an early bootup stage - I had somehow managed to skip a comment in the Teensy library sources about having to wait for a moment before the USB UART is actually ready to use.

Next I'll write a driver of some sort for the SD card and we'll see what happens then.

[chettrick]

Hey there,
Congratulations on getting it all set up and signs of life for the Teensy 3.5 board! It is definitely a non-trivial task.
If you go through my old commits you can see the steps I took to get to the current state for stm32. I tend to be verbose and explicit with commit messages to aid others who may go back and glean information from them.

It seems that the Teensy 3.5 is out of production with no stock left from distributors (I checked pjrc.com, Mouser, and Digikey).
I do not have one of these development boards, so I will not be able to do any testing or development at the moment, but I can provide insight for finding ways around the codebase and how I implemented the DiscoBSD/stm32 port.

I would very much like to architect the codebase to be able to support many more ports and targets, without blowing up the binary sizes. My current direction was on implementing support for ARMv6-M (Cortex-M0+) so that the Raspberry Pi Pico RP2040 can be supported.
The NetBSD and OpenBSD codebases are good examples of supporting multiple ports. I do not use or reference any Linux code, as this is all BSD or BSD-friendly licensed code.

I heavily relied on the STM32F4 cube HAL and LL libraries for the stm32 port. Primarily, as I am familiar with them, and they are BSD licensed (hard requirement for inclusion), so there will be some work in generalizing the barrier between vendor code and kernel code in making the codebase able to support many ports and targets without getting unwieldy.

How is the SD card driver coming along?

[onre]

Hi,

Given how the PJRC-written Teensy support code is laid out, getting this going on the current Teensy 4.x would probably be rather trivial. I might grab one from the upcoming sale.

I used to participate a bit in NetBSD development more than 20 years ago so I am more or less aware of what you describe.

SD driver works now, right now I'm past the first fork() and can turn off the onboard led from the userspace. However, I'm probably overwriting the USB UART buffers or other control data with something. I'll figure this out soon enough, will probably go find a TTL USB UART so that I can just use the hardware UART - I was lazy and just did a quick hack to get USB virtual UART console without having to go find the hardware equivalent.

Debugging with proper tools would probably make the process faster, all I have are printf() and the LED.

Here's what it says now, complete with all sorts of debug prints:

20241121 04:10:25    connected to /dev/ttyACM1    press C-z q to exit


DiscoBSD 2.3-current (TEENSY35) #88 562: Thu Nov 21 04:10:17 2024
     esp@tieteislaskin:/sys/mk64/teensy35
cpu: MK64FX512, 120 MHz, bus 60 MHz, MPU enabled, 12 regions
mpu: region 0: 00000000-ffffffff, aac 0361f7df
oscillator: oscillating
usbuart0: USB, console
sd0: port sdio0
sd0: type SDHC, size 7822336 kbytes
bcopy (1fffe71e, 1fff6d38, 64)
sd0a: partition type b7, sector 2, size 204800 kbytes
sd0b: partition type b8, sector 409602, size 2048 kbytes
sd0c: partition type b7, sector 413698, size 204800 kbytes
bcopy (1fff4fc8, 1fff0ab0, 1024)
bcopy (0001379c, 1fff0e84, 27)
bcopy (0001379c, 1fff0f12, 1)
bcopy (00013794, 1fff0eb8, 4)
phys mem  = 256 kbytes
user mem  = 192 kbytes
root dev  = (0,1)
swap dev  = (0,2)
root size = 204800 kbytes
swap size = 2048 kbytes
bcopy (1fff4c18, 1fff5cb0, 28)

[chettrick]

Given how the PJRC-written Teensy support code is laid out, getting this going on the current Teensy 4.x would probably be rather trivial. I might grab one from the upcoming sale.

Teensy 4.x is Cortex-M7. There might be a bit of work to do for context switching (kernel setjmp and longjmp) for the M7.
M7 asm instructions should be a superset of M4 instructions, but I haven't verified that. M0+ is missing a lot of M4 instructions, so I have had to implement some of libc specific to either M0+ or M4. Still more work to be done.

I used to participate a bit in NetBSD development more than 20 years ago so I am more or less aware of what you describe.

Good good. NetBSD does a lot of things right, and we can freely learn from their work.

Debugging with proper tools would probably make the process faster, all I have are printf() and the LED.

Are you using GDB and OpenOCD and some sort of jtag adapter?

cpu: MK64FX512, 120 MHz, bus 60 MHz, MPU enabled, 12 regions

Are you setting up the MPU?

[onre]

Given how the PJRC-written Teensy support code is laid out, getting this going on the current Teensy 4.x would probably be rather trivial. I might grab one from the upcoming sale.

Teensy 4.x is Cortex-M7. There might be a bit of work to do for context switching (kernel setjmp and longjmp) for the M7. M7 asm instructions should be a superset of M4 instructions, but I haven't verified that. M0+ is missing a lot of M4 instructions, so I have had to implement some of libc specific to either M0+ or M4. Still more work to be done.

M4 and M7 are both Armv7-M, biggest difference from ISA standpoint seems to be M7 supporting a double-precision FPU whereas M4 is limited to single-precision. FPU is optional in both, many implementations don't have it but the particular Kinetis variant I have has it.

I used to participate a bit in NetBSD development more than 20 years ago so I am more or less aware of what you describe.

Good good. NetBSD does a lot of things right, and we can freely learn from their work.

...there are right ways of doing things?

Debugging with proper tools would probably make the process faster, all I have are printf() and the LED.

Are you using GDB and OpenOCD and some sort of jtag adapter?

Unfortunately I mean what I wrote. Though I did hook up a few more leds, way easier to debug the UART driver that way.

Truth be told, I do have a STM32 Blue Pill on my desk which I've been meaning to flash with DirtyJTAG for quite some time but haven't gotten around to that. This is what getting used to poor tools does to people's minds.

cpu: MK64FX512, 120 MHz, bus 60 MHz, MPU enabled, 12 regions

Are you setting up the MPU?

I paid - probably at least $0.03 - for the MPU so I'm damn sure using it too. :) This particular chip powers up with MPU enabled and configured so that all of the address space is region 0 as seen in the dmesg - all it does is granting or denying DMA bus master permissions. I tried protecting the first 56 kB as a separate supervisor-only region and that seems to work. I also hoped that I could somehow abuse it for some crude form of actual address translation but seems like there isn't such an option. However, using it for the purpose it's designed for would be a good addition for any system that has it, I'd say.

[onre]

Looks like my 5 am posting failed to contain things I meant to put in such as "thank you for encouragement and the code!"

We are now deep in userspace, running into things like this:

20241121 08:46:15    connected to /dev/ttyACM0    press C-z q to exit


DiscoBSD 2.3-current (TEENSY35) #62 568: Thu Nov 21 08:26:37 2024
     esp@tieteislaskin:/sys/mk64/teensy35
cpu: MK64FX512, 120 MHz, bus 60 MHz
oscillator: oscillating
usbuart0: USB, console
sd0: port sdio0
sd0: type SDHC, size 7822336 kbytes
sd0a: partition type b7, sector 2, size 204800 kbytes
sd0b: partition type b8, sector 409602, size 2048 kbytes
sd0c: partition type b7, sector 413698, size 204800 kbytes
phys mem  = 255 kbytes
user mem  = 191 kbytes
root dev  = (0,1)
swap dev  = (0,2)
root size = 204800 kbytes
swap size = 2048 kbytes
fault type: 0x4
fault 9: precise data bus access, BusFault
fault trap frame:
 r0:    0x1fff53c4      ip:     0x1fffffa8
 r1:    0x1ffff400      lr:     0x00001a07
 r2:    0x1fff5e28      pc:     0x000019a2
 r3:    0x00013ab4      psr:    0x4100000e
fault status registers:
 hfsr:  0x00000000      cfsr:   0x00008200
fault address registers:
 mmfar: 0x7ffeb354      not valid
 bfar:  0x7ffeb354      valid
fault entry EXC_RETURN value:
 lr:    0xfffffff1

Need to read up on the ARM fault types. The ARM documentation has been excellent so far - thorough and so far I at least think I've understood everything I've read. In general, this has been - and still is - a fun learning exercise. I do have some knowledge of similar concepts with other architectures but two weeks ago I knew nothing specific about ARM architecture other than "umm, it's a RISC so probably lots of registers" and my command of ARM assembly was limited to "does this thing accept LDA like 6510 did?"

edit; the discrepancy between 256/192 and 255/191 kB of RAM is a funny one. Back when these chips were new, Freescale had an incredible lapse where they mistakenly described the MK64FX512 as having 192 kB of RAM and this had been stated as a fact in the chip-specific reference manual all that time[1]. So, because of a lot of code for these devices already existed out there, this was solved in Teensyverse so that _estack is at 0x2002fff8 in 3.5 firmware binaries so that the loader utility can distinguish them from the 256-kB-since-day-one 3.6 binaries, which have _estack at 0x20030000 as one would expect. To gain back control of those 8 bytes I worked my way around this, but opted to go back to "stock" for now because of the convenience of being able to use the GUI loader application.

[1] https://community.nxp.com/t5/Kinetis-Microcontrollers/What-addresses-map-to-SRAM-in-the-K64-sub-family/m-p/720301
[2] https://forum.pjrc.com/index.php?threads/does-the-teensy-3-5-have-256k-of-ram.49522/

[onre]

Almost there. Sorted out the alignment issues in the SD driver, possibly found a subtle bug in the IPL-to-ARM priority juggling and learned a lot about the kernel internals. Right now the quick'n'dirty nature of the USB console driver I wrote is starting to bite me. If I cause enough output after /bin/sh is started, it will print the shell prompt, amongst other things, but input is unreliable. I'll write a proper version of the driver tonight and start cleaning up the tree & the commit history.

I envy people capable of working in a structured, professional manner. My method consists of creating chaos, then making sense of that chaos and, in the end, attempting to make everything look like there never was any chaos.

edit; license-wise, I'll make a couple of confirmations but I believe everything I have in my tree is BSD, MIT or equivalent. Also, as a disclaimer and general rule, everything I write is MIT - has always been and will always be.

[chettrick]

What is the subtle priority bug that you found? I couldn't find it in your commit diffs.

[onre]

It's not a branch of any decent kind, basically just an online backup of what I have locally. My usual working method is to first get something going, then apply changes in a decent, controlled manner one-by-one. I know it's not optimal but I blame pre-git version control systems where merge could take half a day.

[onre]

Oh and the bug itself was mine, not yours.

[onre]

Yay userspace!

DiscoBSD 2.3-current (TEENSY35) #2 588: Fri Nov 29 11:17:32 EET 2024
     esp@tieteislaskin:/sys/mk64/teensy35
cpu: Kinetis K64, 120 MHz, bus 60 MHz
mpu: enabled, 12 regions
mpu: region 0: 0x00000000-0xffffffff flags 0x0061f7df
oscillator: oscillating
uart1: found, 64 tx/64 rx bytes FIFO, console
sd0: port sdio0
sd0: type SDHC, size 7822336 kbytes
sd0a: partition type b7, sector 2, size 204800 kbytes
sd0b: partition type b8, sector 409602, size 2048 kbytes
sd0c: partition type b7, sector 413698, size 204800 kbytes
phys mem  = 255 kbytes
user mem  = 191 kbytes
root dev  = (0,1)
swap dev  = (0,2)
root size = 204800 kbytes
swap size = 2048 kbytes
alive for = 1833 ms
init: starting /bin/sh
erase ^?, kill ^U, intr ^C
# ps aux
USER       PID NICE SZ TTY  TIME COMMAND
root         1   0  37 0    0:00  (-)
# ls -l /
total 12
lrw-rw-r--  1 root           13 Nov 29 01:02 .profile -> root/.profile
drwxrwxr-x  2 root         1024 Nov 29 01:02 bin
drwxrwxr-x  2 root         1024 Nov 29 01:02 etc
drwxrwxr-x  2 root         1024 Nov 29 01:02 home
drwxrwxr-x  2 root         1024 Nov 29 01:02 root
drwxrwxr-x  2 root         1024 Nov 29 01:02 sbin
drwxrwxr-x  9 root         1024 Nov 29 01:02 usr
drwxrwxr-x  6 root         1024 Nov 29 01:02 var
# 

Next, cleaning up all the debug cruft. Then, do the one-by-one multi-commit/merge "rewriting the history" operation so that the posterity will think it was all done in a disciplined, well thought out manner.

Bonus pic of the debugging setup:

Left to right: fault led, eight-bit "show me whatever I want to see" register, three bits of SPL, programming button, reset button, user-defined interrupt button. The piece of cardboard protecting the MCU is courtesy of local bakery.

[onre]

Multiuser with a small quirk - need to push the interrupt button twice to cause an interrupt and have the startup sequence proceed. I know what's wrong with it, need to fix it later tonight. Got a long weekend here because of a national holiday on Friday, this means good things for the port.

How reliable are the UARTs with supported hardware?

[chettrick]

The UARTs on the stm32 boards seem good, and I haven't had an issue with the pic32 boards I have tested.
Sometimes there may be a dropped character or poor formatting in the terminal (say, output from grep or tar, or just catting a long file) but it doesn't seem to happen when redirected to a file. Not sure why that is.

[onre]

Sounds like exactly the behaviour I'm getting. These should be rock solid at 115200 bps (and more) running something like SLIP or PPP, but they are not. I've also noticed that disk I/O is a part of the problem, and if I turn the interrupt priority around (splbio over spltty), UARTs work better but disk I/O fails when there's terminal output.

I need to look into this in detail, after all the serial ports are the lifeline of a small Unix system so they must work flawlessly. I have a pic32 board of some sort which could probably run DiscoBSD to have a reference point. It's a "USB starter kit II" with a "Graphics LCD Controller PICtail Plus SSD1926 Board", the latter has an SDMMC slot. I'll look into it, got the board years ago and haven't ever powered it up.

[onre]

Also as a side note, got rid of the interrupt button nonsense. There was an execution path which did not resume the previous spl and thus we ended up in a silly race.

[onre]

Been running with MPU enabled for some time now. No idea how any of this would be done in the CMSISverse, but if nothing else, this could be thought of as an example of a configuration that is known to work. MPU is set up as the first thing in startup() and everything works so far. I think I'll refine the configuration somewhat, not happy with the one region I had to spend for that one piece of code that's technically user mode but is run from flash. Anyway, here's the functionality:

https://github.com/onre/discobsd/blob/master/sys/mk64/mpu.c
https://github.com/onre/discobsd/blob/master/sys/mk64/mpu.h

It does prevent silly things like this one from crashing the system - instead a bus fault is printed on the console. Added the relevant MPU registers to fault.c, though there is still some cruft in there as well.

I'm thinking of attempting to use the MPU for process isolation as it does support 8-bit process IDs for user mode, and with this config there are seven unused MPU regions left to configure. I paid for all of those regions.