Upgrade to SDK 3.0

[marcelstoer]

https://github.com/espressif/ESP8266_NONOS_SDK/releases/tag/v3.0

Memory optimization: iram can be used as memory, thus saving about 17KB memory for user application.

@nwf up for another SDK upgrade?

[nwf]

So, I started looking, and most of this goes through fine. There is, however, some additional complexity in that the firmware now expects to be told about a partition table. The examples derive this from static values known at compilation time, but AFAICT we don't have those, so I am tempted to do something crazy like ship in ROM a template of the partition table, copy it out to the stack, fill in the sizes based on the result from system_get_flash_size_map(), and then pass that to system_partition_table_regist. Opinions?

I'll try to get some code for everyone to look at this weekend, or a distress call if I brick my test board. ;)

[marcelstoer]

Seems they forgot to add the documentation about those partition tables before they tagged: https://git.io/fAtbU. They also forgot to update version.h - again. See espressif/ESP8266_NONOS_SDK#164

[nwf]

I am aware that this makes me be "that guy", but that's awfully thin documentation. The examples all, for example, pass static, constant data in to the function. I have no idea if this is required and can't look at the source, because hooray blobs. I'll prod at it this weekend.

[devsaurus]

Opinions?

After a brief read-through of https://git.io/fAtbU (thanks Marcel!) I would see the following implications when porting our firmware to the 3.0 partition table concept:

1. Automatic flashing of the init data in user_start_trampoline() must consider the address range of the SYSTEM_PARTITION_PHY_DATA partition.
2. Same applies to the documentation at SDK Init Data.
3. The SPIFFS code also has to consider the partition layout. Currently it assumes that the available address range begins after the firmware image and ends SYS_PARAM_SEC_NUM before end of physical flash.
4. How to handle our SPIFFS region? Not sure whether this should go into a customer partition table.
5. I assume that the flash size byte correction in nodemcu_init() can remain as it is.

My first shot would be to construct a partition table which mimics the current flash layout as we had it up to now. With this, you can safely ignore the above items for the time being and focus on getting the table in shape.

[nwf]

@devsaurus The partition table I attempt to register (but don't appear to make it far enough into boot to do so) is of the form you describe, I think? At least, I meant it to be.

[devsaurus]

Yes, I saw this as well when I read through your PR ;-)

[TerryE]

https://github.com/espressif/ESP8266_NONOS_SDK/releases/tag/v3.0

Memory optimization: iram can be used as memory, thus saving about 17KB memory for user application.

@marcelstoer, As I have mentioned before this is a terrible setting for nodemcu-firmware which runs out of flash. This 32Kb RAM segment is used as a L1 cache for flash instruction memory. Cache hits run at CPU clock speed. Flash misses require the cache line to be filled before the instruction can be executed. If you go from a 32Kb to 16Kb L1 cache and your miss rate goes from 10% to 30%, say, then your effective speed will fall by roughly 3×.

[devsaurus]

Memory optimization: iram can be used as memory, thus saving about 17KB memory for user application.

If you go from a 32Kb to 16Kb L1 cache and your miss rate goes from 10% to 30%, say, then your effective speed will fall by roughly 3×.

Free heap from @nwf's 3.0 branch with 32Kb cache is reported as 58880 for a float build with default configuration. It bumps up to impressive 80526 with reduced cache. There might be pitfalls ahead, but I'm tempted to propose that we provide this as a configuration setting at compile time (defaulting to 32Kb).

Since most of the CPU execution is within the Lua interpreter, a few (single?) performance measurement cases with representative Lua code should be enough to provide users with a rough indication what they have to expect as a trade off.

[nwf]

Heap numbers get even bigger if we were to ship with ./lua_examples/lfs/dummy_strings.lua in LFS, even if we don't otherwise ship an LFS image. It freed up about 2.5KB in my local build, without any further tuning. We could squeeze a few tens more bytes out the heap we also baked in the names of modules (and the types they create, like "cron.entry" and "crypto.hash") toggled on, it looks like.

Aside from that, I attempted to write a little benchmark to see the importance of the iram flash cache, and it's huge. The following program will run 180 loops of "busy" in the 2 seconds allotted for the test on a 32K cache, but will reliably trip the watchdog timer on a 16K cache. This suggests to me that 16K just isn't enough for the Lua interpreter to fit. It's a little tricky to benchmark NodeMCU due to the need to return control to the SDK so often, thus the 1 msec timer and all that. The benchmark will never report above 2000, but as 180 is much lower, we can be sure we're not mostly waiting for time to pass and really are keeping the CPU busy.

n = 0
function busy()
  for i = 0, 1 do
    s = "s"
    for j = 0, 10 do
      s = s .. s
    end
  end
  n = n+1
end

bt = tmr.create()
bt:alarm(1, tmr.ALARM_AUTO, busy)
tmr.create():alarm(2000, tmr.ALARM_SINGLE, function() bt:stop() ; print(n) end)

[TerryE]

@nwf Nathaniel, dropping the ICACHE from 32K to 16K does have a usecase -- for example if you have a small C app that needs a lot of RAM for data collection or the like. But in the case of an interpreted dynamically typed language like Lua the execution patterns in the firmware are scattered and the performance impact of dropping it from 32K to 16K will be very significant, IMO.

Maybe we should do some proper benchmarks? You can still benchmark code with watchdog timer feeds in it, because we aren't trying to get data to the last few %, but more is the 16K version 20% slower, 100% or what.

There are other low hanging fruit in terms of memory savings -- for example the DNS code statically preallocates some big buffers than are rarely used.

[nwf]

@TerryE Oh sure, I understand the general impetus for dropping the ICACHE, I just don't think it's good for us. I'll leave "proper" benchmarking to someone who knows what they're doing. :)

Backing up a bit, I think it's probably best that we do another master drop with LFS and only then think about tidying up the move to SDK 3.0 (I am sure there are lingering nits, and maybe we should be actually using the partition table rather than ignoring it; I do not understand the gdbstub well enough to judge interplay with the SDK exception vector.). Sorry for doing this prematurely; is there anything I can do to help with the master drop?

[TerryE]

OK, picking up my last comment on #2468 (comment), I do think that there are some general issues that aren't about the specific implementation details, but more about approach and use that we need to reach consensus on before I and Nathaniel finalise the implementation.

• SDK 3.0 introduces a new void ICACHE_FLASH_ATTR user_pre_init(void) entry-point for doing initialisation needed before the Flash ROM is mapped. The mandatory element here is to invoke system_partition_table_regist() to declare the partition table. We already have a pre-init entry point albeit called user_start_trampoline(). IMO, there is no point in having two pre-init functions so all required pre-init code should be merged into a single user_pre_init() function.

• We need to decide what partitions go into the partition_item_t at_partition_table[]. Our partition table should include:

  • EAGLE_FLASH_BIN
  • EAGLE_IROM0_TEXT_BIN
  • NODEMCU_LFS_PARTITION
  • SPIFFS_PARTITION
  • SYSTEM_PARTITION_RF_CAL
  • SYSTEM_PARTITION_PHY_DATA
  • SYSTEM_PARTITION_SYSTEM_PARAMETER

• We might also need extra partitions for TLD certs and the eventual NodeMCU OTA bootstrap.

• Clearly the EAGLE_FLASH_BIN and EAGLE_IROM0_TEXT_BIN can only be determined as an output of the make process.

• However, we do have some flexibility on how and when we set the other parameters and allocate the storage.

  1. At one end of the spectrum we can extend the app/user/user_config.h (or other static config file) so that we configure the make to a specific flash size and the make generate a static built-time parameter table and even assembles a single bin file containing all initial partition data. The would make for simple initial provisioning for advanced builders who make and provision multiple devices.

  2. We could extend Marcel's PyFlasher concept to manipulate the copy of the partition table in the binary to select preferred flash size, LFS region, SPIFFS size, etc, as an on-host post build step. This would be relatively lightweight and only need the same Python components as esptool.py, and not the full build toolchain. Developers could still use Cloud-built images as a basis.

  3. We could move some of this code into pre_init and do this at runtime

  4. We could move some of this code into a NodeNCU IROM0_TEXT library function and do this at runtime.

At the moment we do a mix of (i), (iii) and (iv). Some of these parameters can be set in user_config.h, etc. for the make. Some are set in our pre_init function (for example the Flash is scanned to determine its length and the flash size is set accordingly. If the Init Data sector is not initialised then this is set to the correct default. Also the RF_cal sector is calculated at runtime.

All of this pre-init runtime code burns the limited 32KB TEXT_SECTION space and adds runtime on every start including deep sleep wake times. We should check whether design designs which triggered these decisions still apply (e.g. does SDK 3.0 now cope with an uninitialised Init Data?

Also my preference would be to move as much of the code as possible out of pre-init and move it the make, with an optional host Python pre-flash update or to make available a node.setParamTable() which would validate then overwrite the relevant sector in the FLASH_BIN ROM image and restart the ESP.

[TerryE]

Sorry guys, but getting my head around all of this is proving to be a total PITA. As far as I can see, the partition_table type was introduced for RTOS and is embedded in the ESP32 implementation. I can see the logic of extending its use into NodeMCU / nonOS SDK and that it is now mandated in ver 3.0. Even so in the case of the ESP8266 and in particular NodeMCU understanding its use and how it interacts with the rest of the architecture is not easy as none of this is properly documented. So some observations:

NodeMCU uses a non-FOTA type 1 image format

Type 1 images are mapped by the boot ROM using a standard header at 0x00000 offset in the flash. The formal is very simple: each section is prefixed by a 2 word (length, load addr) header. esptool loads these from this first (up to 64Kb) of flash, so on a current build that I am working on:

Segment	start	len	Flash start
iram1_0_seg	0x07a74	0x40100000	0x00000008
dram0_0_seg	0x008c4	0x3ffe8000	0x00007a84
.rodata	0x00088	0x3ffe88c4	0x00008350

What is mapped and when

The ROM bootloader simply copies these segments from ROM into the specified ESP8266 addresses and then jumps via an indirection at 0x3ffe8000 to the required start address in the image which in this case is user_start_trampoline at 0x40106074. This would be reverted to the SDK call_user_start for SDK 3.0 and this trampoline code is moved into user_pre_init function. This code runs before the SDK has mapped irom0_0_seg at flash offset 0x10000 to address 0x40210000 and enabled the ICACHE so:

• Calling any SDK or application functions that use irom0_0_seg mapped code or data will cause the load to crater.
• The 2×16Kb ICACHE address space (starting at 0x40108000) isn't being used for flash caching at this stage and is effectively unused / available RAM at this initialisation stage before flash caching is enabled. (The bootloaders use page 0 for scratch RAM during the load process and the esptool loader stub uses the second half of page 1, 0x4010E000 for 8K, to run its code from.)

So it is worth noting that at the moment we load initialisation code into dram0_0_seg but in fact there would be nothing to prevent us adding a small initialisation segment at ICACHE page 1 or page 2. The runtime impact is that this data needs to be copied on cold-start / deep sleep by the bootloader, but an option worth considering nonetheless.

Note that the SDK dram0_0_seg entry-point is call_user_start and we currently replace this by call_user_trampoline which then callscall_user_start (rather than an asm tailcall so wasting a stack frame). In SDK 3.0, call_user_start calls the ICACHE_FLASH_ATTR user_pre_init(). It then maps which then maps irom0_0_seg on return and calls user_init (typically in this segment) before finishing off SDK startup.

Use of the partition table

As discussed in the Espressif Partition Table reference, there are system partition types that SDK uses and customer ones which it ignores. The RF_CAL, PHY_DATA and SYSTEM_PARAMETER system partitions are defined and used. Any others seem to be ignored by the SDK and are for application convenience only. In particular, I can't find any evidence that any partition definition for irom0_0_seg is used. I suspect that if this is a non-OTA image (that is the OTA_1 and OTA_2 partitions are not defined) then the SDK simple defaults back to non-FOTA map and maps firmware offset 0x10000 to 0x40210000.

Flash autosizing

For various historic reasons we have designed our firmware to be resilient and support two main features that allow firmware builds to be loaded onto a range of ESP chips and which still robustly start.

• The image does need to be built with the correct flash size set. On initialisation, the start-code validates that the size and if necessary recomputes all related offsets and restarts the image with the correct settings

• The system parameter sector must contain the correct initialisation data; if this is blank (all-ones), then it is overwritten with a correct init_data content to prevent the SDK cratering.

Even though many advanced developers are comfortable with full toolchain builds, I have to admit the reality that most Lua IoT developers are not. IMO, we therefore need to be able to configure the partition table as a Python-based firmware load utility or through a runtime API node.setPartionTable() just as we support an LFS reflash API.

esptool gives you all of the low level utilities to wrap this and other sizing support into a simple script that updates a NodeMCU partition table at a known location to allow the developer to configure the image flexibly from a WinX dev PC without the full toolchain loaded. For example we could add this functionality to PyFlasher.

However, given @marcelstoer Marcel's comments about the volume of issues that we used to have with users having init_data problems, then a runtime option is also desirable -- but we don't want an implementation which will break our dram0_0_seg constaints. And as noted above this is possible.

Whatever we do, if we do this properly SDK 3.0 support isn't a simple patch

Thought? @jmattsson, @devsaurus, @pjsg, @nwf et al? I could go on, but I'd prefer to cover specific contentious issues in dialogue.

[NicolSpies]

Even though I am not involved in the detail covered above , some random thoughts while starting to research the TLS options of #2587:

• Defining the ESP8266 as a medium resource platform results in the possibility to implement and scale functionality up to a certain point while making compromises e.g. available functions, available memory, speed etc.
• Based on the assumption that latest SDKs are always an improvement on earlier SDKs, I assume it makes sense to implement and better the core and development tools.
• Increasing requirements vary depending what the user wants/needs to do inevitability resulting in trade off compromises.
• Living within the platform hardware constraints thus results in smarter ways of doing things (e.g. LFS, partition table etc.) to extend/stretch what can be done with the ESP8266.
• At some point the increasing needs/wants will intersect with the platform constraints. Should we define this point as an absolute when it is required to cross over to the larger less constrained platform ?

So, after this philosophical rambling, what am I actually saying?

A universal constraint most of us have is; what to spend our limited time on. So if it was possible to peg the maximum the ESP8266 could give and define where where we are now, we could somehow define how far we will go before accepting and living with the ESP8266 limitations.

So, from a TLS perspective, the possibility of an extra 17 Kb memory could be what is needed for TLS, obviously compromising something in the process.

So, consensus between individual perspectives will most probably result in a healthy "tug of war", dependent on individual perspectives, wants and needs.

[TerryE]

TLS ...

As you say, this is a separate issue, but I can't see anything that we do here materially helping with issue that our current TLS implementation is too RAM heavy for this class of CPU. Sorry

Based on the assumption that latest SDKs are always an improvement on earlier SDKs

The SDKs always seem to add functionality so the footprint grows. What Espressif have done is a series of optimisations to move some constant data from .rodata to Flash, but at best these seems to mitigate the overall trend of increasing resources.

[NicolSpies]

TLS..

I agree the current TLS implementation is too heavy. The problem is not unique to us, a number of options exists how to lighten the load...but not in this topic.

Rest of my ramblings..

I suppose the elegance will lie in how to slipstream the ESP32 in such a manner to have a common development environment while building on all the effort that has been spent / will be spend on the ESP8266.

[TerryE]

Returning to my post of the 6th, #2467 (comment), The firmware image format is documented in the esptool wiki, an as the File Header section describes, the esptool -fs nMB option during the write_flash``function (where **n** is the number of Mbytes flash capacity). This -fsoption can override the value in the staticmake` image. I see little advantage in adding another layer of runtime size detection.

This being said, because the -fs option can be used to change the flash size from that specified in the image, and the the 3 system partitions are placed relative to the flash size, we either need to be able to recompute the partition table (as a potential one-off) at runtime or alternatively have a esptool-wrapper which can adjust the partition table and assemble a complete image or even reload individual partitions using the partition table.

Another tweak here is that if we go the Python route then I think it sensible at some point to drop the -a option as the new LFS PIC format is trivial to decode and fix to an absolute address with a simple python function.

Taking the above example, the ROM boot-loaded section of the flash (essentially the 0x00000.bin bit) ends at 0x000083D7, so there are 15 unused flash pages from 0x00008400 - 0x0000FFFF we could easily fit an 8 page overlay into this region with a linked segment address of 0x40108000, say. We could remove this from the ROMboot image chain but leave in the flash, hence the bootloader would not load this for normal boot or deep sleep restart. However, if needed, the user_pre_init stub could read this from flash into 0x40108000 with a single SPIRead and enter this code. This would be a "no-return" call to adjust the parameter table, insert init_data into the system parameter partition, reload an LFS or even overwrite the firmware from a copy in SPIFFS (yes, proper OTA, at least via SPIFFS). On exit this code would then re-enter the ROM bootloader at _ResetHandler to reset the CPU and reload the updated image.

Anyway, that's my current thinking. Any objections?

[TerryE]

@jmattsson, @devsaurus, @pjsg, @nwf, @marcelstoer et al. I am a little surprised that I haven't received feedback from you guys as this might ultimately be a fairly high-impact PR. Anyway, I will do my usual and grind away and produce the PR -- or at least two PRs because I don't think that I should include firmware upgrade in the first PR.

[devsaurus]

Terry - very sorry. But to be honest, you lost me right from the beginning.

Apart from the complex technical implications you describe, I didn't understand why there is now a need to restructure things. SDK 3.0 forces us to provide this partition table so

• it can map/call our code (I'm not even sure about that)
• it knows where the rfcal and init data resides
• ?maybe other items?

Their Locations were kind of hard-coded up to now and we had to adjust the firmware to these boundary conditions imposed by the SDK. E.g. the annoying "init data at end of flash" thingy. To maintain status quo, the firmware now needs to provide such information to the SDK, fine. Once that's done, we have the same freedom to lay out the remaining flash memory range as before. Maybe I'm being a bit naïve here, but this seemed to be a viable first step.

Your posts read like you're looking into extending the firmware functionality by making use of things that a partition table would enable now. Not sure. It would be helpful for me to understand the urgency if you explained potential shortcomings of the approach in #2468.

I also didn't get the reason for:

IMO, we therefore need to be able to configure the partition table as a Python-based firmware load utility or through a runtime API

Which use case is enabled by this?

[marcelstoer]

Quite frankly I didn't get most of the expert talk here apart from the 32Kb vs. 16Kb L1 cache discussion.

[jmattsson]

Late response because this whole area is a minefield and I haven't had time to even vaguely context-switch into this space for a while.

Thoughts:

Partition tables good. Getting rid of our magically reserved pages would be a blessing. I'd recommend sticking all the system-required ones (RF / PHY / SYSPARAM) at the start of flash, before any software partitions. That should make life easier with regard to our flash-auto-size adjusting. If necessary, drop a tiny trampoline (a load and a jump) at the start that chains into the software partition. Also, limiting the Lua-visible partition api to only allow modifications of software and spiffs partition is probably a good idea. Anyone needing to tinker with the system partitions should be skilled enough to do a custom partition table at build time. Make it simple and safe for the common use case would be my preference.

user_pre_init dangerous. I'd need to actually look at the disassembly of that function to know what it does and whether it conflicts and/or depends on our user_start_trampoline. We might want/need to get the exception handler installed before chaining through to this new user_pre_init function. And yes, that might mean splitting the trampoline into two parts (and that might be a good time to dig up the old RTOS exception handler instead, to get the speed boost from the raw asm version without all the C thunking).

Having our once-off code generally not mapped is interesting. I'd see this a separate exercise for later optimisation though. It's likely to have "interesting" interactions with OTA, so I'd prefer to see this handled later, once we're on top of all the other changes. Premature optimisation being the root of all evil etc.

In general, keep stuff out of early init. The available RAM resources are precious, and it's all too easy to feature bloat. >.> Also, it can make OTA support more difficult.

Oh for real PIC. As far as I'm aware we can't do proper position-independent-code, so we'll still need to build for absolute addresses with a specific flash mapping in mind. This is of course a real drag for OTA where either you need to very carefully build slot-specific images of the same firmware version and then just as carefully pick the correct image to download and apply. What I did for $work (and RBoot also supports) is to designate meg 1 and 2 as the OTA partitions, with a small exclusion at the front of each where the loader itself lives. Then, switching between the two installed firmware versions is as simple as telling the chip to map either meg 1 or meg 2 to 0x4100000, and voila there's no need for per-slot images, since the logical addresses end up the same due to the flash mapping. With the partition support, we might loose a few more pages at the start, but that's not a huge deal in my mind.

I don't suppose the SDK OTA support has evolved to the point where you don't need to build individual images for slot 1 and 2?

Rebooting via reset-handler jump fragile. There are gremlins, and it's generally safer to trigger the watchdog for a "real" reboot :) Not saying "don't do it", just be vary of weird and latent bugs due to incomplete chip resets (mapping registers, internal h/w module states, etc).

Instruction cache as instruction cache by default. Having the option of switching the instruction cache to general RAM would be good, but I'm wary of the performance impact considering we've moved more and more towards running straight out of flash. Does the LVM provide an easy way of adding a memory pool? If so it'd be very nifty to be able to do something like node.icache2ram(1) and/or node.icache2ram(2) to (once) on the fly drop one or two banks of instruction cache, rather than needing to have it as a build-time option only.

At what point to we say no to SDK feature bloat? As you've mentioned, the SDK pretty much only keeps growing. Is there / will there be a point where we'll need to go "thanks, but no thanks"? Do we need to start lobbying Espressif for a more modular SDK for example?

[TerryE]

@devsaurus, IMO we should either use a partition table properly or not bother and stick with the current SDK. In this first case we should use a partition table for all logical partitions in the Lua firmware, including the LFS region, our SPIFFS region and our TLS certificate regions, and our libraries that need to reference these should use the partition table to locate and size them.

As I said we have one of three strategies for configuring the system parameter table:

1. This must be done at build time through a bunch of user_config.h parameters
2. It can be done at flash time. The partition table is at a fixed location in the image and so there is nothing to stop us having a wrapper around esptool to calculate and overwrite the partition table.
3. It can be done at runtime through a Lua API call.

I agree that it should be trivial for advantaged developers who are familiar with Xtensa toolchain to do this all at build time, However LFS has taught me that most of our users are Lua developers who are very reluctant to get into building firmware images themselves. If we use a partition table (PT) properly then there is absolutely no reason why the table has to be hard-coded into the compiled image. We could have a simple mechanism for Lua developers to take a cloud image and then configure it for a given Flash size, LFS region, SPIFFS partition, etc during the flash operation.

As to the current #2468 commit, it needs to handle system partition initialisation and flash resizing, plus the other NodeMCU partition placement, and this is going to take more iram1_0_seg and .rodata code and we are already bumping hard against the 32kB limit for these -- which is why I suggest moving this one-shot pre-cache enabled code into a small overlay segment to be loaded into the available 0x40108000 RAM slot. If we are going to have the LFS and SPIFFS code get their partition addresses from the PT, then we need to modify these to read the PT.

[TerryE]

@jmattsson, comments below using same / similar headings.

Partition tables good. Sticking all the system-required partitions (RF / PHY / SYSPARAM) at the start of flash, before any software partitions could be problematic, as the SDK doesn't use the partition table properly for non-FOTA loads. If you read the [Partition Table documentation](/espressif/ESP8266_NONOS_SDK/blob/master/documents/EN/ Partition Table.md) then you will see that the 0x00000 and 0x10000 partitions are not system partitions but are user defined "customer" partitions. This is because non-FOTA images are type 1 images that use the ROM bootloader to load the iram1_0_seg, dram0_0_seg .rodata segments starting at 0x00000.

I need to look at the rBoot source and disassemble the relevant SDK routine, but the SDK enables the instruction cache somewhere in the SDK function call_user_start() and this maps flash address 0x10000 -> 0x40210000 and on via the 32Kb instruction cache. Any attempt to reference irom0_0_seg constants and code before the cache is mapped will cause the SDK to crater. (Our trampoline code and the new user_pre_init() are executed before the cache is mapped).

I am not sure were the parameters of this mapping ( 0x10000, 0x40210000 and the length cached are defined, but they seem to be hardwired in the SDK. Our trampoline code and the new user_pre_init() are executed before the cache is mapped and so t this point.

user_pre_init is NOT dangerous. It is actually straight foward. It is a user exit for doing tampoline code that is called from call_user_init(). The mandatory element is that you must set the PT address using a callback, but the normal pre-cache-enabled rules apply. We can establist our unaligned exception handler here if we want instead of doing our current trampoline since all this code runs before the cache has been abled and can't use flash based constants anyway up to this point.

AFAIK, the SDK 3.0 still does not detect and initialise the system parameter partition with init_data, so out pre-cache enabled code will still need to do this.

I question whether we need the non-C exception handler. Switching to -O2 and the ROTable lookaside cache have already dropped unaligned exceptions by an order of magnitude.

Having our once-off code generally not mapped is interesting. The 32Kb hard limit for the iram1_0_seg is a total PITA to work within. It is really quite easy to break this limit in build. I see having an on-demand overlay for pre-cache-enabled code being a simple way to avoid this hassle. Note that if we do go into this overlay code then we don't return and continue with the SDK startup; instead we immediately restart the CPU.

Early init. The overlay apporach makes RAM for init no longer a scarse resource, so simplicity and clarity become the priority rather than bloat avoidance. If we have to configure the PT at start-up then it is best placed in RAM, so the code that initialises it must be in early init.

Johny, you might have missed my suggested approach to OTA in previous issues. we don't do FOTA directly but instead do FfromSPIFFS. The dev downloads the deflated image into SPIFFS and calls a Lua API call which will read and inflate the image back into Flash. With this approach, we don't need two partitions. This is a separate issue for later really, but this this overlay apporach is a steppping stone.

Real PIC. We don't have it and don't really need it on the ESP8266, IMO. I need to have a detailed chat with you about the best way to go here. We still have users with ESP-01-class devices (such as the Sonoff devices) with only 1Mb RAM. We should still support these, and this makes mandating a 1/2meg split problematic, IMO. However you can do FfromSPIFFS in a 1Mb flash part.

FfromSPIFFS is pretty resiliant just as LFS reload is. It's a flash-to-flash copy. There is no network dependency here. The application needs to use some mechanism to copy the new image (typically under 300Kb) over the network and onto SPIFSS and then call a node.reflash_firmware(filename) to do the reimage. If it fails then this will only be due to power fail. Whilst there are pathelogical cases where a powerfail during copy could crater the ESP this would require the corruption of a single 4kB page write within the copy. any other failure would simply retart the cpu and the copy.

Rebooting via reset-handler jump fragile. Noted. Will check and need to discuss.

Instruction cache as instruction cache by default. The Lua firmware runs like a total dog when there is only 16Kb flash enabled. I will leave someone else to bleed on this.

At what point to we say no to SDK feature bloat? The linker already does a level of feature triming: if you don't use an SDK function then the linker does not include it. Espressif have said that 3.0 is the last non_OS SDK release so it makes sense to move to it. RTOS is too memory hungry for the RTOS+Lua runtime to load with the ~48Kb RAM available and still ave enough left for apps.

[jmattsson]

@TerryE

Re partitions and loading: Yeah, that's why I was saying we might need a tiny trampoline that loads and jumps to the actual nodemcu firmware partition. Aside from RBoot you can also have a look at the loader we're using at work. It takes the place of the SDK's OTA loader for us (and has the automatic-rollback feature we need).

Re flash mapping: It's not just hardwired in the SDK, it's hardwired in the silicon I'm pretty sure :)

Re on-demand mapping into IRAM: I'm not recalling any way of mapping into the IRAM cache, I thought it was only the linker-and-therefore-the-bootloader (and explicit memmove()s) which got bytes in there. Am I forgetting something?

Re OTA approach: yes, I'm aware of that aspect. I'm just trying to keep the door open for the existing OTA method we're using at $work :) A pure LFS upgrade doesn't cut it for our needs, since we have a separate non-NodeMCU app which boots first, and only when needed chains through to the full NodeMCU.

Re instruction cache bleeding: I'll pass :) For now at least... #notime

Re last non-OS SDK: Have we looked at a recent RTOS and checked the current memory situation? Maybe they've managed to clean up the act sufficiently?

[TerryE]

@jmattsson, I realise that you understand most of the below, but I'll respond in detail for other readers.

Mapping into the IRAM cache. You can't and you don't need to. So long as the cache is not enabled, then this region 0x40108000:0x4010FFFFF is accessible as normal RAM and you can use this both for data and to run code from, so you can memmove or SPIRead() code and data directly into this. For example the esptool uses this technique and loads its own flasher stub into this region. (See the stub's ld file, which is what gave me the idea. Esptool uses this stub code essentially to add on-the fly inflation and some extra functions that are missing on the ESP8266.)

DUiS can't currently use this technique in its code, because (for some reason I don't understand) your loader enables caching and uses caching to map the source bytes from flash; this invalidates other use of the cache region. The ROM bootloader itself instead uses SPIRead() and this is just as fast for such serial flash-as-data access, this is just as simple to program. (See the ROM function boot_from_flash() 0x40001308:0x400017c9)

Looking at rBoot, you can programmatically select the 1Mb region of the flash to map, but the 0x4020000 base offset is as you say in the H/W. rBoot uses a symmetric boot ROM approach -- that is you can have N images each linked to its own base address and dynamically select which to boot at boot time. But the downside of this is that a region with flash offset 0x10000 with be addressable at 0x40210000, one at offset 0x70000 will be addressable at 0x40270000, etc.

IMO, it would make a lot of sense moving the system and any bootstrap / overlay partitions below the firmware as this would simplify the partition table layout, but I believe that the SDK essentially does a if (!FOTA) cache_enable(0x10000) hard-coded somewhere in its call_user_start and I am not sure how we could change it. The alternative is that the code uses and decodes the external reference _irom0_text_start. Need to do some disassembly to investigate.

Instruction cache bleeding. Surely this isn't an issue with my approach as the bootstap never enables the cache, establishes any interrupt services, etc. and I only call _ResetHandler from this code?

More details on my suggested reflash approach. This is still one option for futures, and a lot more discussion on the pros and cons is needed. So I've this content into another issue #2606, as the current PR only implements the changes needed for SDK3.0 support.

Related Issues and PRs

• Reimplement esp_init_data_default without hardcoded data: Reimplement esp_init_data_default without hardcoded data #1527
• Update our use of esptool.py: Update our use of esptool.py #2404

[TerryE]

OK, I've worked out how to shift the base of the irom0_0_seg from 0x40210000. This is described in the rBoot documentation. This mapping is done in the SDK routine Cache_Read_Enable_New() and as per the rBoot documention and code example, we need to replace this with out own version that maps this segment to 0x40214000 or whatever we need for the system partitions.

[TerryE]

I've been going through the SDK 3.0 changes and one of the side-effects of the expanding scope of the SDK, is that there are now overlaps between the Espressif maintained code and our own firmware, for example:

• As discussed in DON'T MERGE; first attempt at sdk 3.0 #2468, Espressif has added its own load_non_32_wide_handler() and @nwf has proposed that we drop our own maintained version for this.
• The SDK now includes maintained sources for lwip and mbedtls. These largely duplicate the versions that we forked at 2.0. Being a fork, we now have two diverging versions where we have made some small changes and where Espressif have done a range of memory optimisations and bug fixes which are not in our code.

My suggestion in this case is that we follow Nathaniel's example and adopt the SDK versions as these in general are better maintained than ours. Especially as they have lower memory footprints than ours.

My suggestion is that we should use by default the SDK versions of code where it exists and not maintain our own fork. We should apply a reasonably high NodeMCU requirement / benefits threshold on any patches that we deem justified and on an exceptional basis where our historic changes add essential NodeMCU specific functionality, then we should maintain an SDK overrides patch file within our repo and apply this to the SDK to layer such changes back onto the SDK code.

However, my instinct is that we should defer such renormalisation of the NodeMCU and Espressif repos to a separate PR.

[jmattsson]

@TerryE Sounds very reasonable. Just double-check that whatever it was we fixed in lwip is also fixed in the SDK version :D
If we have something they don't, at least these days it should be easy to submit a PR to them about it to get it in upstream.

[TerryE]

The non-cached iram1_0_seg is only 32Kb long. The other ROM boot loaded segments are small, so there's enough dead space to stick the rf_call, phy_data, sys_parm partitions at 0x0b000,0x0c000 and0x0d000 respectively. Seems to work fine. On my current test build this leaves about 11K headroom if I want pre-icache trampoline code, but the phy-data partition is the one that need to be initialised, so we should keep that below the line. The others could easily be moved to top of flash.

[TerryE]

@johny, as I said, the user_pre_init()code is called early in the boot sequence, and early enough IMO that is makes it worthwhile putting the rtctime_early_startup() hook here. This is already a bit of a botch as it enables icache, and having it here rather than trampoline hook make little difference. It seems that if you do want to hoist it to a true trampoline entry then you do need to have this and the code it executes properly inlined in non-cached code. There's nothing to stop you loading this into pre-icache segment avoiding any extra RAM overhead. This is a few hours work, but I will defer it for now.

[TerryE]

@jmattsson @devsaurus @nwf @marcelstoer, one of the features implemented in nodemcu_init() is an algo to scan flash to determine if the readable size of the Flash is consistent with the Flash header size field.

It does this by using (the only use) of the platform/flash_api.c:flash_detect_size_byte() routine. This climbs through flash pages, reading the last page in each power of 2 size until a timeout occurs. It them rewrites the flash header page with the correct flash size. This rewrite is unsafe and can brick the ESP.

Given that esptool can set the flash size at flash programming with the --flash_size parameter, I think that all of this complexity is overblown and should be stripped out of the firmware. The rule should be simple IMO: the flash size is specified in the build, but can be overridden by the correct flash size when downloading a new firmware image.

Comments?

[TerryE]

Another Q for all. At the moment, the LFS code is only compiled into the firmware if LUA_FLASH_STORE is defined. If defined, then it is used to allocate a flash store. I propose to change this so that the LFS code is always compiled in (it's only got a few Kb footprint), but the PT defines the size of the LFS (using LUA_FLASH_STORE as its default value). The LFS code is only enabled at runtime if the LFS size > 0. The will allow you to reconfigure the LFS at image load time as well as build time. Again comments?

[nwf]

@TerryE Eliminating the use of flash_detect_size_byte seems perfectly reasonable to me, as does always including the LFS code in the generated images. Do you intend for the hooks exposed to Lua to be hidden from the C API export tables or that they just fail if the LFS size is 0? On the topic of dynamism, I don't suppose it's easily possible to ask SPIFFS to shrink itself so that we could allocate a LFS partition "later"?

[TerryE]

I've put a base copy of the PT at the start of the 0x10000 segment. You can change the base address and sizes of LFS and SPIFFS in this. The startup will adjust defaults as needed. The #ifdef code is in effect replaced by a runtime size>0 test, but the overhead of this compared to some of the other recent optimisations that I've done is tiny.

[devsaurus]

The rule should be simple IMO: the flash size is specified in the build, but can be overridden by the correct flash size when downloading a new firmware image.

This suits the pro type user but might be an obstacle for the novice. flash_detect_size_byte saved us a lot of headaches back in the days when flashing tools weren't smart enough. This has changed since esptool.py can now autodetect the flash size, so flash_detect_size_byte is redundant if you assume a host with esptool.py. Users with other flash tools might hit a wall though and will need to switch to esptool.py or the NodeMCU PyFlasher.

This rewrite is unsafe and can brick the ESP.

I wasn't aware of this case - the assumed failure can't be resolved by a complete flash erase?

propose to change this so that the LFS code is always compiled in

Ok for me.

[marcelstoer]

The rule should be simple IMO: the flash size is specified in the build, but can be overridden by the correct flash size when downloading a new firmware image.

Objection. my2cents:

I may not fully understand all details the under the hood but I feel I need to defend the interests of the non-pro users (i.e. our largest user group). It's great esptool added auto-flash detection to remove one more obstacle on the way to a successful firmware flash process. The new auto-serial-port detection is also nice but not nearly as essential IMHO. esptool's auto-flash detection allowed me to remove one more field from the GUI of my PyFlasher.

Besides, how about flashing the same NodeMCU binary to multiple devices with different flash sizes? Works just fine now as I don't need to define the flash size anywhere in the process.

Requiring that parameter now would be a step backwards. Technology needs to be as accessible as possible in order to reach more than just a handful of nerds. Every parameter saved helps in achieving that goal.

[nwf]

@marcelstoer As I understand, since the flash tool detects the flash size at flash time (naturally enough), your concern about needing multiple binaries isn't really justified: the same image will be adjusted differently for each flash operation to each device.

[TerryE]

@marcelstoer, the one thing that we would need to be careful with is when we get around to having the ability to upgrade the firmware effectively OTA. In the case where the previous firmware has a given flash size and PT then the upgrade should inherit these. I also need to take a look at your PyFlasher code because it would be good to be able to set override the partition table parameters such as the LFS region size, SPIFFS, etc. as well as being able to (over)write SPIFFS and LFS or even have multiple alternative SPIFFS and LFS regions on the same module -- something that a number of people have requested. But these are all incremental to the core SDK 3 functionality as an enabler.

One other thing that I've got to think through are the implications of the extra RAM. When I first got involved with NodeMCU we had about 15Kb available for both code and data. It looks like Espressif have used their implementation of John's unaligned exception handler to move a lot of constant data into Flash, and I suspect that's the main reason that the SDK has freed up so much RAM, but I'll need to compare before and after mapfiles to verify this. Even so we now seem to have roughly 256Kb for code and 57Kb for data -- effectively an order of magnitude improvement.

[jmattsson]

@TerryE I'm not comfortable removing the automatic flash size fix-up. As Arnim already said, it really helped out a lot of non-expert users (yours truly included). If it can be completely superseded by other means then it might be okay, but even so I'd at least prefer to see a staggered approach of removal, by which I mean that we leave the test in, but rather than do the rewrite ourselves, only print a message for the user with pointers to how to do so. (At which point, it could still be argued we should just bloody well fix it ourselves - it's only on the first boot after flashing anyway). I've never bricked an ESP8266, even when I was mucking around with this particular area (and got it hilariously wrong on occasion) - an outside reflash always sorted things out. Forcing new users to be aware of and understand the importance of that flash size byte would not be a good thing. I do wonder whether the size byte matters at all once we have partition tables though? In the past it's been used by the SDK to find the reserved pages at end of flash...

Regarding always having LFS in and then enable/disable at runtime according to whether there exists an LFS partition, that sounds reasonable to me. It might still be worthwhile to have a master #ifdef to disable it for those who are still on 512k chips?

[nwf]

@jmattsson Do we still support the 512K chips with modern nodemcu versions (i.e. anything post 1.5.4.1?)

[TerryE]

@jmattsson my concern is that this can all sorts of unintended consequences. For example what this means is that if you have a 16mb part and you haven't specified a max SPIFFS size then you will have a ~15Mb SPIFFS whether you want it or not, and formatting a 15Mb SPIFFS takes ages; it's a PITA. In my case if I have a mix of Wemos D1 and D1 Pros, this means that I can't configure them all identically as 32m parts.

We are firmly recommending that users use esptool or its `PyFlasher``wrapper and this does a flashsize detect on download unless you explicitly override it. Even if you do explicitly override it then this patch means that the firmware will then undo this override on first boot -- Durrhhhh.

IMO, this is truly stupid behaviour that we should drop and explicitly describe how to do this in the documentation.

As far as LFS goes, you can define the region size to be 0. The extra LFS code itself is less than I've already saved by moving the 20Kb system partitions to 0x0B000 - 0x0ffff. (And also about the same saving as removing all this update flash size nibble functionality).

SDK 3.0 has moved a ton of stuff out of the iram1 segment into irom0. This frees up a load of RAM, but since irom0 starts at 0x10000, this means that the overall minimum flash size has grown, so 512Kb parts can't be supported for NodeMCU + SDK3.0. It was pretty much impossible to get a working 2.1 build with a 512Kb part but forget it in the case of 3.0.

I also came across an undocumented feature in node.flashsize(). According to the documentation it returns the flash chip size in bytes. What it doesn't tell you is that if you pass it a valid flash size then it will just overwrite the Flash header accordingly. No validation. No reboot. Yukkk!

[TerryE]

Oh, they keep coming out of the woodwork. @pjsg, I need your advice here. spiffs/spiffs.c has all sorts of variant formulae for settting the FS bounds depending on factors like:

• Is this an explicit or implicit format (on first boot)
• The flash size
• Whether the available flash is over 700,000 bytes or not
• Whether a hard start, max size and 1Mb aligned parameters have been set.

This logic is spread though various paths so it is difficult to understand and summarise. I want to be able to use the Partition table to allocated a reserved start and size to SPIFFS with in effect a single entry. I am happy for it to decided pages sizes, offsets, alignments, etc. but the algo should be simple and understandable, and stay within this assigned partition.

BTW, the spiffs code uses INTERNAL_FLASH_SIZE to size its FS, however this is defined in the SDK based on the FLASH_xxM define in user_config.h -- which is rather inconsistent with the Flash Auto / re - sizing that has been discussed above.

PS. I now have LFS using the partition table to allocate the LFS region, so you can adjust this at image flash without needing to rebuild the firmware. I want to be able to do the same for SPIFFS.

[pjsg]

The spiffs code is a bit of a mess as it tries to preserve the contents of the filesystem across reflashes of the firmware. Given a partition, the options really appear to be:

• Is filesystem aligned at the start (or at least as close as possible)? Or do we align on a bigger boundary to improve the chances of find an intact file system?
• Does the filesystem occupy the entire partition or is it artificially constrained (so that it formats faster)?
• Is there a filesystem image being built at build time that gets flashed into the spiffs partition?
• Does the firmware try and use a valid filesystem if it finds one, even if it would have put it somewhere else (or given it a different size)?

I suspect that the answers to these questions depend on the precise use case.....

[jmattsson]

@TerryE If we're getting proper partition support, then most of those options are no longer relevant. I think the only one is the check for the size of the file system, as that determines which mode it's better to format it in. I don't think I know anything about the implicit/explicit formatting options.

People are already used to losing their files if they repartition their storage, so I would be fine with having us assume the SPIFFS sits from the start of partition (aligned as necessary) to the end (again, aligned as necessary). When deciding where the partition goes, if that's done automatically that would probably be where I'd recommend keeping @pjsg's give-it-a-bit-of-room-after-the-firmware-so-the-fs-doesn't-get-lost-all-the-time logic.

[TerryE]

Because the irom0 copy of the partition is a standard table at a fixed location (the start of the 0x10000.bin file / flash offset 0x10000), it is trivial to extend / wrap esptool so that we can add directives to change this either in the file or directly on the firmware. So for example we could:

• Have the option to specify LFS size / and SPIFFS settings in the same way that we can do --flash_size 32m
• Write the LFS or SPIFFS images using a parametric option such as write_lfs path/lfs.img or write_spiffs path/spiffs.img
• Have a --keep_partitions option for reflashing over an existing install -- that is the PT is kept as-is so long as the 0x10000.bin file fits.
• If we do have some guru rules about SPIFFS layout, e.g. by default 1Mb-align the SPIFFS partition if the flash size >= 2Mb then IMO these are best moved to the flashing tool and not the firmware runtime.

Once we have the basic framework in place and the version 1 flashing tool, then we can always tweak things in subsequent PRs

[TerryE]

OK, another update. I've got LFS and SPIFFS properly integrated into the PT. I still have to finish the nodemcutool wapper for espool to integrate PT support. The new exception handling in SDK 3.0 has also broken our GDB interface and I need to fix this. I am also considering extending the PT segment to be a full page size so that we can also use it as NAND write-only for boot-to-boot parameter passing.

Another aspect that I'd like to back out is the 16Kb (rather than the RFC standard 32Kb) dictionary size for LFS compression. The main reason that I dropped from the standard 32Kb to 16Kb was that we had insufficient RAM to be reasonably confident that an invocation of node.flashreload() would succeed. We no longer have this concern with the extra RAM available in SDK 3.0 builds.

[TerryE]

I've got the remote GDB stub working as well now. @nwf Nathaniel, do you want me to push an interim commit to my TerryE repo for you to have a look / play?

[nwf]

@TerryE Sounds good to me. I can't promise I'll have time this weekend, but I'm happy to look over your shoulder.

[TerryE]

Nathaniel, if you haven't got time this w/e, then join the club that I am usually in! This was turned out to be quite a big patch because of all the things that using a PT touches. I'll tidy it up tomorrow then push it.

[TerryE]

@nwf, @NicolSpies, @jpeletier, I've pinged you guys because you all quite active on the project at the moment and might have the time to do a decent review. I just pushed an evaluation version of the V3.0 rebaseline to my fork c3145e3. Feedback / suggestions / whatever from you and any other committer / contributor is very much invited.

This is very much a work in progress, but with a basic LFS loaded and including the lua example _init.lua and dummy_strings.lua, this reports:

NodeMCU 2.2.0.0 build unspecified powered by Lua 5.1.4 on SDK 3.0.0(d49923c)
lua: cannot open init.lua
> =node.flashindex'_init'()
> =node.heap()
59008
> 

It's the 59008 free RAM that's noteworthy.

Note that:

• SPIFFS and LFS now use the partition table
• tools/esplfs uses the irom0 version of the pt and can be used to resize the LFS partition as well as load a new LFS image. This Python code also uses the -f (compressed relocatable) version of the image, so the -a <LFSstart> option is really redundant. I don't claim to be an expert or even competent Python programmer, so improvement / rework / polishing is needed here. This is very much a proof of principle of how we can extend the esptool functionality to add nodemcu specifics. I decided to use the fully encapsulated esptool rather than using its lower level classes.
• I've still got to add the DNS memory optimisation which will free up another couple of 2Kb RAM.
• There's a load of other improvements that we could add or polish, but my instinct is push a version into dev sooner rather than later. Rather than go into these, I will pause for feedback. 😄

After this first pass of discussion, we can decide how to handle the PR.

[TerryE]

So just to summarise the various feedback items in this c3145e3 review:

• The LFS code is now always compiled in, but the functionality is disabled at runtime if the partition size is 0. The rationale here is that the code is stable and relatively small so there is little harm in compiling it in. By using the LFS region size in the PT, this means that LFS can be enabled in any firmware by updating the PT.

• SPIFFS sizing parameters. At the moment, the parameters in user_config.h, SPIFFS_FIXED_LOCATION and SPIFFS_MAX_FILESYSTEM_SIZE are now use to set up the default SPIFFS partition, but these can again be overwritten in the PT on chip. The SPIFFS runtime only uses the SPIFFS partition in the current PT. I think that some refinement is needed here in future PRs.

• Placeholders for extra partitions. The symbolic names for the LFS and SPIFFS partitions are LFS1 and SPIFFS1; LFS2 and SPIFFS2 are reserved for future use.

• ESP IDF Docs » API Guides » Partition Tables defines the current PT interface for the ESP32. I suggest that we mirror this approach as detailed below.

Defining Partition tables

(Deferred on this PR)
The ESP32 IDF has a builder function which creates a Partition Table definition file which can then be used to initiialise the PT for the ESP module. I suggest that we introduce a similar editable file to allow ESP8266 developers to configure their partitions:

• Any lines starting with # are treated as comments and ignored. All other lines are a set of CSV fields:
• Name : Name field can be any meaningful name, but is otherwise ignored. Names longer than 16 characters will be truncated.
• Type and SubType: These have a different interpretation to the ESP32 but the general format is maintained: The type can be one of ....
• Start & Size: If the start is omitted then the partition is assumed to immediately follow the previous partition. If the size is omitted then in the case of the irom1 partition, this is determined from the size of the bin/0x10000.bin file, in all other cases it is assumed to be zero. To simplify sizing the normal sizing qualifiers (mMkK) can be used.
• Flags: No flags are currently supported

I will add a partition_table target to the make to enable this to be invoked without other scripting. May need some tweaking.

[HHHartmann]

@TerryE that's great news. Will the SPIFFS and LFS configuration then be moved or of user_config.h into the PT definition?
To ease configuration for different flash sizes it mögt be good to have the last partition grab the remaining space if the size is set to zero. That way SPIFFS could be auto sized as it is now.
Maybe having the possibility to define "up to size" could solve the problem. The partition would then be of that size if flash size allows and smaller of not.

[nwf]

@TerryE I have merged c3145e3 and the current dev head. Some syntactic noise that's easily dealt with, and a small typo in ldblib.c (3lfls should just be 3).

Things generally seem to be working, though I must be holding something about SPIFFS and the partition table scheme wrong; I can't seem to get my spiffs images to take -- either the board doesn't see the fs or it locks at bootup. Anyway, starting up to 57080 of heap sure feels roomy!

I've put my merge up at https://github.com/nwf/nodemcu-firmware/tree/dev-sdk-3.0-terrye-merge .

[TerryE]

@nwf Nathaniel, In fact I'd already done this but as I said in the PR, I was holding off doing the final merge and push because these changes conflicted with #2659 and #2665, so I wanted to merge these into dev first before rebaselining. I'd also sorted out the SPIFFS stuff in my local fork, so I suggest that we go with rebooting this Issue and PR because the existing ones are long and contain a lot of closed material; we can roll up and summarise the key outstanding points on the new Issue and PR.

[TerryE]

Further discussion carried out on new #2689