Skip to content

System DT Meeting Notes 2019

Jump to bottom Edit New page
Nathalie Chan King Choy edited this page Dec 13, 2019 · 9 revisions

Table of Contents

2019-Dec-11

Agenda

  • Discuss Rob's feedback, including the naming scheme for cpus, cluster nodes, for indirect-bus nodes
  • How we can make address-map shorter and more convenient to write.
  • Feedback from others on the latest examples
  • Discuss next steps for the proposal.

Attended

  • Jeremy G (Qualcomm),
  • Krzysztof Kepa (GE Research),
  • Rob Herring (Arm),
  • Stefano Stabellini (Xilinx),
  • Benjamin Gaignard (ST),
  • Dan Milea (Windriver),
  • Mathieu Poirier(Linaro),
  • Tomas Evensen (Xilinx),
  • Tony McDowell (Xilinx),
  • Dan Driscoll (Mentor),
  • Steve McIntyre (Arm/Linaro),
  • Etsam Anjum (MGC),
  • Pierre Guironnet de Massas (Kalray),
  • Bill Mills (TI),
  • Arvind (Mentor),
  • Bruce Ashfield (Xilinx),
  • Ed Mooring (Xilinx),
  • Nathalie Chan King Choy (Xilinx)

Action items

  • Rob: Reply to Stefano's latest response on list
  • All: Give a look at Stefano's example. Reply on the list about what doesn't work for your use case
  • Stefano: Bring access field back into the proposal for discussion sooner than later
  • Stefano: Include chosen node proposal in example to list so we have whole view of what we are trying to achieve. Target early 2020. Looking for intent & what it's doing. Can get into syntax later.
  • Stefano: Note open question: What about top level memory node & reserved memory b/c that's connected?
  • Stefano: Start a thread on CPUs, chosen, memory, reserved memory nodes on list & we can discuss each of them & cases we might have missed
  • Bruce: Send out info on how to prune System DT into DTs. Target early 2020.
  • Nathalie: Confirm if Rob can make Jan 8th at 9am PST timeslot

Notes

  • Stefano: Recap of where we are
    • We are here to discuss set of changes to device tree spec to describe full heterogeneous platform. 2 subsets:
      • Describe HW as is
      • To configure what SW running on each CPU cluster (out of scope b/c haven't written clearly defined bindings yet
    • Describing HW as is
      • Have sent out to list how want to describe
      • Example with Cortex-A, Cortex-R, Cortex-M
      • Differences in address mapping
      • Visibility of interrupt controller to correct CPU cluster
      • 3 new bindings
        • CPU cluster binding to advertise presence of another CPU cluster. Need b/c otherwise just default CPU node (1 cluster)
        • address-map
        • Indirect-bus: similar to simple-bus. Key difference is it doesn't automatically map into parent address space. Need to explicitly map w/ address-map.
  • Discuss Rob's feedback:
    • Rob:
      • Keep details of specific binding on the list for discussion
      • Rob owes Stefano a reply
      • Think we should discuss high level issues in the call
      • Previously we had said to solve things 1-by-1, but got burned on a lot of bindings in the past. Seemed complete & then had to add more soon after.
    • Stefano: Agree. All, please give it a look & give it a try. Does it work for your use case? Please reply on the list if you think it would work.
    • Rob: Want to get to point where we think this is how we should do bus for multiple clusters & then move on. Need to give option to go back to revisit topic. Because not yet ready to commit this to the spec. Maybe could have separate branch.
    • Stefano: Agree
    • Tomas: Agree.
      • Mentor has done something similar before. Dan, would be good to get your feedback
    • Benjamin:
      • We have not done like this for ST, more static bindings.
      • Noticed you have removed access field from your proposal. This is where I have concern on how to program the firmware. Think this needs to be covered.
      • Or, do you just want to focus on bus mapping & memory mapping & leave rest for another mapping?
    • Stefano: Why I removed:
      • I still have intention to send them out. I don't plan to drop from proposal. Still important & want to do it.
      • I didn't feel too comfortable about that part & very unstable
      • This first initial set (not to program HW), should be able to stand on its own without the domain descriptions & access list.
      • I still have a full detailed example internally w/ access list. If you want, I can send & discuss it.
    • Benjamin:
      • In ST, we have completely different proposal for access part
    • Stefano:
      • We can start discussing those sooner than later & trying to make them work for both our use cases
      • Think access part will change more than other bindings. Open to change.
      • What should be done by the access list & what here? Separation clear: This set of changes is to describe what HW is there & what they can see on bus.
    • Benjamin:
      • Agree
    • Rob:
      • While you can solve each of these separately, they do have different views. Think the CPU clusters need ID b/c could be useful in other places, whether access controls.
      • Interrupts has major open issue: How do decide which entry you're selecting? You need to associate each interrupt controller w/ CPU cluster. You can do mapping to map each device to multiple parents, but how do you select which parent you're using? Maybe can extend interrupt parent to indicate which CPU you're on…
    • Stefano:
      • This problem didn't occur to me before.
      • Not to say this isn't a problem. Had expected selection of interrupt would be once you configure "OpenAMP" domains. Domains decide what goes where. Once you decide device goes to cluster, then selection comes naturally from there
    • Rob:
      • Are domains or CPU clusters 1 set of numbers or 2?
      • We need to figure that out. Maybe it can be one
    • Stefano:
      • OK
    • Benjamin:
      • For me, 1
    • Tomas:
      • Xilinx: Dual core R5 can be lockstep (1 domain) or they can be separate (each running a different OS) -> Don't want to change the HW description
      • For Hypervisor virtual domain, seems like you can configure for each cluster if it's 1 or multiple domains on some HW
    • Stefano:
      • I can't think of case where domain would span multiple clusters
      • Can think of case where 1 cluster can have multiple domains. R5 lockstep/non-lockstep is an example. Still part of same cluster in HW description, but treated as different SW domains from firewall view & more.
        • Suppose we have chosen node w/ boot parameter. If there are multiple clusters and multiple domains, which domain is affected by the parameters? And how can others have their boot parameters too?
        • VM is a second example. Need to handle both physical & virtual domains.
    • Tomas:
      • Also, TrustZone, OP-TEE -> Different operating environments on same CPU
    • Benjamin:
      • Similar to what we have in the pin control
      • Configuration pair context that you may want for each CPU
    • Tomas:
      • It's hard to discuss without the chosen node proposals
      • You define the domain & say you want to use this part of the CPU cluster. I want to run it in trusted or EL*.
    • Stefano:
      • I took out this info from example to list b/c thought it was confusing
      • Next time I will include it so we have whole view of what we are trying to achieve
    • Dan Driscoll
      • Will system DT always be processed?
      • Can it be used as-is without Lopper?
    • Rob
      • That is a goal
      • I think we should consider what it would look like if it's not a goal, to see if it's worth it
    • Dan D
      • Can see case where it is more complex
      • Show HW description as is, but OS perspective could be much simpler
    • Tomas
      • That's how we started, then worked on it with Grant
      • Grant is big proponent of being able to use it without processing
      • Changed so you have a default & backwards compatibility
      • Nice to have default domain
        • We have programmable HW & add-on boards
        • If it's gong w/ big OS then don't have to do anything
        • Could be solved other ways
    • Stefano:
      • Don't have to think of it as just compatible & not compatible. Could have some state in between.
      • Last example I sent, I think would be completely compatible for Linux. Coherent & OK.
      • With Rob's last email suggestion: Would still be fully compatible w/ spec, but Linux not able to boot. With some limited changes to Linux, could boot.
    • Rob:
      • If minor changes to Linux could make defining System DT more uniform/cleaner, then perhaps we should consider doing those changes.
    • Stefano:
      • Like the idea b/c pragmatic
      • Give us clean spec + we are not really breaking compatibility. These are very small details that can be easily retrofitted
      • Initial proposal was not compatible but now the deviations are not as significant & pieces fit together better
  • Stefano: Questions about indirect bus for Rob. Because it doesn't really map automatically into address map of parent, how would you do indirect bus #?
    • Rob:
      • Good Q. Don't have an answer yet.
      • Think it would be good to go through all the common top level nodes & define how those are going to work with multiple domains.
        • CPUs (discussing already)
        • Chosen
        • Memory nodes
        • Reserved memory node
      • We assume there's only 1 except memory node
    • Stefano: Memory node is a good topic. If there is difference in mapping of memory, then we would need 2 memory nodes (not common but possible) and one of them would need to be under an indirect-bus.
    • Rob: It's possible to have 1 node per bank of memory. It's associated to the top level address space.
      • Stefano: In that case indirect-bus wouldn't work unless you put 2nd memory node under an indirect bus
      • Rob: Could have with multiple nodes in I it. Is memory only used by the "default" processor. Is it divided up across domains? Across CPU clusters? Lots of cases to work out there
    • Stefano: From CPU cluster perspective, there are interesting addressing problems. TCM bus in example is a little memory region. Splitting up a single memory region is like firewall config we were discussing. differences in addressing would only work at controller or whole memory level. So maybe don't need to take care of at indirect bus address map.
      • Rob: May depend on whether there's a different address for different views
      • Stefano: Right. Let's say you have memory controller & memory range could show up at different address for different cluster. e.g. carve out top GB from 1 bank, I don't expect it is possible to have different addressing of the carve-out at the different clusters.
      • Rob: There's configuration of dividing it up vs. HW where view is different
        • Or, maybe you don't use memory node at all, and you run from TCM or some other on-chip RAM
      • Stefano: Need to figure out how to make memory node not accessible. Putting it under indirect bus seems unsatisfactory, but it's only way we have today to do it. This is an open question: What about top level memory node & reserved memory b/c that's connected?
    • Stefano: Chosen is interesting too
      • e.g. bootargs on Linux command line or stdout argument (removed from example, Which is UART you should be using?)
      • Probably need 1 domain rather than 1 global
      • Or 1 global & each domain could override it with their copy of it
    • Stefano: Rob can you go over top level nodes we should think about?
      • Rob: CPUs, chosen, memory, reserved memory
      • Stefano to start a thread on that on list & we can discuss each of them & cases we might have missed
      • Rob: Sounds good
  • Benjamin: How many properties do you want to change? Do you have list of impact you have in mind?
    • Rob: Not really. Today, we look for cpus & /cpus. Maybe that changes to look in if they are /cpus then look in /cpus/cluster with 0 cluster being the default or something. Think minor changes where you add place to go look.
    • Benjamin: That should be limited to cluster node?
    • Rob: That's just 1 example that's the most fleshed out so far. We haven't thought about list of other top level nodes.
  • (Rob had to leave here)
  • Tomas: Next steps
    • Get the proposal on the chosen node so we can discuss in context of other issues
      • Target early next year
      • Stefano: We are not looking for detailed/final writing of binding. Can give reference w/ more context & explanation for review & testing.
      • Tomas: Looking for intent & what it's doing. Then can go into syntax later.
    • When can Bruce send out info on how to prune System DT into DTs
      • Bruce: Can send out in early new year
  • Dan Driscoll: CPU topology binding - is it used? Is that going to be part of standard usage & how what we are discussing here fits?
    • Stefano:
      • Rob would know more on this topic but will try to answer
      • You're referring to set of bindings to describe NUMA system. Some may access at different speeds. Linux runs on all of them.
      • CPU clusters we are defining are completely heterogeneous w/ different instruction sets & different everything.
      • Could use CPU topology inside CPU cluster if any has NUMA properties
    • Dan D: Terminology a little confusing
    • Stefano: Yes. Did have to do a bit of dance w/ words to avoid misunderstanding. Cluster & nodes used in a lot of contexts & difficult to explain.
  • Dan D: Virtual __ support
    • What I've seen so far don't have a lot of context related to virtualization. Will that be ignored or push to later?
    • Will have to think of a lot more attributes to enable hypervisor. Is it beyond scope for now?
    • Our system definition tree is used for both OpenAMP & hypervisor
    • Stefano:
      • We have set of bindings that are Xen specific & would like to avoid 2 more sets of bindings
      • There are questions related to hypervisor that are complex
      • I am maintaining an example with open questions for hypervisor
      • I do expect the 3 properties defined under OpenAMP (cpu, access, memory) to also be able to use for virtualization. Expect this minimal core to stay the same
      • Virtualization: have to add more & don't have answers yet, but will build on top of initial set of OpenAMP properties
      • Expect virtualization case will need more info
  • Dan D: Chosen node bindings: Is intent to have bindings w/ direct access? How hard/solid will these bindings be, or vendor specific?
    • Tomas: Intent is to minimze vendor extensions. Preferred place is if it happens in devicetree.org, but doesn't have to be. We are trying to unify what's happening with Xen & that didn't go into devicetree.org
    • Dan D: Chosen node being used b/c it's an open thing w/ no HW specific usage of it instead of new bindings for these new domains. Chosen is bit of wild west. Even in Linux, there's lots of different possibilities of what is supported/not depending on version of kernel. How cleanly would bindings in chosen node be followed?
    • Tomas: Agree, we need to define it well, just TBD where [i.e.]
    • Dan D: Understand drive to make it usable for Linux, but from proprietary solution standpoint it creates complexity
    • Tomas: Reason for picking chosen node:
      • DTs are really meant for defining the HW not configuration. The chosen node was the place to put configuration in.
      • Not because it's wild west.
      • We have to get back to that as we're standardizing.
    • Tomas: Linaro has Device Tree Evolution (DTE) project. Discussing things like where do we put DT & how Linux-centric should it be?
    • Dan: Know @ Linaro conference there was discussion on separate repo for DT. Is that moving forward?
    • Steve: Still open Q. Has had pushback from some maintainers. Some on board with it, some not. Someone working on prototype for the workflow.
  • Nathalie: When to have next call? When are ppl back from Holidays? 8th or 22nd Jan or Feb? (Assuming we use same slot in alternating week when DTE call is not on)
    • Tomas: Definitely want something in Jan
    • Bill M: Linaro core members meetings conflict w/ 22nd
    • Steve: Looks like Rob's calendar is open for 8th, but you should confirm with him

2019-10-31

Attended

Grant L. (Arm), Tomas E. (X), Benjamin G. (ST), Dan M. (WR), Don H. (L), Etsam A. (MGC), Loic P. (ST), Mark G. (TI), Mathieu P. (L), Rob H. (Arm), Steve McI. (L), Stefano S. (X), Poonam A. (NXP), Vincent G. (L), Joakim B. (L), Ed M. (X), Manju HM (X), Nathalie CKC (X)

Action items

  • Stefano: Remove ranges from amba_rpu
  • Stefano: In next couple weeks, look at GPIO controller extensions & write in detail & send to list so we can discuss.
  • All: if you can think of simple solution to interrupt-map info duplication, let Stefano know & he will try to add to the example.
  • Nathalie: Confirm with Kumar that Wed slot will work & send out meeting invitation for 4 weeks from now.
  • Nathalie: Add Connect meeting attendees & send out info on how to subscribe to OpenAMP system-dt mailing list.
  • Nathalie: At bottom of all system-dt-related emails, put how to join the list/get involved/learn more

Notes

  • Mailing list
    • Tomas: Use DT list? OpenAMP list? OpenAMP, then DT list?
      • Grant: Since we're not quite sure what it's going to look like, until we have concrete proposal, then doing stuff on OpenAMP list is fine. Once have something concrete, then can move over. This mitigates concern for too much noise.
      • Steve: Agree. Let's thrash out prototype before letting ppl pick it apart.
      • Rob: Think only crickets will respond. Suggested device tree spec list b/c not a lot of traffic there. Pretty much everyone cares b/c everyone has complex systems. Any SoC has Trust Zone, system controller & those start to use DT.
    • Tomas: Let's start on OpenAMP list. Want to include proprietary OS vendors in the discussion too. Once we converge, then can move over to DT list.
    • Nathalie: Who should we have on the list? OpenAMP TSC list, DTE meeting attendees so far.
      • Tomas: Add anyone who joined the Connect call. Email out details for how to join the list.
    • Steve: Put bottom of any mail out to ppl info on how to join list & where to get info to get involved
    • Tomas: Also include pointers to the wiki landing page, where we have the video from the intro presentation.
  • What works for a regular meeting slot and cadence? 
    • Tomas: Proposing once a month for cadence
      • Steve, Grant: Seems about right
    • Nathalie: What about the Wednesday slot in the weeks Steve isn't holding the DTE call, every 4 weeks?
      • Tomas & Steve: OK
      • No objections on the call
      • Nathalie: Circle back with Kumar to confirm Wed slot works for the Zephyr folks
      • Nathalie: Next call in 4 weeks on Wed, if Kumar/Zephyr don't have conflict
  • Stefano: Update on what happened since Connect
    • Stefano: Worked with Grant to get system DT to work in default case
      • Grant: Regardless of the overall intent of system DT, want to see all the changes/bindings that go into making this work be fairly well defined for each individual component & they shouldn't be too interlinked.
        • E.g. Address mapping: DT has always had assumption that address map starts at root node & develops from there. By using the address map, that changes the default & can fan out to different nodes in the tree. It's well-described, well-contained & can be implemented independently from the other system DT stuff.
        • Don't think it's required for you to build a DT that will work with an old kernel, but would like it to be possible
        • Think Stefano & I were able to get to that point
    • Stefano: Based example on Xilinx Versal device tree. Changes minimal. Rob has been asking for a full example. Have a full example now that can be shown.
      • Address map & CPU cluster still there
      • Interrupt done with intermediary interrupt controller node. Still using the standard bindings, so don't need new bindings.
      • New bus type (indirect bus - thanks, Grant!) so can put things that don't map into the root address space. If you have default CPU cluster that is Cortex A. Linux will boot & fall naturally into that address space. System DT will describe things that might not be accessible/visible (e.g. interrupt controller of cortex-R5, which is not visible to Cortex-A) -> Indirect bus will hide it from the Cortex-A. Wouldn't translate automatically in the parent address node, so Linux wouldn't try to access it.
      • Thanks to Grant for example
    • Stefano showed this example early draft system DTS file
      • This time started with full device tree as example. Short example shown at Connect was artificial.
      • A72s as default CPU node
        • Has "cpus" name & doesn't have address map
      • cpus_r5
        • Has address map & compatible CPU cluster
      • How do we build with R5 GIC? A72 GIC should be only visible & accessible from A72.
        • Amba_rpu with indirect bus
          • Grant: This node has a ranges property, which says that it maps to root node. Is that what you intended?
          • Stefano: Wasn't sure - question mark.
          • Grant: It should not have. Ranges should be removed.
      • Address ranges: Only update is this indrect bus
      • Next, interrupts
        • Refer to "interrupt-multiplex" in the example
          • Will be parent of all devices, going to both gic
          • "interrupt-map" property
            • First three numbers on left match the device interrupt. The zeros -> ignoring those fields.
            • e.g. 0x14 going to gic_a72
            • Then numbers on right indicate where going to instance on A72 & could be different
          • Down side is this is cumbersome & repeat every interrupt 3 times (at the device, second time for the A72, third time to say it's going to the R5)
            • Grant: This could be somewhere that needs new binding. Moving into a new node does make sense b/c the device still has only 1 output line. Probably only need 1 cell to describe the input to the distributor node b/c there is no state associated (don't care if level or edge). Think only need the fields if it's a real interrupt controller - what you have here is probably just wire going to multiple GICs.
              • Stefano: Yes, think just wire.
            • Grant: This works & would allow us to make progress, but think it's a strong candidate for a new binding.
              • Stefano: The awkwardness is a good reason to simplify.
            • Grant: It's an enumeration. You have a range of interrupts & you want to take the group & send them somewhere else.
              • Rob: GPIO maps address this already. Has pass through mask or property so you can pass through the GPIO flags. You could do that with interrupt # & flags. That would greatly shorten the map. It's generic now in DT spec: host 0.2 edition of spec. There haven't been lots of changes, so should be easy to find.
              • Stefano: Take a look at DT spec for GPIO maps
      • There will be more things that we'll need, but this is the minimal set to be able to describe interrupt & address map. Now we have technique to do both.
      • Loic: How interrupts are defined in the different peripherals which could be accessed by both CPUs?
        • Stefano: all the interrupt maps under imux 0x14 to A72 GIC & 0x14 to R5 GIC
        • Grant: Not sure what will happen with current software when it finds same interrupt mapped to 2 different places. Not clear spec-wise what should happen.
        • Stefano: Read spec. Don't think this is covered. This is reason to come up with different binding, so we can clarify what should happen. Don't have a strong opinion - happy to follow your advice.
        • Under interrupt-multiplex have interrupt-parent: There are 2 (gic_a72 & gic_r5). Not sure if that's OK.
          • Grant: Don't think it's required
          • Rob: Interrupt controller should not be there either. Looks for either interrupt controller or interrupt map.
        • Stefano: Now only uncertainty remains: What if put 2 times under interrupt map
          • Grant: Does spec require numerical order?
          • Stefano: No, could have put 0x14 A72, 0x14 R5. Not sure if could get rid of 1st & 2nd zero field.
          • Grant: You should be able to just have 1 cell & have it be a number…. Or, maybe need flags field if have mix & match of level & edge interrupts going through
          • Stefano: Would like to come up with best possible binding. We have option to carry flag field. No simple way to show interrupt being duplicated. Still need the 3 cells on the right, which are duplicate of device side. Let's take this offline.
      • Loic: Current proposal lists all interrupts in one unique map (1st you have all A72 then R5) and so SW will have to parse all map to find configuration associated to physical interrupt controller. Would it be possible to have a map table per interrupt parent and to select the right table according to selected parent? A bit like GPIO for which we have several states possible? Can we have interrupt map 0 for A72 & map 1 for R5 and then say 1 will use according to parent.
        • Rob: Then you could extend interrupt-parent to have more than 1 phandle. We probably have to support both the case where the interrupt #s are all the same & case where different.
        • Loic: Can consider virtual lines & physical lines and do the translations to simplify
          • Rob: What do you mean by virtual line? Make up some #s for DT?
          • Loic: Yes
        • Stefano: Currently, the # on the left is a fake #. That 0x14 on the left is matching the device. We care about what's going to the parent - the # on the right. Not saying I like this, but describing how it works. It's essentially a virtual interrupt b/c doesn't exist anywhere.
        • Rob: OK, if there's 2 number spaces then need a 3rd to be common.
        • Stefano: Will look at GPIO controller extensions & write in detail & send to list so we can discuss. Meanwhile, if you can think of simple solution to this, let Stefano know & he will try to add to the example. In next couple weeks.
        • Stefano: There is duplication that's not ideal. It gets us going, and might work, but there is room for improvement.
      • Stefano: There is more in the example, but just covered the highlights relevant to System DT for discussion today
      • Rob: What about memory nodes? What is the R5 using for memory?
        • Stefano:
          • Haven't ported all the OpenAMP domain things. In our system, the R5 sees all the RAM the same as the A72.
          • For the carve out, those are configurable & done w/ reserved memory & openAMP stuff.
          • The only interesting bit for memory is list of TCM regions, which showed as example in previous discussions. It's only a few hundred KB & shows up at 2 addresses on Cortex-R & 1 on the Cortex-A. Don't have it here b/c not in Versal DT yet. Would be under amba & under address-map for cluster, or amba_rpu will make it appear in 2 locations for R5. See TCM comment in example under cpus_r5.

2019-09-26 System Device Tree F2F and call at Connect SAN19

Attended

Tomas, Nathalie, Stefano, Arnd, Maarten, Dan (Mentor), Grant, Ed, Vincent, Matthew, Kumar, Azzedine, Jeremy, Dan (Wind River), Loic, Erwan, Steve, Bruce, Mark, Joakim, Achin, Sudipto, Anders, Manju, Saravana, Arvind, Krzystof, Clement, Mark (Xilinx), Etsam, Waqas, Vivek, Bjorn, Rob, Trilokkumar, Etienne, Victor

Objectives

Dive into the details of what we have done so far, get feedback, who is interested in engaging, ask questions, identify open issues to work on

Summary (grouped by subject, not necessarily chronological)

  • 2 parts
    • What changes needed to spec
    • Configuration
  • Stefano: showed proposed changes to spec using example code
  • Arnd: What about interrupts?
    • Stefano: idea is to use interrupt extended property
      • e.g. R5 cluster will have different GIC from A53 cluster
      • Lopper will be able to chop out interrupt hierarchy that's not relevant to you CPU cluster
  • On use of term "ranges"
    • Rob: We should not call it ranges b/c it's not ranges. Something map, but map has a defined meaning as well.
  • Grant: If you put the full system tree to OS, will it still make sense? Do you need lopper? Want simplest case to work with System DT as given. Want additive & not transformative. Not opposed to changes to Linux. Would like there to be very high bar for proposal that breaks Linux (e.g. go to 2 step tool). Only if it's the only way to do it.
    • Rob: We should minimize the processing that takes place to create a bootable DT
    • Bjorn brought up that UART might require processing
    • Stefano: Need 1 more hook for interrupts. Don't think 0 changes to Linux is possible.
    • Arnd: Will depend on platform.
    • Kumar: Why not be explicit, like interrupt extended property. We get in trouble with defaults. Being explicit removes question of wanting root cluster a certain way. "cpu" cluster has not kept up & is antiquated. To maintain compatibility, we should be more explicit in map property & then you don't have to worry about which cluster. Map applies to CPUs and which it applies to.
    • Ed: The current system is broken for someone trying to use non-Linux OS on a heterogeneous system. 1/3 time of OpenAMP development is trying to get Linux DT to match what you're doing. There's a lot of breakage on the other side of the fence.
    • Grant: Let's take it offline. This is awfully close. If a53 was just named cpus and in common case where there isn't a CPU map, it works. Let's go through some scenarios when it gets concrete. We know it's an issue.
    • Tomas: System DT is not for everyone. You can skip it if you don't need it, and continue to use include files. Ideal if you could make it backward compatible.
    • Grant: Would like to see it as core part of DT tooling. Don't want another tool off to the side.
    • Tomas: Agree
    • Azzedine: Does it mean we have action to define what mapping needs to be?
    • Stefano: Yes
  • Software configurations
    • Chosen node -> SW configuration
      • What kernel runs where
      • Which resources supposed to & not supposed to use. No HW limitation in usage of the resources, it's SW
    • Kumar & Stefano: on where to draw the line for what goes in "chosen"
      • Stefano has Secure/Non-secure and R5 lockstep covered under chosen. Lockstep is strange b/c affects CPU configuration.
      • Kumar: Don't think you should put info about how the HW works in chosen. Chosen is for configuration. HW map in secure should be natural DT.
      • Stefano: Some things are in grey area - this is one. If we see System DT as static that comes from manufacturer in your box with your kit. The board in the box can be one or the other. It's the customer that will decide if lockstep or not.
      • Kumar: Then every config option now has to move here. Don't think that's feasible. Either you have references to everything or duplicate everything. You'll have 1001 properties you'll have to configure this way.
      • Kumar: Whether it's a CPU or a UART, it's just a sub part of a system and has properties & can have defaults that cause behavior to occur.
      • Stefano: Will take this as a note to think more on it.
      • Tomas: Somewhere all this info needs to be specified. Shouldn't be device mfrer, should be customer. We started with a YAML file & lopper can handle. We decided to continue using DT syntax b/c easier for the tool. We need to separate personas doing different things. Don't want to make someone change in many places for 1 thing. Should we have a separate format? That's a separate discussion.
      • Kumar: There is a DT initiative around configuration. As we have schema info for the bindings, how to specify if property needs to be configured. Should user specify? Should it be expert mode? What is more convenient to user?
      • To consider: OS or firmware as consumer?
  • Tomas: Have added some real use cases around OpenAMP & realize that what, for e.g. Xilinx, ST, TI, need are different
    • Should Linux be able to handle system DT without lopper? Is it viable? Do we need to change Linux?
    • How to handle interrupt controller?
    • Ranges map - should it be called something different
    • Secure: Where to put the info - hw vs. configuration. Don't want to duplicate notes. Want SoC vendor to be able to describe the HW & say what you can do. What should input be? DT format, YAML
  • Rob: Suggestion for process
    • Better if we do implementation details on email review in device tree spec list
    • In the call, stick to more high-level things like what do we want to solve
      • Ranges
      • Interrupts
      • How do we assign devices
    • Azzedine: Pick 1 topic at a time to focus on in the follow up meetings.
  • Tomas: There are lots of ppl who are interested -> should recommend they join the device tree spec mailing list
  • Stefano: Virtualization - memory property - address range
    • Azzedine & Stefano to discuss offline

2019-09-11 First System Device Tree call

Proposed agenda (didn't make it through all of it)

  • Logistics: When to meet at Linaro Connect? When is a good time to have regular meetings, how often, other people to involve, ... - Tomas/Nathalie
  • Quick recap of the problems that System-Device Trees are solving - Tomas
  • Proposed additions to specification to describe multiple CPU clusters - Stefano
  • Proposed conventions to allocate resources to domains - Stefano
  • Update on Lopper tool - Bruce

Attended:

Arnd Bergmann, Arvind (Mentor), Benjamin Gaignard (ST), Bill Fletcher - Linaro, Bill Mills (TI), Bruce Ashfield, Clément Léger (Kalray), Dan Driscoll, Danut Milea, Ed Mooring, Etsam Anjum (MGC), felix, Iliad, Jan Kiszka, jeremy gilbert (Qualcomm), Joakim Bech, Loic Pallardy (ST), Maarten Koning, Manju kumar, Nathalie Chan King Choy, Stefano Stabellini, Steve McIntyre, Suman Anna (TI), Tomas Evensen, Tony McDowell (Xilinx), Trilok Soni

Link to the recording:

Link to recording

Notes

  • Tomas: Challenge with System DT is that it's cross-functional, so no 1 single mailing list to discuss
    • Want to start with some live discussions
    • Will formalize proposal & take to mailing lists
  • Anyone opposed to recording: No. Recording starts here.
  • Intros by attendees
  • Tomas: Quick intro on System DT
    • Background:
      • SoCs very heterogeneous w/ multiple CPU clusters
      • System SW needs info about the system (registers in memory map, topologies, DDR allocation, etc.)
      • Different personas involved
        • HW architect
        • System architect
        • Info for each operating system
      • Config info comes in at different times: Sometimes the config info is compiled in, sometimes loaded at boot, etc.
      • Not just multiple CPUs
        • other accelerators may need memory
        • Execution levels
        • Trustzone
    • Today, how this is specified is ad hoc. Want 1 true source & tooling that can get the info to the right place.
      • A lot of RTOS vendors started to do work with Device Trees, so a natural source
    • Verification is also a really important part of it
    • At Xilinx, had an XML file & tooling to give the info to Linux, baremetal, free RTOS
      • Not standardized
      • Need an industry standard. Device Tree is almost there.
    • DT looks at 1 address space, what Linux sees or what Xen sees.
    • Need to expand DT -> System DT
      • It should do 2 things:
        • Describe HW for all the different masters, or CPU clusters
          • Need to add some things into devicetree.org spec & tooling
        • Describe AMP config info: what goes where & domain
          • Suggesting "chosen" section
          • Want to be compatible with hypervisor. (Stefano a maintainer with Xen, so makes sense to align)
    • Lopper creates traditional DT based on System DT
      • So, we don't have change the DT that Linux or U-Boot sees
      • Maybe longer term, Linux may want to know about System DT
    • System DT data flow (RTOS & Linux) diagram showing different personas & sources of info
      • Can add new nodes, add attributes into node, suppress things
      • System architect info can be added as snippet
      • Want to change 1 part of system without editing everyone else's files, but could still be merged together later
      • Showing some possible flows for RTOS/baremetal & Linux
    • Example: System DT + domain info from system architect -> Linux DT using Lopper
    • Flow diagram for Xen & QEMU
      • Intent to get it to work with System DT in future
      • What is shown with QEMU is Xilinx QEMU. Upstream does differently. Trying to get Xilinx method upstream.
    • Domains & resource groups
      • Is what you are sharing intended to be shared?
  • Questions for Tomas about vision?:
    • No questions
  • Tomas: We found a lot of ppl have solved similar problems, which makes it interesting
    • Mentor has some tooling & syntax to specify allocations. Ahead of where we are in terms of the problems encountered. Looking forward to your feedback.
    • Wind River using DT for VxWorks in different way
    • How do you distinguish between in TZ & outside TZ?
    • Really want this to be a universal solution as much as possible
  • Face to Face at Linaro Connect
    • Will have possibility for people to call in
    • Wed afternoon or Thur morning
    • Steve: Suggest Wed 3:30pm PDT for SystemDT but OK to move
    • Who would be calling in from other time zones? Will have Europe
    • Morning is boot session from LEDGE, maybe could swap? Wed morning is kernel & ppl need to be there.
    • 9am-10:30am PDT Thur morning looks most promising. 16:00 UTC
    • Steve will try to make these show up on public schedule
    • Perhaps may have ~30 ppl
  • Stefano: Representative example (not complete) of System DT that would be input to Lopper
    • Output of Lopper could be used by Linux, or RTOS
    • 1st set of changes: How to represent CPU clusters
      • DT mandates only 1 CPU notes called "cpus"
      • We want to represent A53 cluster & R5 cluster, which have different architectures
      • Want to introduce concept of CPU cluster -> small change to spec, but deep impact to DT
      • Keep in mind: Different CPU clusters can have different memory maps
      • Example on Xilinx board where A53 memory map is different from R5 memory map: e.g. tcm_bus region
        • Available at 2 addresses for R5 cluster (low address and high address)
        • Purpose is to show different memory map for 2 different CPU clusters
        • ranges-map is showing AMBA is accessible 1:1 & TCM bus mapping in A53, in R5 have additional entry for TCM
      • These are changes that we need for System DT that can represent multiple heterogeneous CPUs
      • Nothing about configuration shown here
    • Domain section under "chosen": How to explain configuration
      • Could be changed in SW or firmware
      • What is configured to run where
      • What privilege level on CPUs
      • What shared memory we will have between different domains (area where 1 set of SW runs, e.g. R5 domain vs. A53 domain)
      • We have a memory property that shows address & size for RAM region accessible from this domain
      • CPU attribute is interesting b/c tells us where this domain is running (e.g. R5 cluster & CPU mask shows which CPU used to run this domain)
      • Architecture-specific # can represent an architecture-specific config of the CPU
        • e.g. ARM R5 can be lockstep or not lockstep and secure mode or normal mode
      • Other key part is the "access" property
        • Expressing in list of links what is accessible from this domain
        • On Xilinx & other vendor boards, there is possibility to enforce isolation
        • This expresses if this OpenAMP domain can access
        • Example: Showing normal memory, TCM bus, reserved memory region for communicating with A53, IPI is Xilinx HW block for communication between A53 & R5 cluster
        • Flag: provide extra info
          • e.g. mailbox: Which channel, TX or RX
          • Each vendor could express vendor-specific meaning for flag to say how mapping should be done
          • Could then be translated into remoteproc configuration as used by Linux. Conceptually maps to the info we saw before. Could be generated by Lopper from System DT.
    • Stefano has been getting inputs from different groups & starting to reach out wider.
Add a custom footer
Add a custom sidebar
Clone this wiki locally