Skip to content
This repository has been archived by the owner on Nov 24, 2022. It is now read-only.

[WIP] Revamp linker loading (do not load all object files during linking) #666

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gkaracha
Copy link
Member

Closes #665. This PR is a WIP.

Roadmap for addressing the first bullet of #665:

  • Define new type for dependencyMaps, and serialization/deserialization logic for it.
  • Add an index in AsteriusCachedModule after the dependencyMap field, as a map from EntitySymbol to both the file that the entity originates from and the offset in the file. Though initially the offset is enough, when we concatenate modules we end up having entities from different files so we need both the file and the offset in the file to pinpoint the location of the entity.
  • Implement retrieval of entities without reading any other part of the file.

Roadmap for addressing the second bullet of #665:

  • TBD

@TerrorJack
Copy link
Member

In #665, the "dependency map" is just an abstract concept, it doesn't have to be a concrete data type. Adding a separate data type here (which is just another wrapper of SymbolMap) is a bit overwork IMO.

@gkaracha
Copy link
Member Author

You might be right, but I cannot be sure about how much we'll end up passing it around. If it is a lot, I'd say it deserves it's own type. Let's see how it goes and if it ends up being just noise I'll remove it before we merge this 🙂

@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from 4bb04b8 to 667682b Compare May 29, 2020 10:40
@gkaracha
Copy link
Member Author

@TerrorJack lest we forget:

Since we need (at least parts of) the FFIMarshalState to build the set of root symbols (see https://github.com/tweag/asterius/blob/master/asterius/src/Asterius/Passes/GCSections.hs#L43), having some parts of the FFIMarshalState in AsteriusCachedModule looks unavoidable.

@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch 3 times, most recently from 3c59c12 to 8cbd98c Compare June 5, 2020 15:20
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from 8cbd98c to 6b7b950 Compare June 12, 2020 09:13
@gkaracha
Copy link
Member Author

After discussing this with @TerrorJack, the current plan is to follow a different approach than the one originally discussed in #665. We avoid the hassle of adding an index inside the object files and instead:

  • Make object files contain (a) the dependencyMap and (b) the actual AsteriusModule. Before gcSections we only deserialize the dependency map, and after gcSections we deserialize the asterius module. This means that both should be length-prefixed so that we can skip them.
  • After gcSections has determined which symbols we need to keep, we do indeed read all object and archive files that we need to read, but on-the-fly keep the parts that we need. Probably this means some extra work for the garbage collector, but it still avoids keeping all the data in memory.

@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch 2 times, most recently from 1c7ed2f to 1a0d5aa Compare June 17, 2020 16:37
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch 3 times, most recently from d0cd705 to 917df65 Compare June 23, 2020 14:21
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from ad00fd9 to 8163215 Compare July 2, 2020 15:42
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from 8163215 to 39d1959 Compare July 6, 2020 13:56
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch 8 times, most recently from 2124c21 to 2e1598f Compare July 10, 2020 15:51
@gkaracha gkaracha changed the title WIP Revamp linker loading (do not load all object files during linking) [WIP] Revamp linker loading (do not load all object files during linking) Jul 13, 2020
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from 2e1598f to b6f6cf4 Compare July 14, 2020 08:27
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from b6f6cf4 to 1cfa63f Compare July 22, 2020 11:58
@gkaracha gkaracha self-assigned this Jul 22, 2020
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from 1cfa63f to aa36bd5 Compare August 11, 2020 12:34
@gkaracha gkaracha force-pushed the wip-revamp-linker-loading branch from aa36bd5 to 0b37556 Compare August 11, 2020 13:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avoid loading all object files at link-time
2 participants