Support opaque and acquire/release memory semantics

[Spencer-Comin]

Expand the possible memory ordering semantics of symbols from volatile or non-volatile to volatile, acquire/release, opaque, or transparent. The memory ordering semantics are defined as follows:

Transparent

Only guaranteed to be bitwise atomic for data 32 bits or smaller and addresses.
This is the same as non-volatile semantics prior to this change.

Opaque

Accesses to opaque symbols are bitwise atomic.
The execution order of all opaque accesses to any given address in a single thread is the same as the program order of accesses to that address.

Acquire/Release

Loads of acquire/release symbols are acquire loads; i.e., loads and stores after a given acquire load will not be reordered to before that load. This matches the semantics of C's memory_order_acquire.
Stores to acquire/release symbols are release stores; i.e., loads and stores before a given release store wil not be reordered to after that store. This matches the semantics of C's memory_order_release.
Acquire/release accesses have a release-acquire ordering.
Acquire/release symbols also have all the same guarantees that opaque symbols have.

Volatile

Volatile accesses have a sequentially-consistent ordering. This matches the semantics of C's memory_order_seq_cst
Volatile symbols also have all the same guarantees that acquire/release symbols have.
This is the same as volatile semantics prior to this change

Additionally, see the notes on memory ordering semantics in the documentation for Java's VarHandle

[Spencer-Comin]

OpenJ9 note: This requires a coordinated merge with eclipse-openj9/openj9#20475.

[Spencer-Comin]

@hzongaro @zl-wang @0xdaryl @vijaysun-omr FYI

[hzongaro]

This is a review based on an initial pass through the changes. I'll come back for a more detailed review.

One high-level question I wanted to ask is whether these changes ought to be vetted at an OMR architecture meeting.

[Spencer-Comin]

Re @hzongaro's question

One high-level question I wanted to ask is whether these changes ought to be vetted at an OMR architecture meeting.

@0xdaryl @vijaysun-omr what do you think?

[vijaysun-omr]

Jenkins build all

[vijaysun-omr]

I will defer the question on whether we need architecture meeting review to Daryl, but I'll start running tests based on a review I just did.

[zl-wang]

it looks like you are mixing up two types of ordering: 1) the order instructions are laid down in memory (more closely related to instruction-dispatching order. that might be what you meant by program order. relevant to compiler optimization.); 2) the order in which memory accesses are executed/observed (more closely related to instruction-issuing or cache RC-machine actioning. need of memory-barriers to enforce a certain order.).

by a quick glimpse of the code, i have a high-level comment for you to consider further: you seemed changing a lot of places querying 2) above into using querying 1) above. is that optimal or even possibly having correctness-implication?

[Spencer-Comin]

Opaque semantics means that the execution order of an access to the symbol will match the program order as observed by the executing thread.

[zl-wang]

on weakly-consistent machine (e.g. POWER), you don't have guarantee of the execution order at all without memory barriers. i.e. program order is meaning-less unless we are talking about accesses to the same location (e.g. something can lead to paradoxical situations).

[Spencer-Comin]

From the Java VarHandle description Opaque semantics description:

Opaque operations are bitwise atomic and coherently ordered with respect to accesses to the same variable.

If I'm understanding those semantics correctly, for Opaque we only need to ensure that accesses to the same variable/address are executed in the same order (as seen by the executing thread) with respect to each other as they are laid down in the program order. If I'm understanding the weakly-consistent machine memory model correctly, the address dependency between the accesses should be enough to ensure this order without memory barriers.

[0xdaryl]

Thanks for making these changes and thinking through the effects on each architecture.

My main concern is potential confusion around the terms used for the various memory orderings. "Opaque", for example, isn't a well-known term and I think is only used in Java circles. So seeing it appear throughout the code does take some mental adjustment if someone isn't already familiar with the Java definitions. Nevertheless, I am supportive of providing refinement to the kinds of memory ordering the compiler has to deal with, and we have to call them something.

In many places where isOpaque() appears, I think the semantics you're really trying to capture is !isTransparent(), because you're relying on the opaqueness property to be also true for acquire/release and volatile memory orderings. If that's the case, then (to me at least), it might be more readable to use that instead in those places.

[Spencer-Comin]

@0xdaryl perhaps to be more clear we could steal the naming scheme from LLVM's atomic ordering and rename Volatile to SequentiallyConsistent, Opaque to Monotonic, and Transparent to NonAtomic and use isAtLeastOrStrongerThan* helpers

[0xdaryl]

Also, please see the CI failures. There are real build issues in some of them.

[0xdaryl]

It looks like LLVM aligns more with the C++ memory model and can accommodate Java semantics too (as well as other frontends). I don't think OMR has to pivot there just yet. Changing the memory model is something worthy of a longer, architectural discussion for sure. I'm happy for us to have that discussion, but I think what you have here is a fine bridge between what we have now (simple vs volatile) and those more granular orderings.

I do like the isAtLeastOrStrongerThan* helpers idea though as it can make the code clearer as well as the author's intents.

[hzongaro]

Sorry for the really basic questions, but I'm still trying to understand the semantics in the various cases.

Expand the possible memory ordering semantics of symbols from volatile or non-volatile to volatile, acquire/release, opaque, or transparent. The memory ordering semantics are defined as follows:

• Volatile: same as volatile before this change.
• Acquire/Release: as defined for C's memory_order_acq_rel
• Opaque: accessed in program order, but without any assurance of memory ordering effects on other threads. This is similar to volatile in C or C++.
• Transparent: same as non-volatile before this change.

Additionally, see the notes on memory ordering semantics in the documentation for Java's VarHandle

The description of memory_order_acq_rel seems to be specific to operations that perform both a read and write. I assume that a symbol that is marked with TR::Symbol::AcquireReleaseSemantics will be treated like memory_order_acquire in a read operation and like memory_order_release in a write operation - is that correct?

Also, the descriptions of memory ordering semantics for Java's VarHandle states, in part:

In addition to obeying Opaque properties, Acquire mode reads and their subsequent accesses are ordered after matching Release mode writes and their previous accesses.

But if my understanding of the current implementation is correct, TR::Symbol::AcquireReleaseSemantics doesn't seem to imply TR::Symbol::OpaqueSemantics. If so, what parts of the VarHandle documentation should I read and how do they map to the semantics in this implementation?

[Spencer-Comin]

Sorry for the really basic questions

@hzongaro if those were basic questions this change would have been a lot easier to implement!

My initial description (and likely my initial understanding) of the memory semantics may not have been 100% accurate. The important parts from the VarHandle description are the descriptions of getVolatile, setVolatile, getOpaque, setOpaque, getAcquire, and setRelease, and the following paragraph:

Access modes control atomicity and consistency properties. Plain read (get) and write (set) accesses are guaranteed to be bitwise atomic only for references and for primitive values of at most 32 bits, and impose no observable ordering constraints with respect to threads other than the executing thread. Opaque operations are bitwise atomic and coherently ordered with respect to accesses to the same variable. In addition to obeying Opaque properties, Acquire mode reads and their subsequent accesses are ordered after matching Release mode writes and their previous accesses. In addition to obeying Acquire and Release properties, all Volatile operations are totally ordered with respect to each other.

I'll update the description of this PR to have a more thorough description of the different semantics.