Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XStream serialization broken on JDK 17 #3288

Closed
AlexisDrogoul opened this issue Jan 24, 2022 · 13 comments
Closed

XStream serialization broken on JDK 17 #3288

AlexisDrogoul opened this issue Jan 24, 2022 · 13 comments
Assignees
@benoitgaudou @gnoubi
Labels
About Networking Concerns the network plugin and all network-related activities in GAMA 😱 Bug The issue reveals a bug in GAMA 👍 Fix to be tested Priority High

Comments

@AlexisDrogoul
Copy link
Member

Describe the bug
When running the network tests, which make use of XStream for serialising / deserialising messages and agents, the following error appears :

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Comparator java.util.TreeMap.comparator accessible: module java.base does not "opens java.util" to unnamed module

And adding the JVM option --add-opens java.base/java.util=ALL-UNNAMED (described in x-stream/xstream#262 (comment)) changes the error, but the serialisation still breaks.

java.lang.NoClassDefFoundError: Could not initialize class com.thoughtworks.xstream.converters.extended.DynamicProxyConverter
reportAndThrowIfNeeded : Java error: NoClassDefFoundError
	at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:833)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:574)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:496)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:465)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:411)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:378)
	at ummisco.gama.serializer.factory.StreamConverter.loadAndBuildNetwork(StreamConverter.java:58)
	at ummisco.gama.serializer.factory.StreamConverter.convertNetworkObjectToStream(StreamConverter.java:73)
	at ummisco.gama.network.common.CompositeGamaMessage.<init>(CompositeGamaMessage.java:23)
	at ummisco.gama.network.common.Connector.send(Connector.java:136)
	at ummisco.gama.network.skills.NetworkSkill.effectiveSend(NetworkSkill.java:430)
	at msi.gama.extensions.messaging.MessagingSkill.primSendMessage(MessagingSkill.java:113)

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.thoughtworks.xstream.converters.extended.DynamicProxyConverter

Problem is that the whole network plugin, written by @gnoubi relies on it, as well as the saving of simulations written by @benoitgaudou ...

To Reproduce
Run the tests on GAMA 1.8.2 on JDK 17.

Expected behavior
No error.

Desktop (please complete the following information):

  • OS: macOS Big Sur
  • PC Model: MBP 16
  • GAMA version: [e.g. git, 1.7.1, etc.]
  • Java version: [e.g. 1.8]
  • Graphics cards / Display system: [can be useful for displays, OpenGL problems]
@AlexisDrogoul AlexisDrogoul added 😱 Bug The issue reveals a bug in GAMA OS All About Networking Concerns the network plugin and all network-related activities in GAMA labels Jan 24, 2022
@AlexisDrogoul AlexisDrogoul added this to the GAMA 1.8.2 milestone Jan 24, 2022
@gnoubi
Copy link

gnoubi commented Jan 24, 2022 via email •
edited by AlexisDrogoul
Loading

@AlexisDrogoul
Copy link
Member Author

OK. Thanks. I've made myself some very basic and preliminary tests with XStream 1.4.18 (http://x-stream.github.io/download.html) and MXParser, and it solves this issue, but now there is another one:

com.thoughtworks.xstream.security.ForbiddenClassException: ummisco.gama.network.common.CompositeGamaMessage
	at com.thoughtworks.xstream.security.NoTypePermission.allows(NoTypePermission.java:26)
	at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:74)
	at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
	at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
	at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:133)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1391)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1376)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1261)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1252)
	at ummisco.gama.serializer.factory.StreamConverter.convertStreamToObject(StreamConverter.java:49)
	at ummisco.gama.network.common.NetworkMessage.getCompositeContent(NetworkMessage.java:72)
	at ummisco.gama.network.common.NetworkMessage.getContents(NetworkMessage.java:62)
	at ummisco.gama.network.skills.NetworkSkill.fetchMessagesOfAgents(NetworkSkill.java:457)

when running the series of tests on interaction.

Since XStream moved from a blacklist to a whitelist based management of "authorised" classes, does it mean we need to add all possible classes as "authorised" at startup ? And how to do it ?

AlexisDrogoul added a commit that referenced this issue Jan 24, 2022
@AlexisDrogoul
Addresses Issue #3288 by providing :
e30c0ff 
- a new version of XStream library (1.4.18)
- a static initialization of the data streamer (should be thread safe)
- a "whitelisting" of all classes to avoir security errors

Please test more thoroughly...
@AlexisDrogoul
Copy link
Member Author

I've figured it out (http://x-stream.github.io/security.html) I've pushed a possible fix. Please test. Especially the static initialization of the instance of XStream. It is normally thread safe, but in case you need one instance per agent / model / message sending/receiving, it's not too complex to turn it into a method called by the registration of converters.

lesquoyb added a commit that referenced this issue Feb 11, 2022
@lesquoyb
adding support for GamaColor serialization (see #3288 )
898ae1f
@lesquoyb
Copy link
Collaborator

While testing the serialize plugin models I found out that some of them ( Test - Create Simu From File - Container types.gaml for example) were throwing errors of this kind :

---- Debugging information ----
message             : No converter available
type                : msi.gama.util.GamaColor
converter           : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
message[1]          : Unable to make field int java.awt.Color.value accessible: module java.desktop does not "opens java.awt" to unnamed module @2f395397
-------------------------------
	at com.thoughtworks.xstream.core.DefaultConverterLookup.lookupConverterForType(DefaultConverterLookup.java:88)
	at com.thoughtworks.xstream.XStream$1.lookupConverterForType(XStream.java:472)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:48)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
	at ummisco.gama.serializer.gamaType.converters.SavedAgentConverter.marshal(SavedAgentConverter.java:70)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1243)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1232)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:1205)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:1192)
	at ummisco.gama.serializer.factory.StreamConverter.convertObjectToStream(StreamConverter.java:73)

So I implemented a converter for GamaColor ( in this commit ) and all tests seem to pass now, but maybe there are more unsupported types.

@benoitgaudou
Copy link
Contributor

More generally it seems that the serialization does not work anymore.
As an example, consider the model: Memorize Experiment.gaml.

It is supposed to make agents move toward the target. When clicking on the reverse step button, agents are expected to go back toward their initial position.
Here, the steep back button does not seem to have any effect....

@AlexisDrogoul
Copy link
Member Author

@benoitgaudou @lesquoyb @gnoubi can we have a clear view of what is working and not in the serialisation ? The problem with XStream seems to be solved, but you seem to imply that we have other issues ? If so, would it be possible to open specific issues and see if they can be solved ? Otherwise they will be buried here and we take the risk of releasing a version where it does not work...

@hqnghi88
Copy link
Member

hqnghi88 commented Mar 24, 2022 •
edited
Loading

Thanks, i did not follow this issue, so #3334 seems to be related with this.

@lesquoyb
Copy link
Collaborator

It also seems that #2828 is related to xstream

@hqnghi88 hqnghi88 mentioned this issue Apr 5, 2022
Closed
@LucasGrjs
Copy link
Contributor

Problèmes pour run les tests de network sur Gama 1.8.2 :

1 occurence in Tests for ummisco.gama.network0 at cycle 0: Java error: NoClassDefFoundError
in 
NoClassDefFoundError: org/eclipse/paho/client/mqttv3/MqttException
ummisco.gama.network.skills.NetworkSkill.connectToServer(NetworkSkill.java:240)
gaml.additions.network.GamlAdditions.lambda$3(GamlAdditions.java:68)
msi.gaml.compilation.GamaHelper.run(GamaHelper.java:73)
msi.gaml.compilation.IGamaHelper.run(IGamaHelper.java:49)
msi.gaml.statements.PrimitiveStatement.privateExecuteIn(PrimitiveStatement.java:100)
msi.gaml.statements.AbstractStatement.executeOn(AbstractStatement.java:46)
msi.gaml.statements.AbstractStatementSequence.executeOn(AbstractStatementSequence.java:62)
in agent Tests for ummisco.gama.network0

Le fichier MqttException semble manquer donc impossible de vérifier si le problème initiale est résolé

@gnoubi
Copy link

gnoubi commented May 4, 2022 via email

@gnoubi
Copy link

gnoubi commented May 4, 2022 via email

LucasGrjs pushed a commit that referenced this issue May 4, 2022
Fixing issue #3288
2aa7439 
Updating :  org.eclipse.paho.client.mqttv3-1.0.2-3.jar to org.eclipse.paho.client.mqttv3-1.2.5
@LucasGrjs
Copy link
Contributor

Mis à jour de la librarie org.eclipse.paho.client.mqttv3 de sa version 1.0.2-3 vers sa version 1.2.5.
Les 3 tests de network passent tous après cela.

@r3n4ud r3n4ud mentioned this issue Mar 13, 2023
Open
@navid-shokri
Copy link

I have same issue on using JMapper with java v17.

you can find more detail on this link.
https://stackoverflow.com/questions/75959252/why-am-i-getting-error-when-try-to-use-jmapper-in-java-spring-boot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benoitgaudou benoitgaudou

@gnoubi gnoubi

Labels
About Networking Concerns the network plugin and all network-related activities in GAMA 😱 Bug The issue reveals a bug in GAMA 👍 Fix to be tested Priority High
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@AlexisDrogoul @benoitgaudou @gnoubi @hqnghi88 @lesquoyb @LucasGrjs @navid-shokri