[JENKINS-63343] Validate element types in `RobustCollectionConverter` in some cases #9727

dwnusbaum · 2024-09-12T19:03:09Z

See JENKINS-63343. I am just filing a draft PR to see if it breaks tests and as a reference in case anyone looks at the ticket in the future. I am not really sure about this approach, and it seems like maybe there is no good way to solve this issue in general given that XStream APIs use java.lang.Class rather than java.lang.reflect.Type.

Testing done

Proposed changelog entries

human-readable text

Proposed upgrade guidelines

N/A

Submitter checklist

Give feedback

The Jira issue, if it exists, is well-described.
The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
There is automated testing or an explanation as to why this change has no tests.
New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
For dependency updates, there are links to external changelogs and, if possible, full differentials.
For new APIs and extension points, there is a link to at least one consumer.
Options

Desired reviewers

@mention

Before the changes are marked as ready-for-merge:

Maintainer checklist

Give feedback

There are at least two (2) approvals for the pull request and no outstanding requests for change.
Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
Proper changelog labels are set so that the changelog can be generated automatically.
If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
Options

…n some cases

core/src/main/java/hudson/util/RobustCollectionConverter.java

jglick · 2024-09-12T21:13:12Z

core/src/test/java/hudson/util/RobustCollectionConverterTest.java

+                </hudson.util.RobustCollectionConverterTest_-Numbers>
+                """;
+        var actual = (Numbers) new XStream2().fromXML(xmlContent);
+        assertEquals(expected, actual);


And I guess OldDataMonitor will report the mistyped element?

I did not check it, but yes that is the hope.

As of 61db49c the error messages have been improved and the tests log the warnings. Here is an example of the warning for a bad list element:

Sep 18, 2024 5:09:31 PM hudson.diagnosis.OldDataMonitor report WARNING: could not read hudson.util.RobustCollectionConverterTest$Data@2cc3ad05 (and Jenkins did not start up) com.thoughtworks.xstream.converters.ConversionException: Invalid type for collection element ---- Debugging information ---- message : Invalid type for collection element cause-exception : java.lang.ClassCastException cause-message : Cannot cast java.lang.String to java.lang.Integer path : /hudson.util.RobustCollectionConverterTest$Data/numbers/string line number : 5 ------------------------------- at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:120) at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81) at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:101) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:461) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464) at hudson.util.XStream2.unmarshal(XStream2.java:230) at hudson.util.XStream2.unmarshal(XStream2.java:201) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1321) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1312) at hudson.util.RobustCollectionConverterTest.checkElementTypes(RobustCollectionConverterTest.java:207) ... surefire ... at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495) Caused by: java.lang.ClassCastException: Cannot cast java.lang.String to java.lang.Integer at java.base/java.lang.Class.cast(Class.java:3889) at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:114) ... 65 more

Here is an example for a bad map value:

WARNING: could not read hudson.util.RobustMapConverterTest$Data@2a5b3fee (and Jenkins did not start up) com.thoughtworks.xstream.converters.ConversionException: Invalid type for map entry key/value ---- Debugging information ---- message : Invalid type for map entry key/value cause-exception : java.lang.ClassCastException cause-message : Cannot cast java.lang.Integer to java.lang.String path : /hudson.util.RobustMapConverterTest$Data/map/entry[2]/int line number : 9 ------------------------------- at hudson.util.RobustMapConverter.read(RobustMapConverter.java:114) at hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:81) at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:99) at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:93) at com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:88) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:461) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464) at hudson.util.XStream2.unmarshal(XStream2.java:230) at hudson.util.XStream2.unmarshal(XStream2.java:201) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1321) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1312) at hudson.util.RobustMapConverterTest.robustAgainstInvalidValueType(RobustMapConverterTest.java:244) ... surefire ... at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495) Caused by: java.lang.ClassCastException: Cannot cast java.lang.Integer to java.lang.String at java.base/java.lang.Class.cast(Class.java:3889) at hudson.util.RobustMapConverter.read(RobustMapConverter.java:110) ... 67 more

dwnusbaum · 2024-09-16T20:21:02Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

+        // TODO: Is there a better way to make something like this work? Does it break any custom converters?
+        if (converter == null && new RobustCollectionConverter(mapper, reflectionProvider).canConvert(type)) {
+            converter = new RobustCollectionConverter(mapper, reflectionProvider, field.getGenericType());
+        }


In particular I am wondering about things like NodeList and AxisList which extend ArrayList. They might be ok because of the exact equality comparisons here, but I am not sure. I guess also that those classes don't support any of the desired recoverability features in RobustCollectionConverter, but that shouldn't be affected by my PR.

I confirmed that NodeList is unaffected by this change - its custom converter continues to be used and its deserialization is not robust against invalid values.

Looking at #342, I think we only expect getLocalConverter to matter for fields that have the @XStreamConverter annotation, which seems to be more or less unused based on this search, and also none of those annotations are for collections or maps.

dwnusbaum · 2024-09-18T21:20:15Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

@@ -451,6 +451,13 @@ private boolean fieldDefinedInClass(Object result, String attrName) {

    protected Object unmarshalField(final UnmarshallingContext context, final Object result, Class type, Field field) {
        Converter converter = mapper.getLocalConverter(field.getDeclaringClass(), field.getName());
+        if (converter == null) {
+            if (new RobustCollectionConverter(mapper, reflectionProvider).canConvert(type)) {


It's a bit awkward to recreate this for every single field being unmarshalled, should we keep an instance as a local field just for these canConvert calls?

dwnusbaum · 2024-09-18T21:23:43Z

core/src/main/java/hudson/util/RobustReflectionConverter.java

@@ -451,6 +451,13 @@ private boolean fieldDefinedInClass(Object result, String attrName) {

    protected Object unmarshalField(final UnmarshallingContext context, final Object result, Class type, Field field) {
        Converter converter = mapper.getLocalConverter(field.getDeclaringClass(), field.getName());
+        if (converter == null) {


See #9727 (comment) for discussion about cases where converter is not null. In short, I don't think we care.

I am not sure though whether this method covers all of the cases we care about. It seems like we may need similar changes related to "implicit collections". I will look into that a bit.

Specifically, prior to this line, I think we should look up the field and then add code comparable to what we added to RobustCollectionConverter to validate the value type:

jenkins/core/src/main/java/hudson/util/RobustReflectionConverter.java

Line 477 in 1836bd6

collection.add(value);

Needs to be investigated with a test case.

[JENKINS-63343] Validate element types in RobustCollectionConverter i…

2f4e04c

…n some cases

jglick reviewed Sep 12, 2024

View reviewed changes

core/src/main/java/hudson/util/RobustCollectionConverter.java Show resolved Hide resolved

jglick reviewed Sep 12, 2024

View reviewed changes

dwnusbaum commented Sep 16, 2024

View reviewed changes

dwnusbaum added 3 commits September 18, 2024 16:39

[JENKINS-63343] Also validate keys and values in map entries

966f598

Merge branch 'master' into JENKINS-63343

0944533

[JENKINS-63343] Improve OldDataMonitor error messages and update tests

61db49c

dwnusbaum commented Sep 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-63343] Validate element types in `RobustCollectionConverter` in some cases #9727

[JENKINS-63343] Validate element types in `RobustCollectionConverter` in some cases #9727

dwnusbaum commented Sep 12, 2024

Submitter checklist

Maintainer checklist

jglick Sep 12, 2024

dwnusbaum Sep 12, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 16, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

dwnusbaum Sep 18, 2024

[JENKINS-63343] Validate element types in RobustCollectionConverter in some cases #9727

Are you sure you want to change the base?

[JENKINS-63343] Validate element types in RobustCollectionConverter in some cases #9727

Conversation

dwnusbaum commented Sep 12, 2024

Testing done

Proposed changelog entries

Proposed upgrade guidelines

Submitter checklist

Desired reviewers

Maintainer checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[JENKINS-63343] Validate element types in `RobustCollectionConverter` in some cases #9727

[JENKINS-63343] Validate element types in `RobustCollectionConverter` in some cases #9727