CSV mapper does not support Views or filtering correctly for serialization

[triviski]

I am using views to serialize objects as well as get a more clean query output. Below is an example of the two JSONs created, one for serialization and the other for query.

The first is a more sanitized view that is given to the user when queried. The second is a view of what is held in memory and used for serialization. But these are views of the same data using serialization views.

{
  "venue" : "ATLO",
  "sessionNumber" : 0,
  "scid" : 76,
  "dssId" : 0,
  "vcid" : 0,
  "host" : "somehost",
  "isRealTime" : false,
  "channelId" : "A-0001",
  "dnAlarmState" : "Some alarm state",
  "euAlarmState" : "Some alarm state",
  "dnAlarmLevel" : "YELLOW",
  "euAlarmLevel" : "NONE",
  "status" : "",
  "dn" : 1444088455219,
  "eu" : 1.444088455219E12,
  "sclk" : "1444088455.00334",
  "ert" : "1970-017T17:08:08.4550002",
  "scet" : "1970-017T17:08:08.455"
}
{
  "@class" : "jpl.gds.core.globallad.data.EhaGlobalLadData",
  "eventTime" : 1444088455219,
  "sclkCoarse" : 1444088455,
  "sclkFine" : 219,
  "ertMilliseconds" : 1444088455,
  "ertNanoseconds" : 219,
  "scetMilliseconds" : 1444088455,
  "scetNanoseconds" : 219,
  "venue" : "ATLO",
  "sessionNumber" : 0,
  "scid" : 76,
  "dssId" : 0,
  "vcid" : 0,
  "host" : "somehost",
  "insertNumber" : 1,
  "isRealTime" : false,
  "isHeader" : false,
  "isMonitor" : false,
  "isSse" : false,
  "isFsw" : true,
  "channelId" : "A-0001",
  "dnType" : 0,
  "dnAlarmState" : "Some alarm state",
  "euAlarmState" : "Some alarm state",
  "dnAlarmLevel" : "YELLOW",
  "euAlarmLevel" : "NONE",
  "dnRaw" : "AAABUDphIDM=",
  "euRaw" : "QnUDphIDMAA=",
  "status" : ""
}

Issue 1:
If I create a csv schema using an object mapper set up to use the request view (smaller of the pair) the schema still includes everything from the object. I believe that it should only contain the data configured in the view.

When I try to set up the mapper with this view I keep getting this very vague error: "Can not skip a field, expecting a value."

com.fasterxml.jackson.core.JsonGenerationException: Can not skip a field, expecting a value
    at com.fasterxml.jackson.core.JsonGenerator._reportError(JsonGenerator.java:1574)
    at com.fasterxml.jackson.dataformat.csv.CsvGenerator.writeOmittedField(CsvGenerator.java:759)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsOmittedField(BeanPropertyWriter.java:592)
    at com.fasterxml.jackson.databind.ser.impl.FilteredBeanPropertyWriter$SingleView.serializeAsField(FilteredBeanPropertyWriter.java:66)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:666)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:552)
    at com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:129)
    at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1052)
    at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsString(ObjectWriter.java:923)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at junit.framework.TestCase.runTest(TestCase.java:176)
    at junit.framework.TestCase.runBare(TestCase.java:141)
    at junit.framework.TestResult$1.protect(TestResult.java:122)
    at junit.framework.TestResult.runProtected(TestResult.java:142)
    at junit.framework.TestResult.run(TestResult.java:125)
    at junit.framework.TestCase.run(TestCase.java:129)
    at junit.framework.TestSuite.runTest(TestSuite.java:255)
    at junit.framework.TestSuite.run(TestSuite.java:250)
    at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

I traced this in my IDE and see that it is expecting values for fields that have been filtered out (not included) in the view.

I tried to work around this by using filters instead to no avail.

Issue 2:
I set up a filter to mimic the view fro the query data.

        List<String> csvColumns = Arrays.asList("sessionNumber", "host", "venue", "scid", "channelId",  "dssId", "vcid", "ert", "scet", "sclk", "dn", "eu", "status", "dnAlarmState", "dnAlarmLevel", "euAlarmState", "euAlarmLevel", "isRealTime");

I enabled the ignore unknown as well.
csvMapper.enable(JsonGenerator.Feature.IGNORE_UNKNOWN);

This time the issue is that I am not getting any error, but the column names are not properly calling the getter methods. Below is the output I am getting.

sessionNumber,host,venue,scid,channelId,dssId,vcid,ert,scet,sclk,dn,eu,status,dnAlarmState,dnAlarmLevel,euAlarmState,euAlarmLevel,isRealTime
0,somehost,ATLO,76,A-0001,0,0,,,,,,,"Some alarm state",YELLOW,"Some alarm state",NONE,false

As an example, the sclk column should call the "getSclk" method since it has the @JsonProperty("sclk") annotation.

I hope I did something wrong.

[cowtowncoder]

Ok, going forward it would be great if you actually created 2 separate github issues for 2 different problems (even if possibly related).
But for now that is ok.

Now: the best way to ensure my understanding of the issue would be to add a piece of code to show it. This would allow me to reproduce the problem without misunderstanding what is happening.

As to using views when generating schemas: this is an interesting question. I can not say off-hand whether I would expect columns to disappear; esp. since this affects all output formats, not just CSV.
I guess that does make sense. I will file an issue against jackson-databind to tackle this question, and since it is a big behavioral change, can not be implemented before 2.7.

[cowtowncoder]

Ok: Jackson 2.7 will apply Views during schema generation.
Still have the other part of the problem to tackle.

[rob-baily]

Here is some code attempting to use a filter that will reproduce the issue:

package com.etranssystems.tools.csv;

import com.fasterxml.jackson.databind.ObjectWriter;
import com.fasterxml.jackson.databind.introspect.Annotated;
import com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector;
import com.fasterxml.jackson.databind.ser.FilterProvider;
import com.fasterxml.jackson.databind.ser.impl.SimpleBeanPropertyFilter;
import com.fasterxml.jackson.databind.ser.impl.SimpleFilterProvider;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;

import java.io.IOException;
import java.io.OutputStream;
import java.util.HashSet;
import java.util.List;

/**
 * Created by Rob on 4/20/2015.
 */
public class CsvJacksonWriter {

    private final static String CSV_FILTER_NAME = "csvFilter";

    public void writeObjects( OutputStream outputStream,
                              List<?> objects,
                              CsvSchema csvSchema ) throws IOException
    {
        HashSet<String> columnNames = new HashSet<String>();
        for (CsvSchema.Column column : csvSchema) {
            columnNames.add( column.getName() );
        }

        SimpleBeanPropertyFilter csvReponseFilter =
                new SimpleBeanPropertyFilter.FilterExceptFilter(columnNames);
        FilterProvider filterProvider = new SimpleFilterProvider().addFilter( CSV_FILTER_NAME, csvReponseFilter );

        CsvMapper csvMapper = new CsvMapper();
        csvMapper.setFilterProvider( filterProvider );
        csvMapper.setAnnotationIntrospector( new CsvAnnotationIntrospector() );

        ObjectWriter objectWriter = csvMapper.writer(csvSchema);
        objectWriter.writeValue( outputStream, objects);
    }

    private class CsvAnnotationIntrospector extends JacksonAnnotationIntrospector {
        @Override
        public Object findFilterId(Annotated a) {
            return CSV_FILTER_NAME;
        }
    }
}

and then a test class to go with it:

package com.etranssystems.tools.csv;

import com.fasterxml.jackson.dataformat.csv.CsvSchema;
import junit.framework.TestCase;

import java.io.ByteArrayOutputStream;
import java.util.Vector;

/**
 * Created by Rob on 4/20/2015.
 */
public class CsvJacksonWriterTest extends TestCase {

    public void testWriteObjects() throws Exception {
        Vector<Entity> entities = new Vector<Entity>();
        entities.add( new Entity("Test entity 1", "Test description 1", "Test unused field"));
        entities.add(new Entity("Test entity 2", "Test description 2", "Test unused field"));

        CsvSchema csvSchema = CsvSchema.builder()
                .addColumn("name")
                .addColumn("description")
                .setUseHeader( true )
                .build()
                .withLineSeparator("\r\n");

        CsvJacksonWriter csvWriter = new CsvJacksonWriter();
        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        csvWriter.writeObjects(outputStream, entities, csvSchema);

        StringBuffer expectedResults = new StringBuffer();
        expectedResults.append( "name,description\r\n" );
        expectedResults.append( "\"Test entity 1\",\"Test description 1\"\r\n" );
        expectedResults.append( "\"Test entity 2\",\"Test description 2\"\r\n");

        assertEquals( expectedResults.toString(), outputStream.toString() );
    }

    public class Entity {
        private String name;
        private String unusedFieldBetween;
        private String description;
        private String unusedField;

        public Entity( String name, String description, String unusedField ) {
            this.name = name;
            this.description = description;
            this.unusedField = unusedField;
        }

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public String getUnusedFieldBetween() {
            return unusedFieldBetween;
        }

        public void setUnusedFieldBetween(String unusedFieldBetween) {
            this.unusedFieldBetween = unusedFieldBetween;
        }

        public String getDescription() {
            return description;
        }

        public void setDescription(String description) {
            this.description = description;
        }

        public String getUnusedField() {
            return unusedField;
        }

        public void setUnusedField(String unusedField) {
            this.unusedField = unusedField;
        }
    }}

if you remove the field unusedFieldBetween then it works so it has to do with fields in the middle. This seems to have popped up somewhere between 2.4.6 and 2.6.3 as this used to work with all classes at 2.4.6 but does not when things are updated to 2.6.3. If you run the test method here you will get the exception:

com.fasterxml.jackson.core.JsonGenerationException: Can not skip a field, expecting a value

I have a similar problem which I think is related to this. From what I could debug in JsonWriteContext.writeFieldName it sets _gotName to true. It looks like maybe CsvGenerator.writeOmittedField is maybe the culprit here. Let me know if I can help or provide more info. This seems to be a relatively big problem with this version.

[rob-baily]

I actually went ahead and took a shot at fixing it. Found the TestFiltering class there and added a method which fails with the old code but works with the new changes in the writeOmittedField method. @cowtowncoder Please take a look and see what you think. I am guessing that this may have been broken by internal changes in the core Jackson code in the last few versions. I am not exactly sure where the error raised in the method would be applicable so some guidance there would help.

[cowtowncoder]

@rob-baily thanks! This should help figuring out what is going on & fix the issue(s).

[cowtowncoder]

@rob-baily Thank you for the fix: I added it in master and 2.6 branch, latter for inclusion in 2.6.5.