OutOfMemoryException when using WriteRecords with .NET Core 3.1

[adrianwright109]

Hi,

We have been using CsvHelper version 12.1.2 in our .NET Core 2.1 package using the following code to generate a CSV file so that the result can be saved into Azure Blob Storage:

        /// <summary>
        /// Build a CSV file so it can be saved to Azure Blob Storage
        /// </summary>
        /// <returns></returns>
        private byte[] BuildCsvFileToSaveToAzureBlobStorage()
        {
            // get a list of data to transform into CSV
            List<RVExportVehicleAndOption> data = CreateTestData.GetData(numberOfRecordsToReturn: 500000);

            //get byte array so we can save to Azure Blob Storage Account
            var fileBytes = ConvertDataToCsv(data);

            return fileBytes;
        }

        /// <summary>
        /// Convert data into Csv memory stream
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="dataItems"></param>
        /// <returns>Memory stream of Csv data</returns>
        private byte[] ConvertDataToCsv<T>(IEnumerable<T> dataItems)
        {
            using (var memoryStream = new MemoryStream())
            {
                using (var streamWriter = new StreamWriter(memoryStream, Encoding.UTF8))
                {
                    using (var csvWriter = new CsvWriter(streamWriter))
                    {
                        // don't use double quotes around any values by default
                        csvWriter.Configuration.ShouldQuote = (field, context) => false;

                        // add double quotes around string values only
                        csvWriter.Configuration.TypeConverterCache.AddConverter<string>(new DoubleQuotesStringConverter());

                        csvWriter.WriteRecords(dataItems);
                    }
                } // streamWriter gets flushed here.

                return memoryStream.ToArray();
            }
        }

N.B. Data being written to CSV file is large > 500,000 records.

We didn't see any issues with this code running under .NET Core 2.1, but since upgrading the project to .NET Core 3.1 we have started to see OutOfMemoryExceptions when calling csvWriter.WriteRecords(dataItems);

Would it help if we were to:
Upgrade the CsvHelper package to the latest version (15.0.0) ?
Batch the records so that it doesn't write 500,000+ records at a time ?

[JoshClose]

Maybe there was a change in default memory size between 2.1 and 3.1 or something.

You're passing in items, then writing them all to memory. You're essentially duplicating all the data in memory. Is there the possibility that more than one of these could be running at a time? Like if this is a call on a website or something. You could potentially have issues there.

Anything you can do to keep the memory down would be a good idea. Batching them 100 or 1,000 at a time would definitely help. If it's a lot faster to large amounts of data at once, you'll have to balance that speed with how much memory you want to consume.