[BUG] Camera tuning can calculation can overflow causing alternating white and black lines when images are very red.

[julianstirling]

Describe the bug
Camera tuning can calculation can overflow causing alternating white and black lines when images are very red. This happens when using it as a microscope. We use a custom tuning file to do lens shading correction. A similar algorithm was developed for the old picamera stack and does not give this issue with the same hardware.

To Reproduce

I apologise that this is significantly less self contained than you would like! But it gives you an idea of what we are doing.

1. Take a PiCamera V2, this 3D printed lens spacer, and this 3d printed platform.
2. Assemble the camera for microscopy as so:
   
3. Aim camera at white light (this may be easier if you build a whole OpenFlexure Microscope!)
4. Load this customised OS onto a pi.
5. Open the OpenFlexure software and run the autocalibration to generate a tuning file with alsc parameters which should correct for the pinkness around the edge caused by the change in chief ray angle.
6. Take image without a microscope slide and see it is pretty uniform grey. Image a blue stained microscope slide and see the colour is pretty consistent 👍
7. Aim camera at a glass slide covered in red ink from a red marker pen.
8. Get this weird cross with very strange artifacts in the grey area:

Expected behaviour
A uniform-ish red image. The alternating black and white lines in the grey area seem to imply there is some sort of overflow in the alsc calculation, in certain areas of the image if the pixels are too red.

Console Output, Screenshots

• Example tuning filetuning.json
  Without the same lens set up this isn't going to be overly useful for reproducing the images.
• Code that generates tuning file

Hardware :
Pi4, Pi Camera v2, Custom lens setup to make it a microscope as explained above.

Additional context
Tagging others who work on the OpenFlexure project.
@rwb27 @JohemianKnapsody

[davidplowman]

Hi, thanks for the report. Can you provide a link to a DNG file that you've captured that gives rise to a bad LSC tuning? That would give me something I could investigate. Thanks!

[julianstirling]

Thanks for the quick response @davidplowman. Would an npz that holds the a numpy array of the raw Bayer data work? Our software can spit that out. We will generate one tomorrow.

[davidplowman]

Would prefer a DNG really. cam.capture_file("test.dng", 'raw') would do it. You could also run that through our alsc_only.py tool and see what comes out.

[julianstirling]

I currently don't have hardware to hand so this was generated by someone else with:

with self.picamera() as cam:
    r = cam.capture_request()
    r.save_dng(path)

Archive.zip

there is also a jpg. The DNG doesn't look raw? Do we need to redo with cam.capture_file("test.dng", 'raw')?

I have looked at alsc_only.py in the past. We didn't get good results. Our algorithm worked very well with the old picamera stack and works well for images that aren't red (see many stitched together here for a cervical smear scan with very uniform colour https://images.openflexure.org/cap_demo/viewer.html).

Is there also a reason why the table is reduced to only 16x12 compared to the 52x39 that was possible in the old camera stack?

[davidplowman]

I've taken a look, though I'm not quite sure what to make of it. A few comments:

• The LSC tables look plausible enough, though they're not super effective for the included DNG file.
• The white balance seems a bit off, everything is quite orange. Maybe try setting "bayes" to 0?
• The colour matrix seems quite extreme. Maybe try disabling it to be sure nothing is overflowing? (Replace "rpi.ccm" by "x.rpi.ccm" to disable)

In case it's useful, here's what I wrote to process the DNG file into something viewable based on the tuning you have.

import cv2
import numpy as np
import json
import rawpy

dng = rawpy.imread("output.dng")
raw = np.maximum(dng.raw_image - 64, 0).astype(np.uint16)  # black level is 64 (out of 1023)
b = raw[0::2, 0::2]
gb = raw[0::2, 1::2]
gr = raw[1::2, 0::2]
g = ((gb.astype(np.uint32) + gr.astype(np.uint32)) / 2).astype(np.uint16)
r = raw[1::2, 0::2]
full_size = (r.shape[1], r.shape[0])

with open("red_cross_tuning.json", 'r') as f:
    tuning = json.load(f)

alsc = [algo for algo in tuning['algorithms'] if 'rpi.alsc' in algo][0]['rpi.alsc']
b_table = np.array(alsc['calibrations_Cb'][0]['table']).reshape(12, 16)
r_table = np.array(alsc['calibrations_Cr'][0]['table']).reshape(12, 16)
lum_table = np.array(alsc['luminance_lut']).reshape(12, 16)

b_gains = cv2.resize(b_table, dsize=full_size, interpolation=cv2.INTER_LINEAR)
r_gains = cv2.resize(r_table, dsize=full_size, interpolation=cv2.INTER_LINEAR)
lum_gains = cv2.resize(lum_table, dsize=full_size, interpolation=cv2.INTER_LINEAR)

b_gains *= lum_gains
r_gains *= lum_gains
b_gains /= np.min(b_gains)
r_gains /= np.min(r_gains)

b_corrected = b * b_gains
g_corrected = g * lum_gains
r_corrected = r * r_gains

wb_r, wb_g, wb_b, _ = dng.camera_whitebalance
wb_gain_r = 1 / wb_r
wb_gain_b = 1 / wb_b

rgb = np.empty((full_size[1], full_size[0], 3))
rgb[:, :, 0] = b_corrected * wb_gain_b
rgb[:, :, 1] = g_corrected
rgb[:, :, 2] = r_corrected * wb_gain_r
ccm = dng.color_matrix[:, :3][::-1, ::-1]  # rotate for BGR pixel order
rgb = np.maximum(rgb @ ccm, 0)
rgb *= 64  # make 10 bits into 16
rgb = np.power(rgb / 65536, 0.45) * 65536  # approximate gamma, so we can see something!
rgb = np.clip(rgb, 0, 65535).astype(np.uint16)

cv2.imshow("rgb", rgb)
cv2.waitKey(0)

Also, how are you capturing these images? They seem quite low resolution. Are you able to capture with rpicam-still as a comparison (using your tuning file)?

[julianstirling]

Right. I have managed to reproduce this with rpicam-still and a custom tuning file.

So I made my custom tuning file with our software. And added an x in front of the "rpi.ccm", and also set bayes to zero.

I took an image of nothing with the white balance on. I got a nice image. I recorded the gains and using --awbgains option I was able to reproduce a nice white (grey) image.

I moved the microscope slide so I am looking directly at red pen

I then ran:

 rpicam-still --tuning-file tuning.json --awbgains .707,1.53 -o red-x.jpg -r 1

I still get the clear red-cross with the black and white artifacts:

The DNG has no cross and looks quite yellow. It has a bright centre and is dull around the edges. We expect this sort of ring shape due to the lenslet array when using at as a microscope. Hence the custom tuning.

When running your code above we get a uniform-ish image (screenshot):

Though it is more orange than red.

Tuning, DNG, and full-res jpg:
redx.zip