I have a 'sparse colour' camera (KAI-4070 chip) which gives images with clear, red, green, and blue filtered pixels arranged in a 4x4 grid (kind of a generalized Bayer matrix)

G C R C
C G C R
B C G C
C B C G

I need to extract the 4 'colour' channels to individual images and then interpolate back to 2048x2048px. I am working in Python.

I tried numpy reshaping the array to 3D 4x4x256K so that each line through the 3rd dimension is all pixels of the same filter. For each filter I would then set the other filter pixels to NaN and reshape back to 2K x 2K (with many NaN pixels) then interpolate to fill in the gaps. I haven't been able to get reshape to do that - the third dimension always has varying filters. Is this possible with reshape or is there something better.

If this is not a viable approach I would appreciate suggestions. I might brute force it with nested loops but with ~300 images per night times 4M pixels it would be too slow.

I have a 'sparse colour' camera (KAI-4070 chip) which gives images with clear, red, green, and blue filtered pixels arranged in a 4x4 grid (kind of a generalized Bayer matrix)

G C R C
C G C R
B C G C
C B C G

I need to extract the 4 'colour' channels to individual images and then interpolate back to 2048x2048px. I am working in Python.

I tried numpy reshaping the array to 3D 4x4x256K so that each line through the 3rd dimension is all pixels of the same filter. For each filter I would then set the other filter pixels to NaN and reshape back to 2K x 2K (with many NaN pixels) then interpolate to fill in the gaps. I haven't been able to get reshape to do that - the third dimension always has varying filters. Is this possible with reshape or is there something better.

If this is not a viable approach I would appreciate suggestions. I might brute force it with nested loops but with ~300 images per night times 4M pixels it would be too slow.

• python
• numpy
• image-processing

• 1 It would help to have a bit more information here. Could you share your code (even though it didn't work)? And if you already have something that does what you need it to, even if it performs very poorly, it would be good to share it so that people can use it to verify that their answer matches what you actually want to do. – Grismar Commented Mar 6 at 22:32
• you understand it's a Bayer pattern. so you need debayering. many approaches exist, trading between cost and quality. – Christoph Rackwitz Commented Mar 7 at 1:00
• 1 in the case of numpy, don't reshape, but use indexing like data[a::4, b::4] where a and b are "phases", valued 0 to 3. that'll give you a view (modifiable) onto any one of the 16 values in that grid. other indexing can give you other views. -- if you need something more advanced, you can write your own loops, striding and taking from the 4x4 tiles, and then make it execute faster using numba – Christoph Rackwitz Commented Mar 7 at 1:03
• Are you absolutely sure that the 4x4 filter grid you have shown is correct? My recollection is that 4070 chips are usually Bayer8 although that doesn't rule out some having other filter patterns. I've never seen that 4x4 pattern before which is very heavily green weighted so presumably for low light. Resolution in the red channel is dreadful. CMYG is the more common alternative non-Bayer filter pattern. – Martin Brown Commented Mar 7 at 9:52
• emphasis on green has nothing to do with low light as such, but everything to do with human perception. -- I think the presented pattern's color components are distributed badly. that's just an observation. if that's the pattern, that's the pattern. – Christoph Rackwitz Commented Mar 7 at 11:24

2 Answers 2

Looks like you've got a "TRUESENSE Sparse Color Filter Pattern". I saw that in this data sheet, which matches the pattern you present. There is an app note on it too. I also found papers that have a 4x4 pattern that's basically identical to your pattern, "Compton CFA", also "Kumar CFA".

OpenCV does not have that among its demosaic conversion codes.

You could file a feature request on OpenCV's github. I did a quick check and saw no existing issues or PRs on the topic.

That pattern's colors are distributed suboptimally. They clump. It's probably designed to be used in conjunction with 2x2 binning. Such binning would reduce it to a GRBG Bayer grid and a gray plane.

As for DIYing it, if you can come up with an idea how to upsample these patterns, then you can use numpy indexing. an expression like sensor[y::4, x::4] for some x and y would select the same specific pixel out of all tiles. [0::4, 0::4] would be the top left green pixel plane, [0::4, 3::4] would be the top right clear pixel plane, etc.

Or write your own loops and accelerate them with numba. It benefits from plain Python, without numpy functions. The code can even be parallelized. That needs to be done carefully. All the usual pitfalls of parallelization apply.

Answer inspired by a C-based Stackoverflow question:

OpenCV can be used to read in the image and then apply de-mosaicing.

import numpy as np
import cv2 as cv

# Read camera data file
raw_img = cv.imread("KAI_img_001.png")
# Demosaic raw img using edge aware Bayer demosaicing algorithm
dem_img = cv.demosaicing(raw_img, cv.COLOR_BayerBG2BGR_EA)
# Save demosaic-ed image
cv.imwrite("DEM_img_001.png", dem_img)

There is some more information in the opencv documentation about the alternatives to the COLOR_BayerBG2BGR_EA code that I used in the example. You may need to experiment with the right one so that it interprets/converts your raw image data correctly.

Hope this is relevant/helpful :)