iQ lab – Camera Test
Back in 1997, the test laboratory at Image Engineering – iQ lab – began testing the image quality of cameras. Nearly every day over the last 20 years, the lab has been testing cameras for many prominent international magazines (i.e., Color Foto) and manufacturers.
The iQ lab – Camera Test is our original test, and it is one that we regularly adapt to the latest IS standards and camera technologies. Each camera that we receive, we test with two predefined lenses for reliable comparability of the measurements. One lens is suitable for all quality characteristics and the other a regular zoom lens for time measurements.
If you are interested in camera results or want a measurement of your device, you can order the iQ lab – Camera Test or purchase already existing measurement data. We offer the results in two different presentations: A PDF-overview of all numerical results combined with the test images or a test report of all numerical results. The latter also includes an assessment of the results, done by the Image Engineering experts, combined with the test images.
The results of every iQ lab – Camera Test (ordered by magazines) also immediately run into our database, iQ-Data, which already consists of more than 200 consumer cameras and 250 system cameras.
If you have further questions or need information on pricing, please contact us.
Image quality measurement
Reflective
With the multipurpose test chart TE42, we can gain information about resolution, texture loss, shading, distortion, lateral chromatic aberration, sharpening, and color reproduction at once.
For the iQ lab – Camera Test the camera is tested at all ISO speeds (except ISO200) at f-stop 5.6 and in best JPEG quality. RAW images are also available on request.
The following characteristics are measured for each of these settings (the iQ-Analyzer is used to gain the information) and reported.
Resolution measurement
Using the sinusoidal Siemens star chart, the resolution is measured in the center of the chart and the image corners according to ISO12233:2014. The stars are divided into segments (eight for the center star and three for the corner stars) and the resolution is measured for each of them. The obtained spatial frequency response (SFR) is calculated per segment and per star.
The SFR contains a lot of information about sharpness and limiting resolution at the various positions and orientations. The obtained data can be used to check the optical centering of the system, to analyze optical errors like astigmatism, to verify orientation-specific or frequency-specific image processing and to determine the limiting resolution. Also, the acutance as a representation of the sharpness is determined and reported.
The multipurpose test chart enables us to gain an incredible amount of information at once, and we also ensure that all characteristics of the image quality are captured under the same conditions.
Texture loss measurement
The so-called texture loss is a critical parameter in the objective image quality assessment of today’s cameras. Especially cameras build in mobile phones show significant loss of low contrast and fine details which are hard to describe using standard resolution measurement procedures (e.g., as defined in ISO12233:2014).
The combination of very small form factor and high pixel count leads to a high demand for noise reduction in the signal-processing pipeline of these devices. The Dead Leaves pattern is used for quite a while in this context. For an extensive assessment, the TE42 has two Dead Leaves patches, one high contrast and one with a low contrast.
The evaluation method of the Dead Leaves pattern is called “DeadLeaves_cross” and is described in detail in this paper
Sharpening measurement
The TE42 contains four slanted edges, whereby two different contrasts are used, and each is available in horizontal and vertical orientation. The edges are used to evaluate the resolution according to ISO12233:2014. While the Siemens star is more robust against sharpening, the slanted edges are used to evaluate the sharpening behavior of the camera in detail.
Different contrasts are used as the sharpening algorithms in the image signal processor of today’s cameras may detect the edge contrast and adjust the sharpening according to this. The different contrasts of the edges are a low contrast edge (60% edge modulation) and a high contrast edge (80% edge modulation contrast).
Color reproduction measurement
The color patches around the OECF patches are created concerning the Calibrite ColorChecker SG and are used for evaluation of the color reproduction. The color of these patches is well known and has been evaluated using a spectrometer.
The image data and the reference data is converted into color coordinates in the CIE L*a*b* color space that represents the color reception of the human visual system. From these values, the color distance ?E (Delta E) that describes the color reproduction quality is calculated.
Transparent
To determine the optoelectronic conversion function (OECF) an LE6 and a TE241 (1.000.000:1) is used. The camera under test is measured at all ISO speeds, mainly because the higher the ISO speed, the more noise found in the image. Higher noise potentially leads to a lower dynamic range.
The predefined lens for image quality testing is always set to aperture 5.6. For each of these settings following characteristics are measured (the iQ-Analyzer is used to gain the information) and reported.
OECF and noise
The OECF describes how a digital camera transfers luminance into digital values. The curve is specified for all three color channels red, green and blue in color images.
Also, the signal to noise ratio (SNR) and the visual noise (VN) are calculated according to ISO 15739. While the SNR describes the relationship between the signal and noise, the VN is evaluated as output-referred noise. It takes into account that the visual perception of noise can be different for human observers compared to a basic SNR approach.
Dynamic range
The dynamic range (DR) is calculated from the OECF as well. It describes the maximum scene contrast the digital camera can reproduce. The lightest point is chosen at the illumination level where the camera reaches its maximum output value. The darkest point is the illumination level where the SNR level passes the value of 1 according to ISO 15739.
According to our experiences, a very flat curve of a black level clipping can cause problems with some cameras. Therefore a value of 3 was selected – as this threshold is also used for all regular tests Image Engineering produces, comparability is given.
Used digital values
Used digital values can be determined by analyzing the OECF. The OECF curve should start at a digital value of 0 and go up to 255 to utilize the complete contrast available at 8 bit. For 16 Bit data, the range is 0 to 65535.
White balance
If the automatic white balancing works well, the curves for the three channels should lie on top of each other. If the average difference is greater than five digital values, the images show a visible color cast.
Timing measurement
Timing measurements are performed according to ISO 15781 using a LED-Panel in the so-called iQ-AF Box. It consists of 10 rows of ten LEDs that light up at a selectable frequency. Shutter lag, shooting time lag, start-up-time and frame sequence are measured.
To measure shooting time lag, the LED-Panel is started with a microswitch connected with a cable. The microswitch is mounted on display, and both “switches” are pressed and released simultaneously. Beforehand the device is defocused so the focus time is included. In the pictures, the elapsed time can be determined by counting the LEDs that have already been lit up.
This measurement is executed with two illuminations:
- 800 lux
- 30 lux
For the determination of the shutter time lag the device under test is prefocused on the target and then released. The difference between shooting time lag and shutter time lag is the time the device under test needs to focus (focus time).
Additionally, the frame sequence is measured. Therefore the continuous-mode is activated, and so many images are taken until the device slows down or the card is full. Some cameras increase compression or decrease pixel count for extra fast burst mode. In our test, only images with full pixel count and low compression are used.
The time needed to get ready to shoot after the camera-app was started (start-up time) is a significant value and is also determined by using the LED- Panel. The start-up time includes the shooting time lag.
These three tests are only executed with illumination of 800 lux.
