Glass to Silicon - Dynamic Range
This week, we turn our focus (and the pun continues) to the sensor's dynamic range - the silicon in 'Glass to Silicon'. 🙂
Image Sensors and Sensitivity
Other than a handful of specific industrial applications, image sensors are CMOS-based, with every subsequent generation maximizing the amount of light captured by increasingly smaller sensor pixels. Developments from backside illumination (BSI) to deep trench isolation (DTI) to lateral overflow integration capacitors (LOFIC) continue to make these advancements practical for large-scale adoption. After all, photography demands both the quantity and quality of light.
Image sensors' qualities are tied to their sensitivity, for one. In good light, sensitivities remain low as there is already ample signal - light - to work with. This can be the case of a well-working code repository that is augmented with agent-friendly guides, so a model doesn't have to work _so hard or rather assumes less.
In low light, higher sensitivity is achieved by amplifying the electrical signals as photons hit, or by applying a digital gain post-exposure. The downside is that either approach may introduce noise to the output, which can be thought of as the hallucinations that models conjure occasionally in order to compensate for doing more guesswork, in a legacy code repository for example.
The adoption of dual-gain sensors attempts to expand the dynamic range of image sensors, ensuring they can capture both the highlights and shadows while minimizing noise, across differing light conditions.
Dynamic Range
We can think of highlights and shadows as two different characteristics of the codebase. The highlights can be thought of as the obvious signals: a well-understood organizational pattern, stack traces, metadata gathered from strong data lineage practices.
The shadows are then implied: code comments that once described operational procedures (and thus can't be checked at compile or run time), invariants that simply held true due to the robustness principle (be liberal in what you accept, be conservative in what you send) that have not been challenged at runtime by luck.
A model's dynamic range will thus be its ability to understand both at one, while minimizing its loss-in-the-middle problem. On the highlights, the agent blows it by becoming unusually fixated and motivated with a specific stack trace while ignoring the root cause a layer across. Overthinking in this regard starts treating the symptoms instead of the root cause.
On the shadows, they get crushed when the model failed to understand why the code comments matter (oblivious, or the comments were written to explain how) or bothered to validate them through tests. The model did read into its context window but failed to appreciate the signal from the noise.
Lost-in-the-middle is a known problem for models, where the information is capture in frame, but the sensor has failed to resolve the mid-tones. The fix here is not to switch lens but to use a sensor that can handle computational photography, by doing a multi-exposure bracketing at different sensitivities for that high dynamic range (HDR) look.
Bracketing
A simplified approach can be a three-pass workflow. The first pass exposes for the highlights, such as finding out what the main task is. The second pass looks for mid-tones, by mapping the surrounding code when translating to agentic process. The third pass lifts the shadows and looks for the implied patterns that might ordinarily become noise, such as being fully aware of why code comments. Merging these passes gives the team a HDR output, capturing the full-tonal range of the same context.
There is contrast, and there is contrast. HDR offers macro-contrast when both highlights and shadows are rendered in the same photo, while micro-contrast through using the right lens brings the fine details and defines the tonal handovers. Using an image sensor at higher sensitivities may appear to lend the same grain and grit at a brighter exposure but will also exhibit the same unintended consequences.
Taste and the Exposure Triangle
Ultimately, the characteristics of a good photo rest more on a tasteful application of balancing exposure and dynamic range. Overapplication of HDR looks are sometimes passed over, because everything is captured and nothing is left to the imagination. When a learned observer scan for guidelines, they see not only the framing but also the way the different tones communicate with them.
An intentional under- or over-exposure, paired with a tempered measurement of dynamic range, can add a certain level of texture that draws attention to craftsmanship. These are contrary to the desired behaviors of models, so this is perhaps the sharpest point to how the team can treat code craftsmanship as a whole.
They can walk the fully agentic line where a combination of models is trained on different focal lengths and sensitivities, then left to run the computational photography pipeline while the people focus (here it comes again) on content features. Alternatively, they can understand when to interplay highlights and shadows so that the people - the learned observers - can complement the same photography kit to offer that creative flourish. Not everything has to be darkened from in the highlights or boosted from shadows. Knowing when to choose what is the crux of it all.
For our finale, we will explore how we can borrow ideas from photography's exposure triangle for the team's agentic harnessing.
AI disclaimer: Have bounced ideas off, so while text is still my own, the way I arrived at some of them may contain minor influences from such interactions.