Research

Computational photography is an emerging branch of imaging sciences combining the design of optics, image sensor electronics, and a computational system working in concert to produce an image. The cardinal difference with conventional photography is that the image formed by the optics on the sensor (or a system of sensors) is no more necessarily intelligible to the human eye, but does contain all the information that the computational process needs in order to reconstruct an image. Computational cameras promise to revolutionize photography by reducing the camera cost, form factor, and energy requirements as well as allowing novel imaging capabilities such as hyperspectral or depth imaging.  One of the focuses of our research is the use of modern machine learning techniques in computational imaging, and in active and passive 3D imaging in particular.

The pursuit of small form factor, low cost, ever-growing resolution, and uncompromised light sensitivity is putting stringent constraints on the design of modern mobile cameras, such as those we carry with us inside our smartphones. Form factor reduction coupled with the increase in resolution reduces the amount of light collected by a single pixel and results in noisy images. To some extent, this can be countered by increasing the aperture area (decreasing the ratio between the focal distance and the aperture diameter that photographers refer to as F#). However, a decrease in F# reduces the depth-of-field of a camera, meaning that only objects in a limited interval of depths around the point at which the camera is focused will stay in focus, while closer and farther objects will appear blurred. To address this challenge, we designed a special optical system with a phase-encoded aperture.

By placing a piece of glass with tiny a elevated pattern in the pupil of a regular imaging lens allows to focus different wavelengths at different depths in the scene — for example, the red channel may be focused on nearby objects, the green in the middle of the scene, while distant objects stay in focus in the blue channel. An image created by such a camera will appear weirdly blurred to the human eye, but a specially designed computational process recovers a sharp image at a very large range of depths, thus extending the camera’s depth-of-field while still enjoying the light efficiency of a large aperture. We employed machine learning techniques producing not only an optimal way to perform image reconstruction but also simultaneously optimizing the parameters of the optical system that we actually manufactured. As a bonus, our system produces a high-quality depth image indicating the distance of the objects from the camera — with a single exposure, single aperture, and no active illumination of the scene.

We also work on exotic image sensors and imaging systems combining coded active illumination and a combination of several imaging modalities.

Research highlight: Every modern digital camera contains a piece of hardware known as an image signal processing pipeline (ISP) that is responsible for taking the raw pixels from the sensor and producing a good-looking photograph or video. ISPs typically have multiple stages performing demosaicing (interpolating the missing pixels created by the color filter array superimposed on top of the pixels that allows color imaging), denoising, and color and intensity correction just to mention a few.

These steps are performed independently from each other, which results in a waste of silicon area and power. We designed DeepISP — an end-to-end learned image processing pipeline that does all these steps at once. Our experiments show that our design produces high image quality (both objective and subjective) and is more computationally efficient. Our design is particularly good at enhancing and removing the noise from low-light images.

Before the deep learning landslide, we worked on classical aspects of computer vision such as model fitting and feature detection. Nowadays, our focus has shifted more toward a healthy combination of domain-specific modeling and learning.

The reconstruction of the first CT images in the ‘70s could take hours and used very basic algorithms, which still earned the Nobel Prize in medicine to its inventors. With the increase of computational power at our service, the use of more sophisticated signal processing and machine learning promises to revolutionize the field.

Our research focuses on several aspects of medical imaging and medical data analysis. Firstly, we employ modern machine learning techniques in order to improve the quality of the images produced by existing imaging modalities and extend their applicability to previously infeasible settings. This, for example, includes real-time improvement of contrast and frame rate of medical ultrasound. Secondly, we use data modeling and signal processing in order to create new imaging modalities such as a combination of acoustic excitation with regular EEG imaging allowing to capture the electric activity of just a small portion of the brain and separate it from the incredible amount of electromagnetic interference produced by the surrounding brain areas. Lastly, we use machine learning methods to analyze medical data in order to diagnose and predict pathologies.

3D is also often a much more compact way of representing objects, otherwise described by a very large number of 2D projections.

The advent of ubiquitous low-cost depth sensing technologies now allows virtually anyone to generate 3D data. Microsoft Kinect and Intel RealSense made 3D imaging an affordable commodity, while Apple, in its iPhoneX, was among the first to make a 3D sensor our intimate companion. If the history of digital imaging can be used to extrapolate the trend in depth imaging, we are going to face an exponential growth of user-generated 3D content. In order to enable future 3D sensing-based technologies such as autonomous agents that can interact with the environment, it is crucial to provide automatic tools and algorithms for manipulating and understanding 3D data.

Several reasons make understanding and manipulating 3D data challenging. The first reason is representation. While digital images are represented as 2D matrices, in 3D this is not the case. Representations come in many forms and are mainly determined by the type of a sensor and the target application. Secondly, 3D data are presently much scarcer. Most state-of-the-art algorithms are based on learning from huge amounts of data. Without them, special care has to be taken so as to structure the learning using domain knowledge. Another big challenge is transformation invariance: a large portion of the world around us may undergo non-rigid transformations. Treating each deformation as a distinct shape would be intractable and therefore special tools are required to incorporate invariance assumptions.

Our research focuses on several aspects of 3D vision. Firstly, we design and build sensors allowing to digitize the geometry of the physical world — both dedicated 3D cameras as well as computational techniques allowing to infer depth as a byproduct in some form of imaging. We have accumulated a vast experience with all the facets of the designing, productizing and manufacturing of depth sensing technologies. Secondly, we work on solving fundamental problems in computational shape analysis such as establishing correspondence between shapes that undergo a deformation and, in addition, are viewed only partially and in the presence of clutter; determining transformation-invariant similarity between shapes; finding compact representations for geometric objects allowing their efficient retrieval in a large scale database; and manipulating deformable objects such as the human body or face and synthesizing them ex novo. We also build cool applications on top of these methods.

The tremendous advantages of DNNs come at the price of computational complexity. Modern deep neural models typically contain millions of parameters occupying many megabytes of memory and require hundreds of millions of arithmetic operations per output. This gives rise to significant hardware architecture challenges both during training and inference, especially in real-time and other time-critical applications. The most commonly used hardware accelerator for DNNs, are graphics processing units (GPUs); while being able to deliver the required computational throughput, their power consumption precludes the use of GPUs in energy-constrained settings. For this reason, GPU-based DNN accelerators are rarely found on battery-powered platforms such as drones, smartphones, and Internet-of-things devices, which typically opt for custom low-power hardware, such as FPGA and ASIC chips. While significantly reducing the power footprint, FPGAs and ASICs are constrained by a relatively modest amount of on-chip memory, which puts severe limitations on the size of the deep neural models that can run on such chips. One of the focuses of our research is to design a full stack of deep learning from efficient hardware architecture and its FPGA implementation up to hardware-aware training algorithms that make hardware-related tradeoffs explicit and satisfy various constraints such as power, silicon size, and memory footprint.

Another low light of modern deep learning techniques is their poor theoretical explanation (some people even compare deep learning to alchemy). Our research aims at elucidating the workings of deep neural networks by bridging between the empirically successful field deep learning with the theoretically founded world of data models, sparse and redundant representations, and compressed sensing. Borrowing tools from these domains gives a fresh new look and a tiny step toward answering one of the biggest questions in deep learning: why the heck it works?

Lastly, while being spectacularly successful on ordered data such as time signals and images, deep learning faces many barriers when it encounters unordered data such as point clouds, meshes, and graphs. For example, it is not even clear what is the analogy of a convolutional neural network on a graph. The need to learn from such types of data arises from a variety of fields ranging from social networks to computational biology and drug design. We research into the use of various geometric tools, in particular, those borrowed from 3D vision and computational shape analysis, allowing to construct DNNs on non-Euclidean domains and unstructured data.

Research highlight: Fifty years ago, Christian Anfinsen conducted a series of remarkable experiments on ribonuclease proteins — enzymes that “cut” RNA molecules in all living cells and are essential to many biological processes. Anfinsen showed that when the enzyme was “unfolded” by a denaturing agent it lost its function, and when the agent was removed, the protein regained its function. He concluded that the function of a protein was entirely determined by its 3D structure, and the latter, in turn, was entirely determined by the electrostatic forces and thermodynamics of the sequence of amino acids composing the protein. This work, that earned Anfinsen his Nobel prize in 1972, led to the “one sequence, one structure” principle that remains one of the central dogmas in molecular biology. However, within the cell, protein chains are not formed in isolation, to fold alone once produced. Rather, they are translated from genetic coding instructions (for which many synonymous versions exist to code a single amino acid sequence) and begin to fold before the chain has fully formed through a process known as co-translational folding. The effect of coding and co-translational folding mechanisms on the final protein structure are not well understood and there are no studies showing side-by-side structural analysis of protein pairs having alternative synonymous coding.

In our works, we used the wealth of high-resolution protein structures available in the Protein Data Bank (PDB) to computationally explore the association between genetic coding and local protein structure. We observed a surprising statistical dependence between the two that is not readily explainable by known effects. In order to establish the causal direction (i.e., to unambiguously demonstrate that a synonymous mutation may lead to structural changes), we looked for suitable experimental targets. An ideal target would be a protein that has more than one stable conformations; by changing the coding, the protein might get kinetically trapped into a different experimentally measurable conformation. To our surprise, an attentive study of the PDB data indicated that a very considerable fraction of experimentally resolved protein structures exist as an ensemble of several stable conformations thermodynamically isolated from each other — in clear violation to Anfinsen’s dogma.