Smartphone cameras have made tremendous technological progress over the years; in fact camera improvements tend to take the center stage at launch events and in smartphone ads.
If you are reading this you are probably among the 74% of people that choose a new smartphone model based on its camera as one of the key factors.
It’s no surprise that companies increased their investments in camera technology, increased camera component costs, and even sacrificed design and device aesthetics to allow for larger and better cameras.
For example we can look at the iPhone camera bump evolution. We observe the trend going from fully flush in the iPhone 5 with no bump, to a massive 4 mm protrusion on the iPhone 14 Pro. This is a clear indication that camera performance is so important that sacrificing design makes sense.
Smartphone makers like Apple, Samsung, Xiaomi and others spend billions of dollars a year improving their hardware and their software, but are they doing enough? Are they getting the maximum performance out of the hardware that they release?
NO, they aren’t even close!
Look at the two pictures below. They are the same image captured by a mid-range Android smartphone.
The left image is what is saved by default by the phone to its photo gallery. The right one was captured by the same phone hardware but processed by Glass Imaging’s AI software.
How do current smartphone cameras form an image, and what is lacking?
To better understand what Glass AI is we need to look at how photos are formed.
Light emitted from the sun or a light bulb hits an object and reflects in many directions. A camera’s lens captures that light and focuses it on its sensor. The sensor is a planar electronic device split into many tiny pixels that can sense light. Some of these pixels are sensitive to green light, others to red or blue light. The sensor translates light intensity in each pixel into a digital value. High is bright, low is dark.
Reading those raw digital values from all the pixels and transforming them into a high quality full-color photo requires some more algorithmic work. Firstly, since each pixel only senses a single Red, Green or Blue color, an algorithm needs to guess the value of the other two colors for that location.
Furthermore, the raw data on the sensor is very noisy and needs to be cleaned up. Also sensed colors are not reflected exactly as in the real world, and need to be corrected.
The software image processing algorithms to do all this have been around for decades, and are based on hand-crafted logic and code tuned to try to invert the issues just described.
But is this optimal? Designing these algorithms is complicated and involves tuning knobs on lots of different software blocks that all have to work together; sometimes changing one part affects the others and vice versa. And despite years of iteration, these hand-designed algorithms make many heuristic trade-offs in design, in order to keep code manageable by the many engineers who work on them. The result is a decent, but not incredible image.
AI to the rescue
This is where Glass AI comes in. It replaces all these traditional algorithms with a single Neural Network that is trained to receive Raw data from a sensor and by precisely understanding the image formation process, generate a clean and noise free image.
“AI can do these things a lot better if properly developed and specifically tuned towards each smartphone camera model” says Tom Bishop, Glass CTO and co-founder.
But this is not all. Glass AI is also able to learn how to correct the imperfections of the lens which can be significant when using plastic low cost lenses, such as the ones typically found in smartphones.
An ideal smartphone lens is supposed to focus light onto a sharp spot in a single pixel, but in reality it creates some blurry spot that illuminates a few pixels. The result is image softness and lack of details when zooming in. Combined with image noise from small sensors, traditional algorithms often give crunchy, mottled looks to small image features.
Glass AI is learning that blur and reversing its effect to create a super-resolved image that is ultra sharp, yet clean of noise.
“We see this is the next revolution in computational photography and the impact on the smartphone industry will be huge” says Ziv Attar, Glass CEO and co-founder.
The impact of using AI in smartphone photography is so strong that using a $400 phone with a low cost camera can easily outperform many ultra expensive devices like the iPhone 14 Pro.
Here we compare images from the same mid range (around $500) Android device to an iPhone 14 Pro Max, which costs $1300 for a 512GB model.
It’s clear that AI is the right approach when it comes to dealing with small pixel effects, limited resolution and noise that plague current smartphone images.
Traditional algorithms have served us well so far, but it’s time for AI to shine its light on our photos.
What next and how far can AI take us?
Smartphone companies have in the last few years made improvements using traditional hand-tuned computational photography algorithms. These work by capturing a burst of multiple images and fusing them together, in effect digitally stabilizing a longer exposure.
Glass is taking these ideas to the next level with AI. We are developing an AI zoom mode which takes in a burst of raw images taken by a handheld smartphone, and generates up to an 8x zoom while preserving the image sharpness.
The ability of our AI to remove noise in the dark is also phenomenal, and using it to replace night mode on your existing phone will make your murky, awkward looking low-light images seem as if they were taken during daytime.
The great thing is that today’s smartphones have incredibly powerful GPU and Neural engine chips inside that are capable of running our highly optimized AI software. These were originally designed for other tasks such as amazing graphics in video games, or speech recognition, but we have been able to adapt our AI to run efficiently on these chips.
The team at Glass is on a mission to replace all traditional smartphone camera software with AI, pushing quality to levels beyond what professional cameras can deliver, and introducing new features that can only be achieved with AI.
