Google Pixel Imaging USPs: Sensor fusion and Super Res Zoom - why don't all phone camera systems do this?

It remains a mystery to me to this day. Why don't other phone manufacturers and their cameras use the tricks that Google pulls off? An issue of patents perhaps? Or just lack of software expertise?

Regardless, the two 'tricks' demonstrated here are Sensor Fusion and Super Res Zoom. You'll find them on any Pixel in various combinations. On the Pixel 7 and 8 Pro and on the Pixel Fold (below) you'll find both techniques employed.

And yes, I already said that most people don't need to zoom more than 3x in normal life, but hey - this is for when zoom is needed. Perhaps at a sporting event or trying to spot rare wildlife.

To demonstrate both tricks, let's take a standard landscape shot - with an interesting central subject:

So far so good, and any camera phone could take this shot. Here scaled for the web, but don't worry, I'll show off some crops below too.

Now we try zooming at 2x:

In the case of my Pixel Fold, the main sensor is 48MP, so a 2x zoom smart crops into this sensor, using the central 12MP, and the Pixel Camera software is clever enough to handle the Quad Bayer colour stuff, to present an 'optical quality' 2x zoom. So far so good then.

Sensor Fusion

But now things get clever. Really clever. We zoom to 3x in the Pixel Camera interface:

This still looks pretty good, right? Normally when zooming beyond the optical/sensor limit in a digital camera you'd have to contend with digital zoom artefacts, but here that's not quite the case. Things look very sharp and precise. 

Except for the outer third or so of the frame. That's because the Pixels use Sensor Fusion and what you're seeing is the output of the 5x telephoto spliced into the centre of the frame of the main camera when zooming. Something along these lines (forgive the crude approximation):

So detail outside this spliced telephoto data is digitally zoomed (albeit not by much here, at 3x). We can see the effect better if we zoom again, to 4x:

The same sensor fusion is at work and in fact if we crop in then we can see the boundaries of the spliced telephoto shot and the digitally zoomed main sensor image. Look at this 1:1 crop from the bottom right of the image. The left hand side of the crop is mainly optical data from the periscope lens and the right hand side is digital zoom on the main sensor:

The two are skilfully melded together so that you shouldn't notice when viewing the photo as a whole. As indeed you didn't, two images above! It's something of an optical illusion because the eye latches onto a main subject or subjects, usually based around the centre of the frame, and the edges get ignored. So as long as the centre can be kept sharp then Google is onto a winner.

Good stuff. And then we get to the 5x image, which is (obviously) pure 5x optical telephoto on the Pixel Fold:

Despite my personal use case for zoom, i.e. largely not needed for family and trip snaps, I do appreciate a good telephoto lens when I see it and, in good light, the Pixel Fold's is pretty darned good. 

Super Res Zoom

The second trick up Google's sleeve is 'Super Res Zoom', which has been on (almost) all Pixels since the Pixel 3. This involves shooting a quick burst of photos when zoomed, relying on the minute hand movements of your good self to ensure that all photos in the burst are very, very slightly differently framed. 

Then, using clever software, sub-pixel interpolation is done, such that a 7x, or 9x, or, below, a 10x photo can have significantly more actual detail than a digital zoom on the last available optical camera (so 5x, in this case).

By way of proof, here's the same plane at 10x on the Pixel Fold, necessarily now focussing in on one of the engines:

Very sharp and detailed, and Super Res Zoom in action. I should put a caveat on this tech - since it relies on your tiny hand movements, it's not very good when there are large amounts of movement involved. So zooming in on a moving human, animal or (cough) plane will see the system inactive and traditional digital zoom beyond the optical limit.

However, it works very well for static subjects and landscapes, as here. 

Another plane, more Super Res Zoom data points, 6x through 9x

In case, you're not convinced by all this, let's move to another scene, another plane, let's cover all the Super Res Zoom points, and let's go vertical this time, so that you can see more of each image in this blog article format.

Here's 1x zoom on the Gannet (ex-navy) plane, on display:

Now 2x (still optical/lossless, remember):

Using sensor fusion for the central section, here's 3x:

And, using even more sensor fusion, here's 4x, you can just see the digitally zoomed lower quality detail around the edges, even in this rescaled version:

Then 5x, fully optical on the periscope:

Now we'll explore Super Res Zoom, using interpolation from the multiple exposures in each brief photo burst. Here's 6x:

Looking good. Here's 7x:

Excellent. Now to 8x, a full 3x zoom beyond the optical limit:

Perhaps a little processed here and there, but it could still pass for an unzoomed shot, taken closer, etc.

Here's 9x zoomed:

Again, static shots work best for Super Res Zoom, also subjects with artificial detail that can be predicted by algorithms. 

Now here's 10x zoom, matching the optical 10x on the likes of the Samsung S23 Ultra:

I'm impressed. And so should you be. Now, what if we push things ad absurdia? Let's go to 15x zoom. On a phone! Remember that this plane was a good 40m away from the Pixel Fold:

Details, at least in this web scaled shot, are just starting to become a little blurry. But overall I'm very impressed for a phone-shot zoom photo, especially from a folding phone with very little z-depth available for a big camera in its housings.

Of course, the larger sensors in Google's Pixel 7 Pro and 8 Pro do all this even better, but the Fold gets a long way towards similar multi-camera software tricks and results.

PS. If you like this feature and want to support my work then please do so here via PayPal. Thanks.