Google's RealFill AI magic fills photos with what's missing using reference images

Google has filed a trademark for a new AI technology called RealFill, hinting at a potential game-changer in the world of photo editing. According to Google, this approach promises to push the boundaries of image completion, offering a more authentic and faithful way to expand and inpaint existing photos.

Let’s delve into the details of RealFill, understand its functionalities, and explore how it compares to existing AI-based photo editing tools available for Google Pixel and Google Photos users.

As described in a research paper published by Google and Cornell University in 2023, RealFill tackles the challenge of “authentic image completion.” This means aiming to fill in missing regions of an image with content that should have been there with the original scene, ensuring the final result remains faithful to the captured moment. Below is a video sample of how RealFill will work:

On the official website, Google says RealFill leverages the power of reference images. It can utilize up to five reference photos to personalize a generative AI model, allowing it to capture the lighting, style, and content of the specific scene. This personalized model then fills in the missing regions of the target image, generating visually compelling results that are highly consistent with the reference photos.

Here’s another video demo of Google RealFill in action:

One of the key strengths of RealFill lies in its ability to handle diverse scenarios. The reference images don’t need to be perfectly aligned with the target image, and they can even be captured from drastically different viewpoints, lighting conditions, or with varying camera apertures and styles. This flexibility lets users leverage various reference points, potentially even utilizing older photos from their library to enhance existing images.

The results speak for themselves. RealFill consistently outperforms existing solutions like Paint-by-Example and Stable Diffusion Inpainting. The generated images showcase a remarkable level of detail and fidelity, seamlessly blending the filled-in regions with the original content. Here’s an of the side-by-side comparison as shared by Google:

While RealFill presents a significant leap forward, it’s not without its limitations. The fine-tuning process required for generating personalized models can be time-consuming, making it less suitable for real-time applications. This also suggests the feature, if and when it arrives on Google Pixel devices, will likely be cloud-based. Additionally, recovering the 3D scene can be challenging when there’s a significant discrepancy between the reference and target images, especially with limited reference points.

Despite these limitations, RealFill holds immense potential for various applications. It could revolutionize photo editing on Pixel devices, allowing users to expand photos, inpaint photos and even enhance old photos. For instance, if you capture a breathtaking landscape but miss a crucial portion due to the camera angle, RealFill could seamlessly fill in the missing area, creating a complete and captivating image. The feature could also help with repairing minor imperfections or removing unwanted elements from a photo by intelligently analyzing the surrounding content and seamlessly filling in the missing details. By utilizing reference images from similar scenes, RealFill could also improve the quality and detail of older photos.

Google Pixel devices and Google Photos already offer several AI-powered photo editing tools. Magic Eraser, for example, allows users to remove unwanted objects from photos with remarkable accuracy. However, Magic Eraser primarily focuses on object removal, while RealFill takes a broader approach, aiming to complete and enhance the entire image.

Similarly, Google Photos offers tools for automatic image enhancement, such as adjusting brightness, contrast, and color. While these tools improve the overall quality of the image, they don’t fundamentally alter the content. RealFill, on the other hand, actively generates new content to fill in missing regions, potentially leading to a more transformative image editing experience.

RealFill represents a significant step forward in AI-powered image editing. Its ability to leverage reference images for authentic image completion opens doors to exciting possibilities for photographers and casual users alike. While the technology is still under development, its potential to reshape the way we interact with and enhance our photos is undeniable.

As Google continues to refine RealFill, we can expect even more impressive results and potentially its integration into future Pixel devices or Google Photos. Whether it becomes a cloud-based solution or an on-device feature, the potential impact of RealFill on the world of image editing is undeniable. It raises intriguing questions about the future of digital media and the evolving definition of a photograph in the age of powerful AI tools.