生成媒体的下一阶段：Google DeepMind 谈多模态世界模型、评估与真实工作流

Brief Description

This panel brings together Google DeepMind researchers and product leaders working on Gemini Omni, Veo, Nano Banana, reinforcement learning, and multimodal systems. They discuss newly launched image and video APIs, practical uses for generative media, the relationship between language and audiovisual world models, joint audio-video generation, evaluation and aesthetic defaults, the data needed to improve these systems, and why real customer workflows and expert craft are essential to turning frontier models into useful products.

Table of Contents

• New Models and APIs
• Practical Uses for Generative Media
• From Single Models to Video Agents
• Will Everything Converge Into One Model?
• Language, Code, and Intermediate Representations
• World Models and the Path From Understanding to Generation
• Why Audio and Video Belong Together
• Sensory Detail, Realism, and Human Preference
• Aesthetic Defaults, Prompting, and Expert Taste
• Evaluating Generative Media
• The Data and Workflows Models Still Need
• From Customer Feedback to Better Models

New Models and APIs

Moderator: Welcome back, both to those watching the stream and those here in person. We use these longer sessions between the main-stage keynotes to reflect on important developments that do not necessarily have a single launch moment. Today we are fortunate to have people working on Gemini Omni, Veo, Nano Banana, and some of the world's leading generative models.

We have Dumitru Erhan, who works on video models; Shane Gu, who works on Gemini reinforcement learning and Omni; and Nicole Brichtova, who works across the generative-media models, including Nano Banana and the products launched this week. Since you are closest to those launches, what came out this week, and what should people try?

Nicole Brichtova: We had two launch moments yesterday. First, we launched Nano Banana 2 Lite, our fastest and cheapest image model in the Nano Banana family. It is better than the original Nano Banana, so for most people it replaces the original model they used for generation and editing. It gets very close to the frontier quality of the larger models.

The roughly three-second latency unlocks many things in ideation and iteration. It is fun to use, and the quality has reached the point where it is useful not only for iteration but also for production-ready outputs.

Second, we launched the Gemini Omni Flash APIs that we had pre-announced at I/O. This is the first time we are making those APIs available to developers. They support high-quality video generation and editing, and we are pricing them the same as Veo 3.1 Fast. The goal is to provide very strong quality at an accessible price.

Moderator: When you first launched Omni, you did a podcast with Logan and added a sloth, ramen, and other elements to the footage. I wanted to do that to our videos, but I needed an API so I could automate the process. Thank you for finally making that possible.

Nicole Brichtova: That is my favorite use case. Everyone should try it. My version got a cat, which may be the least exotic animal. If you have not seen the examples, look them up. They are very funny. One of our colleagues did that work and is exceptionally good at finding creative ways to show what the model can do.

Practical Uses for Generative Media

Moderator: Cats and sloths are fun demonstrations, but what are the workhorse use cases that go beyond a demo?

Nicole Brichtova: There are two hero capabilities. The first is the ability to take almost anything as input and produce video. In the future we also want to expose other output modalities, but today you might provide a set of storyboard images, an audio track as a voice reference, or other materials and receive a video in return. That unlocks short-film production and creator workflows. We have also launched tools in YouTube Shorts to help creators make content more easily.

The second major capability is video editing. Natural language can be used to add or remove an element from a video. Adding a sloth is a playful example, but there are many practical applications. A consumer might clean up the noise in a beach-vacation video without knowing which traditional editing tools to use. Businesses are already using the models to create marketing and advertising campaigns. Education is another exciting area: media can be customized to a learner's knowledge level and preferred style. The APIs should reveal many more workflows than we can observe in first-party products.

Panelist: I recently saw a small but genuinely useful image example. My parents were visiting, and a device I bought for them had instructions only in English, with several diagrams. I photographed the page and asked Gemini to translate it into Romanian while keeping everything else unchanged. The result looked almost identical, with the diagrams and layout preserved and the text translated reasonably well.

The same idea extends to video. Omni's text rendering is unusually strong, which creates opportunities for translation, internationalization, redubbing, and broader access. These are not merely entertaining effects; they can make information useful to people who otherwise could not access it.

From Single Models to Video Agents

Moderator: In conversations with researchers at the frontier, I keep hearing that the next step is not simply a better single model but a video agent. The idea is to stop expecting one pass to do everything and instead let models work together through an agentic process. Does that terminology resonate, or is the direction still an open research question?

Shane Gu: What excites me most is symbolic foundation models and video foundation models working together. The early progress in generative images and video depended heavily on language models becoming good enough to provide detailed captions. Language is a powerful representation because it is universal, but it may also help with a fundamental machine-learning problem: spurious correlation.

When a feature is predictive, the model does not automatically know whether it is causal. One way to address that is to train on extremely diverse data containing many interventions in the causal graph. Another is to condition on causal information. Language can act as that causal conditioning. A description of how a video or image came to be often resembles a description of the causal structure behind it. That creates the possibility of richer generalization.

We published an evaluation paper titled "Video Models Are Zero-Shot Learners and Reasoners." The Nano Banana team later followed with related work using an image model. The central idea is that a video model can be a strong foundation model for spatial and temporal information. It can zero-shot many traditional computer-vision tasks, attempt visual puzzles, and display physical intuition relevant to robotics. There is still much to improve, but the combination of visual reasoning and text reasoning is powerful.

Whether this ultimately happens inside a unified model or through agentic coordination is likely to evolve incrementally. My expectation is that more of it will eventually move into one model. Today, however, there is already a great deal you can do by combining strong video and image understanding with Gemini in an agentic system and using Omni as a generative component. Our teams are actively exploring that space.

Will Everything Converge Into One Model?

Moderator: Does this all trend toward one product? You have several models today, but the name "Omni" suggests that eventually everything else could disappear into it.

Dumitru Erhan: Perhaps eventually, but there are engineering, research, and product trade-offs. Nano Banana 2 Lite serves a specific niche. It does not necessarily belong immediately in the same literal checkpoint as a system that can generate a high-resolution, long video. Those capabilities may not be trained most effectively in exactly the same way.

Five years from now, perhaps they will all be the same model. Six months from now, we will probably still have multiple models for different tasks because the practical trade-offs favor specialization.

Nicole Brichtova: We called it Gemini Omni because we wanted to hint at a future in which Gemini is fully multimodal in both directions. We will probably move toward Omni generating and editing images as well as video. But Dumitru is right that specialized models have many useful applications along the way. They meet concrete needs today, even if those needs or boundaries change a year from now.

There is also a research question about transfer between modalities. Image and video clearly share some structure. Generating audio and video jointly also makes sense because that is how the world presents them. But transfer between coding and video generation is less obvious. Combining them might produce useful shared learning, or it might waste capacity. The same uncertainty applies to intersections such as three-dimensional representations and code. We should explore those corners while staying focused on what people actually want to do with the models.

Language, Code, and Intermediate Representations

Moderator: One question that feels insufficiently answered is the correct intermediate representation. Captioning works naturally for images and can be extended over time for video, but it can feel inefficient. Perhaps code is better. Video is already generated and manipulated through tools such as FFmpeg and Manim. Is English all we need, or is a different representation likely to win?

Shane Gu: We work heavily on reinforcement learning, agents, and coding inside Gemini, so we are certainly exploring code as a representation. But a similar question has existed around language itself: why must chain-of-thought reasoning be expressed in natural language? Could it instead use arbitrary continuous tokens and simply allocate additional computation?

Adaptive test-time computation can improve results. Yet one reason natural-language reasoning works so well is the current training recipe. Pre-training scales and absorbs enormous amounts of knowledge and capability. Reinforcement learning can extract and refine it, but that process remains computationally intensive. When reasoning is tied to natural language, it can directly reuse the intelligence acquired during pre-training. Remove that constraint, and the system can no longer rely on the same representation as directly. Much of the recent progress in text and multimodal systems has been driven by text functioning as a strong backbone.

Nicole Brichtova: There is also a simpler product reason: text is how humans communicate. If you are building products that people will use, text will naturally appear somewhere in the interface, even if it is not the right representation for every internal operation.

Shane Gu: Some reinforcement-learning maximalists argue that chain of thought and language are unnecessary, that only additional computation matters. I do not think the issue is settled. But language has proved valuable because it connects computation to the intelligence learned during pre-training and to concepts people can communicate.

World Models and the Path From Understanding to Generation

Moderator: "World model" is another phrase we use frequently, but it is not very well defined. It sometimes seems as though everything is called a world model now.

Shane Gu: I recently gave a keynote at a world-model workshop and encouraged people to look at classic definitions from computer vision and model-based reinforcement learning. For me, a world model is essentially the model used in model-based RL: a model that can represent how the world changes and support planning or prediction. That definition is sufficient, even though researchers have proposed useful taxonomies and refinements.

Moderator: I still worry about language as a narrow pipe. If everything must pass through it, language becomes a lossy compression layer.

Shane Gu: We are not proposing language alone. We are proposing the video model and the language model together. Language by itself is not sufficient; that is why video is a complementary foundation model. People often see Veo or Omni as systems for making beautiful videos, but the larger vision is that video is a missing foundation model. It may be required for an artificial general intelligence that matches the breadth of human capability rather than excelling only in a narrow symbolic domain.

Moderator: Many people who began in computer vision have moved toward foundation models, image generation, or video generation. Is that transition as simple as reversing image-to-text into text-to-image?

Dumitru Erhan: I followed that path. I started in computer vision with object detection and recognition. Generation is not just the inverse network; it is a harder and more ambiguous mapping. Going from an image to the label "cat" is very different from going from the word "cat" to one of countless possible images. At the same time, progress in understanding can create synthetic labels and signals that help generation, so there is a loop between the two.

Nicole Brichtova: We have repeatedly seen that better understanding improves generation. Many image-understanding researchers became image-generation researchers, and some then moved into video, where there are many more dimensions to model.

Shane Gu: I recommend starting with understanding and recognition because those discriminative capabilities can lead to better generation, with reinforcement learning acting as a bridge. My own path moved through generative algorithms, reinforcement learning, robotics, dexterity, and then language models. Several years ago I concluded that symbolic AGI would advance faster than physical AGI, so I shifted toward language models. More recently I have enjoyed collaborating with the Omni team.

Researchers should expose themselves to the strongest people and ways of thinking in different communities. Today's video models remind me of early language models. At first, language models were creative demonstrations that could produce a story. Instruction tuning made them usable as chatbots, but they still hallucinated and followed instructions poorly. Once pre-training and post-training became strong enough for reliable reasoning, test-time scaling and reinforcement learning produced many of the best-performing systems.

Video models may follow a similar path. They will improve in instruction following and reduce spurious correlations until they become more reliable world models. Then spatial-temporal simulation from video can be interleaved with textual simulation to solve a broad set of intelligence problems.

Dumitru Erhan: One unresolved difference is that understanding and generation have not yet been fully unified in multimedia. There are not many frontier video models that are genuinely strong at both. Understanding and generation appear to be two sides of the same coin, so there is a case for putting them in the same system, even if we do not always build them that way today.

Why Audio and Video Belong Together

Moderator: Is audio as hard as video, or is it qualitatively different? Several years ago researchers used diffusion-like methods for audio. Humans experience audio and video differently, but perhaps a machine sees them as similar signals.

Dumitru Erhan: There are technical differences, but they are relatively minor. Audio became central to our work when we shipped Veo 3, which was among the first systems to perform true joint audiovisual generation. Other systems could assemble separate components under the hood, but this model generated the picture and sound together.

We made that choice because audiovisual events share one latent causal process. When someone speaks, the pixels showing the face and the sound of the voice are not independent. The lips must move in synchrony with the audio. Earlier pipelines often generated pixels first and then added a separate lip-sync layer, with poor results. Once users experienced a model with native audio, a silent video model no longer felt complete. Generating both modalities in one model was the right choice.

Shane Gu: One difference is that audio outside of speech is less fully verbalized. Text-to-speech is comparatively straightforward to describe, but how do you describe music, a person's precise tone, or pitch? Language may be inadequate. Similar limitations exist for taste, smell, touch, and subtle variations in skin appearance. Our sensory systems are highly sensitive to distinctions that mattered for survival, but our everyday vocabulary does not encode all of them.

A professional wine taster once told me that he borrowed language used to describe people and relationships because ordinary vocabulary was not sufficient to express taste. Sound can have the same problem.

Nicole Brichtova: To some extent, visual information has the same limitation. People with a developed visual palette or aesthetic sensitivity can perceive styles that are difficult to verbalize. Language constrains any attempt to describe rich sensory experience.

That is one reason to invest in world models and in perception and generation. Sensory information is a large part of how humans navigate the world and how embodied AI will navigate it. Language has taken us far and can take us farther, but it remains limiting in many of these areas.

Dumitru Erhan: I do not think we have yet pushed audio far enough to discover a fundamental ceiling. Most current weaknesses still look like solvable problems involving more data, better data, or better modeling. The immediate limit may be captioning rather than audio generation itself.

Nicole Brichtova: References help because it is hard to describe exactly how somebody speaks: their tone, prosody, disfluencies, and room acoustics. Entire technical vocabularies exist that most users do not know. Language models may also have gaps in domains that received less attention during training, and those gaps become visible in generation because the media system relies on language understanding as an intermediary. We need to keep pushing these areas before claiming to know their true ceilings.

Sensory Detail, Realism, and Human Preference

Moderator: As a podcaster, I think about sound in three broad domains: music, voice, and effects. Even within voice, room sound changes the experience. A large room, a small room, a car, a phone, and an in-person conversation all sound different. These properties can be labeled, but we perceive them in much richer ways.

One common tell of an AI video is audio that is too studio-perfect because the training material was recorded in studios. A world model is needed even for sound: if I move farther away, my voice should become softer and more diffuse. Immersive video needs to model that spatial relationship.

Shane Gu: That is a good example of information for which language is not precise enough. If a caption fails to represent the echo or environmental noise, generation conditioned on that language remains highly multimodal and unpredictable. Ideally the latent representation should capture most of the stochastic variation, so the final generation conditioned on it becomes much more determinate.

Nicole Brichtova: Facial expressions are another case. People are extremely sensitive to micro-expressions, gaze, skin texture, and the way a listener nods or reacts. We have improved dramatically in a year, but video has not fully crossed that chasm. Images may be closer: many generated images now look indistinguishable from photographs at first glance.

Dumitru Erhan: We once ran an experiment asking whether generated videos could look better than real videos. We took descriptions of real clips, generated equivalent clips, and showed them side by side in a human evaluation. People often preferred the generated versions.

That does not mean the generated clip was more useful or more realistic. It may simply have been sharper, more saturated, more HDR-like, or more flattering. We optimized qualities that trigger a quick positive reaction. On deeper inspection, the real video might still be more coherent or informative.

Shane Gu: Sensitivity also varies by person and culture. I spoke with a manga artist who found generative imagery unsettling because the eye gaze was subtly wrong. A small misalignment that most people ignore can feel deeply unnatural to an expert.

Dumitru Erhan: That is why human preference alone is not a reliable target. Asking whether people like one image more than another may optimize superficial attraction without producing the outcome you actually want.

Aesthetic Defaults, Prompting, and Expert Taste

Shane Gu: Several years ago, people debated whether prompt engineering would disappear. I argued that it should not, because specifying what you want is how you control a model's output. Models can now perform much more automatic prompting and often produce something sufficient, but users should keep refining their sensitivity and describing the remaining differences rather than accepting the first generated result.

Nicole Brichtova: There is a large difference between the untrained eye and somebody who has honed a craft for decades. Designers, architects, artists, and other experts see things the average person does not. If you poll people on the street, they may prefer a very smooth, highly saturated image—the equivalent of an Instagram filter. That is not necessarily the right default for every task.

Our models need strong instruction following so users can steer away from the default. Sometimes language is enough; sometimes visual or audio references are more effective because language cannot express the desired difference.

Moderator: As a product leader, do you feel pressure from having to set an aesthetic default for the world? Every system has a default, whether or not anyone intended to make a cultural decision.

Nicole Brichtova: We have thought about it. When Nano Banana Pro came out, there was an explosion of generated infographics. Our default aesthetic was sometimes too cluttered, as if an overeager student had learned a great deal about a topic and tried to put every fact into one image. The effect also varies by language and cultural style; Japanese infographics, for example, often favor much higher information density.

Dumitru Erhan: We faced similar decisions with Omni. Near the end of training and tuning, we had to compare styles and decide whether we preferred more muted or more saturated colors. Those are real trade-offs.

Nicole Brichtova: The decisions often fall to modeling and product teams. It is fair to ask whether we are the best people to set them or whether we should work more closely with art directors and people with a distinctive creative perspective. We use trusted testers and weekly feedback sessions, and we take their input seriously. Internal creative experts also test checkpoints and notice things the core team misses.

A tester may report that an optimization ruined the detail in grass, or notice a strange repeated artifact such as wedding rings appearing on too many hands. Preference-based training can create spurious correlations and reward hacking in unexpected ways. These issues are easy to miss when developing a model and obvious to somebody who uses it intensely.

Evaluating Generative Media

Moderator: How do you evaluate video beyond relying on a few exceptional testers? Surely there must also be quantitative methods.

Dumitru Erhan: Improving Gemini improves automatic evaluation for video, but the problem remains very hard.

Nicole Brichtova: Some properties are relatively objective. For an infographic, optical character recognition can reveal a malformed letter. If the text is wrong, the asset may be unusable. Those cases are easier to evaluate automatically.

Aesthetics and open-ended video quality are much harder. We rely extensively on human evaluations. Every new model adds capabilities, which creates more dimensions to test. When two checkpoints are close, a group of us may literally sit in a room and compare videos side by side, asking which output we prefer.

Free-form video editing is especially difficult because the possible requests are nearly unlimited. Even a seemingly simple capability such as adding an object requires evaluation across visual consistency, audio, semantics, and instruction following. There is no single "add a sloth" benchmark that covers the entire problem.

The evaluation scale is larger than a roomful of reviewers. We run human evaluations over thousands of examples, use live experiments to detect small differences at greater scale, and develop automatic raters. Automatic evaluation is mature for language models but much more nascent for media. Human judgment therefore remains important, especially feedback from people with a strong aesthetic point of view and from professionals who use the models in daily workflows.

A model can perform well on a benchmark slice and still break a real workflow. Early-access programs allow us to observe those failures and incorporate feedback before a broad release.

Shane Gu: Human evaluation work should gradually be amortized into better automatic systems. Video understanding is an important intelligence task, especially for detecting failures in generated media. Some problems are aesthetic, but others are semantic. A generated reconstruction of a famous movie scene may look plausible while containing an inconsistency that makes no sense in context. An AI evaluator should eventually detect that.

When I assess generated videos today, I can feel clever because I still notice errors the model misses. We should work to close that gap. Strong video understanding matters beyond simple preference or beauty, and it should eventually reduce the need for repetitive human labeling.

The Data and Workflows Models Still Need

Moderator: Many companies want to offer you video, gaming, real-world footage, images, or labeling services. What data do you actually need? Sorting signal from noise is difficult, so it would help the ecosystem to know what makes a useful proposal.

Dumitru Erhan: It is hard to answer without revealing details of our projects. In general, we already have a great deal of data, so quality matters more than indiscriminate volume. Embodied data is relevant to our robotics work. For Omni, professionally produced, high-quality material can be more useful than another random collection of internet video. We are always looking for such sources.

Nicole Brichtova: For agentic systems, the trajectory of a real task is valuable and difficult to manufacture. Consider a marketing campaign. A person might start with a photograph of a new product, turn it into a video advertisement, and then adapt it into multiple assets for different platforms. The sequence of decisions, iterations, and transformations is important.

That information usually lives inside the products where people actually do the work. A generic vendor often does not have access to the full workflow. We may also lack a first-party surface for a particular professional task. If someone can capture those authentic trajectories well, we would like to hear from them.

Moderator: Media and marketing teams have endless amounts of work. Some of it is artisanal and deserves Hollywood-level attention, but much of it is repetitive and should be modelable.

Panelist: We need both sides: high-quality craft and large volumes of ordinary production work.

Nicole Brichtova: Even an ordinary marketing campaign is not created in five minutes. People iterate, compare alternatives, and choose one option for reasons that may never be written down. Perhaps the gaze felt correct, the composition matched the brand, or a small detail communicated quality. Model builders may not know these things because we are not marketing directors.

Shane Gu: The same is true in natural language. Most information is still inside people and can be extracted only through active dialogue and trust. The internet often contains the final outcome, such as a paper or an article, but not the trajectory: the starting point, the conversations that sparked an idea, the failed approaches, or the moment of inspiration. Creative work contains a great deal of this tacit or "dark" knowledge.

Nicole Brichtova: A novel often speaks to a reader because the author has a personal connection to its story, trajectory, or characters. Much of today's language-model writing falls into default patterns: the wording, structure, and descriptions feel similar, and readers quickly sense that there is nothing personal to connect with. That missing expertise and experience also matters in generative media.

From Customer Feedback to Better Models

Shane Gu: Google Cloud and Google DeepMind have been investing more in forward-deployed engineers for product and creative generative-media work. Public data alone has limits. Partnership with domain experts can create better models and products.

I define post-training broadly as everything between pre-training and the final user experience. Forward-deployed engineering should not be reduced to sales. The role can help a customer use a model, but it should also derive insights that benefit both sides. Those engineers can build the harness around how a customer uses the system while carrying real feedback upstream to the modeling teams.

Nicole Brichtova: It is extremely useful for us to speak directly with people using the models and understand where they struggle. An interior designer might need to reproduce a pattern consistently across ten rug sizes, including a custom size, but the model changes the pattern. Someone doing virtual try-on for earrings needs the jewelry and the person's head to remain at physically meaningful scales. These failures matter in real work even if the modeling team never thought to test them.

Advertising introduces the idea of a brand language. A company may provide images and PDFs describing that language, but the concept remains ambiguous. IKEA's identity is not merely "blue and yellow"; the exact shades, proportions, materials, composition, and visual tone matter. A random blue and yellow result is not necessarily IKEA. Frontier-model research alone does not automatically capture such requirements.

We want to build products that solve concrete tasks rather than remain research artifacts. To do that, we need to understand what users and experts actually care about, incorporate those requirements into evaluation, and create a feedback path all the way to the model.

Moderator: The progress in Nano Banana, Veo, and Omni over only a few years has been remarkable, and there is clearly much more to explore beyond coding. Thank you for your time. I cannot wait to see what comes next.

Panelists: Thank you for having us, and thank you, everyone.