Previsualization for Apple Immersive: Learnings from “Open Hearts” by The Weeknd

Hello everyone, I'm Anton Tammi. Today I'm happy to share the benefits of previsualization that I learned when directing the first Apple Immersive music video, Open Hearts for The Weeknd.

I've directed music videos for a decade now. Like many in my generation, I started with independent music videos surrounded by like minded colleagues, learning directing. With international community of filmmakers. We shared knowledge and tips with each other both offline and online. Eventually, my portfolio grew and big music labels and artists became familiar with my work.

In 2019 and 2020, I directed four music videos for The Weeknd, most well known of them being Blinding Lights.

Then last year, The Weeknd asked me to direct two more videos for him. The first one was dancing in the flames, shot entirely on iPhone 16 Pro and then.

Then the second one. It was the first ever Apple Immersive music video for Apple Vision Pro. I was completely new to the immersive format and being the first to ever direct an immersive music video, well, it felt special.

And I knew this time I wouldn't be able to ask help from the filmmaking community in the same way as before.

The immersive format is something so new for us filmmakers, and after talking to you guys, I. I know that there are people in this room who have never tried an immersive camera yet. Well, I was just like you a year ago, but knowing.

But working with the Apple Immersive team, I began to realize what an exciting world this is. So how did I start thinking and planning my first immersive film as a director? Well, first we needed a story. Together with The Weeknd, we wrote a two part concept, a story that symbolized the end of his career through a surreal journey we envisioned him crashing his car and finding himself in the back of an ambulance, where he would then discover a mysterious door leading to a long corridor. And finally, we would see him in a venue where the film would reach its climax with The Weeknd's final performance.

The first video would be shot on the iPhone 16 Pro, and it would be ending with an ambulance arriving to pick The Weeknd after the car crash. Then the story would continue in the Apple Immersive format, where we would see The Weeknd inside the ambulance. Now storyboarding a music video shot on iPhone was something I was familiar with, and I'm sure many of you are, but what I didn't know was how do you storyboard and communicate the pre-production for an immersive video? So first of all, I had no idea what would or would not work in immersive format. Does it even make sense to have your actor inside an ambulance when shooting with an immersive camera? Would the immersive camera even fit in the back of an ambulance. How fast could the car move without causing Motion discomfort? How do you frame an immersive and where exactly should the camera be positioned? I had no idea, but I knew a classic 2D storyboard wouldn't give me these answers.

And then on top of that, of course, I had an A-list talent to work with. When you're directing with an artist like The Weeknd, everything must be planned perfectly before he steps on set.

He's a global superstar, and with him, you just don't get that time to test, shoot and rehearse.

I also wondered, how can I explain to him how this video would actually end up looking like, if even I had never directed an immersive video before? So I was facing with two challenges working with a major artist and a completely new format. And I realized my team and I needed a space to explore and test these ideas before we shoot.

We basically needed a virtual environment where I could try easily all my wildest ideas before bringing The Weeknd on set. We needed an immersive previs workflow.

So what's an immersive previs? Well, a previs or a previsualization or an animatic is a CG animated video that represents the final film.

It's like a modern version of a storyboard. Ultimately, it gives you a space to experiment, test ideas, and refine your vision. And I mean, of course, a storyboard is sometimes enough when planning a traditional film. And of course, you don't always need a storyboard. But thanks to the modern tools we have now, we have programs like Blender or Unreal Engine available to create CG visualizations. If you make films that really, really need specific planning.

I especially like blender since it's open source and easy to use. It basically allows you to turn your entire script into a set of immersive animations to get going. You first need. Well, you need a team. I chose to collaborate with the one previous artist called Nick Turner, and together only me and Nick. We did the whole previous together.

Nick had created some custom camera rigs in Unreal Engine, and using these custom set of camera blueprints, we had an access to virtual camera in the CG space, allowing us to frame scenes however we wanted.

And then we were able to export stereoscopic versions of the animation. Every frame was exported simultaneously for a slightly different perspective to match the Apple Immersive camera, and after rendering, I could view them in Apple Vision Pro to see how they actually looked like. This workflow allowed me to test all my ideas quickly and affordably, and I'm really happy that we did it, to be honest.

So here, for example, you can see one of our stereoscopic previs exports rendered for App Nap playback, where The Weeknd is walking through a corridor, and when you start rendering immersive privacies, you quickly realize what works differently in immersive compared to the traditional filmmaking.

Since the viewer is watching a 180 degrees half sphere around them. You can't cheat the way you can in classic filmmaking. Everything is visible and fully sharped, and the depth of field is super sharp and there's basically no depth of field. That's how I feel it. And the wide framing reveals every single detail, and you begin to understand how precise and detailed you really have to be. You also learn how close objects are allowed to come to your camera, and how uncomfortable it gets when something actually gets too close.

So the great thing about our previous was that we added this red sphere around our camera, and it defined the minimum distance of comfortable proximity, meaning that it basically showed me what was too close and I could easily turn it on and off to check if my ideal framing was feeling comfortable in Apple Vision Pro. So basically this allowed me to quickly check if things were technically okay or not.

Or for example, I was thinking at some point that I would be showing The Weeknd sitting in a sports car, but after running the previous tests, I realized that in immersive, it feels more natural when you can feel like a like a larger size of space around you and the tight sports car. Well, it just felt small. It made me feel claustrophobic, almost.

And okay, so this one is crazy. I tried rotating 180 degrees rolling camera move, but when I previewed the render, I honestly felt almost like almost fell off my chair.

So if you want to try that once in your life, I guess everything should be tried once, but it's a it's not the most pleasant experience.

But yeah, I mean, as we kept testing, I also started to learn what actually works in immersive.

So let's look at some of these examples.

For example, I learned that the room like space in the back of an ambulance actually works really well in the immersive format as it gives you a sense of space big enough space bigger than the sports car.

I also learned that corridors work really well, placing the camera in the center and moving slowly forward behind your talent creates this feeling that you're walking with them.

Or, as Eliot mentioned yesterday in Immersive Movement, can really be a powerful tool.

Through the previous, I learned that what it actually means and what type of movement really works. So, for example, I discovered that if you position the camera in the center of the road and show the car's point of view with the part of the car visible in frame, you can actually capture very fast movement.

But how do you.

Yeah. Like how do you get like, you kind of like, need to get grounded by something like the hood of the car here. Like it's something that you can always find in a short if you can, like, like a thing that helps you to feel like you are. You are. You are somewhere just not so that you're floating. So all this, all this testing taught me basically that I can do these things that I'm used to do, like fast shots. I did blinding lights. It has a lot of fast car shots. And I was wondering if this video could have those. But I learned with these these ways they actually can.

And what I love about this workflow is that it adds a new phase of craft to filmmaking, and it can often lead even into new ideas that you might not have without it. So, for example, in this road shot, I knew I wanted a POV of the ambulance driving along on an endless straight road. But when we're working on this previs and I saw this shot, I started to wonder, like, what if the road would bend upward as you drove through it? How would it feel in immersive since you have more space above you? So we try it. And this is how this complex VFX shot was already proven to work in Vision Pro before we went to shoot.

And then when we went on set, my VFX supervisor and I, we knew exactly what kind of plates we needed in order to make something this ambitious possible in Apple Vision Pro.

So here you can see the final shot in 2D.

The previous basically allowed me to experiment and test even the wildest ideas freely and affordably, keeping only the ones I knew for sure would work.

So what do you do after finishing the previous then? First of all, I was able to show the full film to The Weeknd himself, hear his comments and get his approval.

Secondly, the previous helped me to educate my team. I showed the previous to my cinematographer Erik Henriksen, and he had he had already worked with me and The Weeknd before. But neither him nor his team had ever shot in the immersive format.

So after seeing the previous, Eric shared the information with his team.

So basically, at the core of all our pre-production was the previous a pre-made stereoscopic animation that everyone we brought in could watch in Apple Vision Pro and kind of see the film before it was made.

That's how everyone knew how to design the sets, how to light the scenes, and how to control the camera movement.

And after completing this workflow and having now directed an immersive film, I can I can tell you this and we can talk about it after this speech. If we see outside, it is not that difficult or necessarily not much more expensive or even that different compared to directing regular films. But and I think now that I'm talking to you on this stage and everything, I sound like a film school teacher in Finland who always, like, gave me these things. What you're supposed to do. But I guess I'm becoming one now. When I say that you do have to prepare more carefully, and you have to spend more time in pre-production. You can't just go run and gun.

So an immersive previs helped me to test different ideas in a cost efficient way. It served as the perfect communication tool for the whole team, allowing me to show something that was impossible to explain with words. It made sure that everyone from my talent to technical crew and myself were all on the same page. And most importantly, since this was a music video for The Weeknd, it allowed me to be fully prepared and confident when he walked on set.

But yeah, what happens then when you move beyond the pre-production into filming? What needs to be done differently on set when you shoot immersive films? Well, here to share more about working with these new technologies is Alex.