Creating Advanced Interactive Movies

QuickTime 4.1 introduced a set of new features for authors and tool developers that allow for the creation of even more advanced, interactive movies. These features included

The current version of QuickTime also introduces new features for manipulating text tracks in QuickTime movies, along with other capabilities.

Embedded Movies

Embedded movies are implemented through the track type––the movie track, which is discussed in the section Movie Track and Movie Wired Actions.

Creating New Types of QuickTime Movies

Because embedded movies can have independent clocks, new types of movies can be created.

For example, you can create a movie containing animated characters that are watching a video. This movie could contain an animation track, perhaps a sprite or Flash track, and an embedded movie track for the video content. In this example, the root movie’s time base would control the animation, but the video’s rate could be controlled independently from the animations’ rate. Wired actions could be sent to the embedded movie when a user clicks on buttons in the animation track. The wired actions could play, pause, and fast forward the video, or switch to a new one.

Another way you can take advantage of independent time bases is to allow long audio tracks to be triggered interactively. For example, if you create a game with sound effects and background music that need to be played back at times defined by events that occur in the game, you can use an embedded movie for each audio track. The advantage of using an embedded movie instead of a music track with a custom sound is that the entire sound does not need to be loaded into memory, so it is appropriate for longer sounds. The disadvantage is that you may not control it similar to a MIDI instrument.

Using Embedded Movies

One way to use embedded movies is to break projects into components, allowing portions to be reused in other projects, and simplifying the authoring process. In QuickTime, this technique could be used in a Web browser using external movie-to-movie communication. In QuickTime, a single movie can be created that is playable in the QuickTime Player or any application that plays QuickTime movies.

For example, an interactive movie could contain three elements, as illustrated in Figure 7-1:

  • A control strip with custom controls for audio and scene navigation

  • An area used to display text

  • An area used for interactive animation

Figure 7-1  An interactive movie example with three elements
An interactive movie example with three elements

By encapsulating these three elements each in a separate movie, you can reload only portions that are necessary to reload. For example, the control strip may persist throughout the entire experience, while the animation area may be replaced once per scene and the displayed text changed several times per scene. By implementing each element as a movie and composing them together using movie tracks in a container movie, you can minimize reload time and memory usage, and even update the source movies on your Web server as you improve or change them.

Dynamically Loading Embedded Movies From URLs

A movie track maintains a list of movies that may be loaded and played within the track. The movie track plays only one movie from the list at a given time. This list is initialized from data in a movie track sample, but the list may be augmented at runtime. Each entry in the list is identified by a unique ID, and contains a data reference to a movie. There are two wired actions, which allow you to add a new URL data reference to this list, and to load and play a movie from this list.

Dynamically loading a movie into a movie track is similar to loading a URL into a frame of a Web page. This dynamic loading allows for movies to manage memory efficiently by loading QuickTime playable content as it is needed. In addition, it allows for content which is generated on a Web server to be loaded into a movie; this content could even be created based upon information that a wired movie sends to the server using the GotoURL wired action.

Triggering Wired Actions When an Embedded Movie is Loaded

A MovieLoaded QuickTime event allows wired actions to be executed when an embedded movie is loaded. There are two places to store these actions. The movie that is being loaded may store actions in its movie properties atom container, and the movie track may store these actions in its samples.

These wired actions can be used to examine the current state of the parent movie, and to make changes both to the movie being loaded and to elements of the parent movie.

Targeting Elements of Embedded Movies with All Wired Actions

All wired actions may be performed on elements of embedded movies, and wired action handlers inside of embedded movies may perform actions on elements of their parent movie.

To understand this targeting hierarchy, we can look at the current runtime state of a movie as a tree, as shown in the example in Figure 7-2. The root of this tree is the root movie itself. The children of the root movie node are tracks. Some types of tracks, such as the sprite track, can contain child nodes that are sprites. In earlier versions of QuickTime, this was as deep as the tree ever got, but with the introduction of the movie track, the tree can be indefinitely deep. The movie track node has child track nodes based on its currently loaded movie; one or more of these tracks may be yet another movie track.

Take, for example, a movie containing a Flash track and a movie track. The movie track’s currently loaded movie contains a video track, an audio track, and a sprite track with two sprites. This constitutes the target hierarchy tree shown in Figure 7-2.

Figure 7-2  An example targeting hierarchy tree
An example targeting hierarchy tree

Conceptually, you may think of the movie track and its currently loaded movie as a single entity––that is, a movie track. A movie track such as the child movie in this example may be the target of both track and movie actions. When specifying a target for an action, you first specify the movie (or movie track) that is the target or that contains the target. Then, you can further specify a track and track object if needed.

These are the ways you can target various elements within a hierarchy:

  • external movies by movie name or movie ID

  • child movie track by track ID, track index or track name

  • movies higher in the hierarchy as the parent movie or the root movie

These targets are relative to the current movie that contains the action handler. A few examples may help clarify how to target various elements.

Example 1: Sprite 1 has an action handler on a mouse click that tells the root movie to play.

Since Sprite 1 is contained in the child movie, the movie target is specified relative to the child movie. This may be accomplished by targeting the parent movie or root movie.

Example 2: Sprite 2 has an action handler that tells the Flash track to pan left.

Again, the target is relative to the child movie, so you can use either the parent movie or the root movie to specify the root movie. Additionally, you specify the Flash track.

Example 3: A button in the Flash track contains a mouse click handler that sets the volume of the audio track in the child movie. In this example, Flash track is contained in the root movie, so the movie target is specified relative to the root movie. You can use any of the child movie target types to specify the child movie. You can specify that the target is the audio track.

Target Type Atoms for Hierarchical Movies

There are target atoms to accommodate embedded movies in QuickTime. They allow for paths to be specified in a hierarchical movie tree.

Target movies may be an external movie, the default movie, or any movie embedded within another movie. Targets are specified using a movie path that may include parent and child movie relationships, and may additionally include track and track object target atoms as needed.

By using embedded kActionTarget atoms along with parent and child movie target atoms, you can build up paths for movie targets. Note that you look for these embedded kActionTargetAtoms only when evaluating a movie target, and any movie target type may contain a sibling kActionTargetAtom.

Paths start from the current movie, which is the movie containing the object that is handling an event. You may go up the tree using a kTargetParentMovie atom or down the tree using one of five child movie atoms. You may use a kTargetRootMovie atom as a shortcut to get to the top of the tree containing an embedded movie and may use the existing movieByName and movieByID atoms to specify a root external movie.

The target atoms are described below. Note that there are five ways to specify an embedded child movie. Three of them specify movie track properties. Two specify properties of the currently loaded movie in a movie track.

kTargetRootMovie (leaf atom –– no data)

The root movie containing the action handler.

kTargetParentMovie (leaf atom –– no data)

The parent movie.

kTargetChildMovieTrackName
[Pstring movieTrackName]

A child movie track specified by track name.

kTargetChildMovieTrackID
[QTAtomID movieTrackID]

A child movie track specified by track ID.

kTargetChildMovieTrackIndex
[long movieTrackIndex]

A child movie track specified by track index.

kTargetChildMovieMovieName
[Pstring movieName]

A child movie specified by the currently loaded movie’s movie name. The child movie must contain movieName user data with the specified name.

kTargetChildMovieMovieID
[QTAtomID movieID]

A child movie specified by the currently loaded movie’s movie ID. The child movie must contain movieID user data with the specified ID.

Example #1

Movie "Root" contains two embedded movies. "Controller" is an embedded movie controller movie. "ControlMe" is an embedded audio/video movie to be controlled. "Controller" and "ControlMe" are both child movies of the Root Movie.

To control the rate of the movie’s audio/video, a sprite in the “Controller” movie uses the following target:

kActionTarget
    kTargetParentMovie
    kActionTarget
        kTargetChildMovieMovieName
            [movieName = "ControlMe"]

Example #2

A sprite in one movie targets a sprite which is embedded in a movie which is again embedded in another movie.

You target a sprite named "Dude" in a track of index 1 in a movie of ID 2 which is nested in a movie whose track name is "Rad", which is nested in a movie whose user data name is OuterExternalMovie. OuterExternalMovie is an external movie to the one that is executing an action handler.

kActionTarget
        kTargetMovieName
                [name = "OuterExternalMovie"]
        kActionTarget
                kTargetChildMovieTrackName
                        [name = "Rad"]
                kActionTarget
                        kTargetChildMovieMovieID
                                [ID = 2]
                        kTargetTrackIndex
                                [Index = 1]
                        kTargetSpriteName
                                [spriteName = "Dude"]

Movie Track and Movie Wired Actions

Two actions operate on a movie track target:

This action adds a movie URL data reference to the targeted movie track’s array of movie data references. The URL data reference is added with the specified ID. If a data reference with the same ID already exists, it is replaced. It is generally a good idea to use an ID that is not contained in the movie track's sample, since this may be reloaded under some conditions.

This action loads a movie specified by childMovieID as the current movie being played by the movie track. The movie replaces the current movie.

Another action operates on a root movie or a movie track target:

kActionMovieRestartAtTime (TimeValue startTime, Fixed rate)

This action restarts the targeted movie at the specified movie time, restarting it at the specified rate. If rate is set to 0, then the current movie rate is used. More specifically, this action stops the current movie, changes the movie’s time to the specified time, and then prerolls the movie from that time at the specified rate.

Note that the wired actions DoScript, GotoURL, DebugString, and StatusString are sent through the root movie’s MCDoActionProc when executed from a child movie, allowing them to work with existing applications that trap for these actions.

Movie Controller Actions

mcActionDoScript allows a movie to send requests to execute scripts of various sorts to a host application. The parameter is of type QTDoScriptPtr.

struct QTDoScriptRecord {
        long scriptTypeFlags;
        char *command;
        char *arguments;
};
typedef QTDoScriptRecord *QTDoScriptPtr;

These are the constants currently defined for the scriptTypeFlags field:

enum {
        kScriptIsUnknownType = 1L << 0
        kScriptIsJavaScript = 1L << 1,
        kScriptIsLingoEvent = 1L << 2,
        kScriptIsVBEvent = 1L << 3,
        kScriptIsProjectorCommand = 1L << 4
};

For more information, see the explanation above of the new wired action kActionDoScript.

mcActionRestartAtTime

This allows a movie to be restarted at a particular time and rate.

The parameter is a QTRestartAtTimePtr.

struct QTRestartAtTimeRecord {
    TimeValue       startTime;  /* time scale is the movie timescale*/
    Fixed           rate;       /* if rate is 0, the movie's current
                                rate is maintained*/
};
typedef struct QTRestartAtTimeRecord QTRestartAtTimeRecord;
typedef QTRestartAtTimeRecord *QTRestartAtTimePtr;

For more information, see the explanation above of the new wired action kActionMovieRestartAtTime.

Wired QT Event

kQTEventMovieLoaded event was added to QuickTime 4.1. This event is sent when an embedded movie is loaded. Embedded movies may be loaded when a movie containing an embedded movie is first opened, when an embedded movie loads a new sample due to the movie’s time changing, or when an embedded movie is sent a kActionMovieTrackLoadChildMovie action.

Action handlers for the kQTEventMovieLoaded event may reside in two places. They may be placed in a sample of a movie track’s media or placed in the movie property atom of a movie that is loaded into a movie track.

If handlers exist in both places, the action list from the sample is appended to the one from the movie property atom. This means that the actions from the sample will be executed second, and if the same action resides in both places for the same target, the effect of the action from the sample will persist.

To add an event handler to a movie media sample, you add an atom of type kQTEventMovieLoaded to the sample, with child atoms defining the actions.

To add an event handler to a child movie that is to be loaded, you add an atom of type kQTEventMovieLoaded to the child movie’s movie property atom using the new QuickTime Movie Toolbox routine SetMoviePropertyAtom. You may use the GetMoviePropertyAtom function to first retrieve the existing movie properties container, add or modify the kQTEventMovieLoaded atom, and then write it back using the SetMoviePropertyAtom function.

Extended Wired Operand Functionality

A special case was added to the wired operand kOperandComponentVersion.

By using the arguments kOperandComponentVersion("mac ", "os ", "vers" ) the version of Mac OS is returned. 0 is returned on Windows.

Wired Actions and JavaScript

kActionDoScript (long flags, CStr commands, CStr arguments)

This new wired action has no target (system target).

The action calls the root movie controller’s mcActionDoScript, so that scripts can be invoked by a host application. For example, the QuickTime Plug-in in a browser can invoke JavaScripts.

If the script flags are set to kScriptIsUnknownType or kScriptIsJavaScript, the QuickTime Plug-in invokes a JavaScript routine in the HTML file that has embedded the QuickTime movie, with the following prototype:

function DoFSCommand(command, arguments) { }

If the movie is embedded with a NAME tag, a movieName tag, or is named by user data, then this prototype is used instead:

function movieName_DoFSCommand(command, arguments) { }

This allows for wired movies to invoke a JavaScript. The QuickTime Plug-in also supports many movie-related JavaScript routines, so it is possible to set sprite track variables and post custom wired events to a wired movie from JavaScript as well. It is important to note that this functionality works only with the QuickTime Plug-in and that some versions of some browsers do not support the necessary interfaces to allow for these things to work.

Movie Property Atom Toolbox Routines

Similar to the GetMediaPropertyAtom and SetMediaPropertyAtom routines, QuickTime 4.1 introduced GetMoviePropertyAtom and SetMoviePropertyAtom routines. These routines allow an atom container of structured data to be associated with a movie. This information is saved in a movie resource.

The kQTEventMovieLoaded looks for an atom of type kQTEventMovieLoaded in a movie's property atom container when a MovieTrack loads a new movie.

GetMoviePropertyAtom (Movie theMovie, QTAtomContainer * propertyAtom)

This routine allocates and returns in propertyAtom an atom container containing a copy of theMovie’s properties’ atom container.

SetMoviePropertyAtom (Movie theMovie, QTAtomContainer  propertyAtom)

This routine sets the contents of theMovie’s property atom container to the contents of the propertyAtom atom container.

Custom Wired Actions

Developers may supplement the set of built-in wired actions by using a plug-in mechanism. You may write custom action handler components to perform new types of actions. For example, you could write a math library component to perform complicated computations quickly, returning the result by setting a sprite track variable. Another use would be for allowing a custom media handler to be scripted using wired actions.

Custom wired actions in a movie are routed to these components for handling. They are passed information about the current QTEvent, the movie element that is to be the target of the action, the default movie element that received the QTEvent and generated the action, the type of the action, and the parameters of the action. The wired action expression evaluation machinery has been exposed as a single API call, allowing general wired expressions to be passed as parameters.

Custom Action Handler Usage

Custom action handler usage falls into one of two categories:

  • as a stateless subroutine library

  • as an object that maintains state across multiple custom action executions

When used as a subroutine library, the movie author simply scripts custom actions. QuickTime opens an instance of the specified custom action handler, passes it the action to execute, and then closes the component. When used as an object that maintains state, the movie author needs to open an instance of the handler with a unique instance ID. In this case, QuickTime keeps the action handler component open until the movie is closed. When making calls to an action handler that has been opened, the unique instance ID is specified, allowing QuickTime to route the request to the correct component instance.

Note that there is currently no way to specify that the handler is a component that has already been opened by QuickTime, such as a media handler or codec that is in use. One special case has been included for developers authoring media handlers that support custom wired actions. If the component description of the custom action’s target matches that of the target media handler, then the custom action is sent to the instance of the media handler that already in use. This means that the movie author does not need to (and should not) open an instance of the component.

When authoring a movie with a custom action, the type of component used to execute it is specified, along with an instance ID. If the component is being used as a simple subroutine library, which does not need to keep track of any state, then the ID may be set to 0. If a particular instance of an open action-handling component is intended to be used, then the ID is used to refer to it. You may use the kActionOpenCustomHandler to open the component with a particular ID. This ID should be obtained by using the kOperandUniqueCustomHandlerID to ensure that your movie will work after edits are performed. After opening a custom action handler you may use kOperandCustomActionHandlerOpen to determine that the component was indeed found and opened successfully.

If the instance ID is set to 0 and the component description matches that of the media handler that is the target of the action, then the media handler is used to execute the custom action.

Authoring Custom Wired Actions

To specify that an action is to be handled by a custom handler, you add the kCustomActionHandler atom as a child of the kAction atom. You define which type of component is to handle the action by adding a kCustomHandlerDesc atom, and optionally specify a particular handler instance ID using the kCustomHandlerID atom.

Extension to Wired Movie Format: Executing Custom Actions

kActionAtom
    <kCustomActionHandler, 1, 1>
       kCustomHandlerDesc, 1, 1
            [ComponentDescription handlerDesc]
        <kCustomHandlerID, 1, 1>
            [long handlerID]

You define parameter atoms as usual and have access to the parameters through an API for writing custom action handlers. This means that parameters may be wired expressions and your component may fetch the result of the evaluated expression.

Wired Actions

kActionOpenCustomHandler
    Supported Flags: None
    Param 1: [long handlerID]
    Param 2: [QTCustomActionHandlerRecord

Opens a custom action-handling component that may be referred to later by its handler ID.

Typically, you would first use the kOperandUniqueCustomActionHandlerID operand to obtain a unique ID. After storing this in a variable, you use this action to open the handler. Then you can use kOperandCustomActionHandlerOpen to determine if the handler was found.

Wired Operands

kOperandUniqueCustomActionHandlerID
    No Params

Returns a unique custom handler ID that may be used with kActionOpenCustomHandler.

kOperandCustomActionHandlerOpen
    Param 1: handlerID

Returns true if a handler with the specified ID is open, otherwise false. This may be used to determine if the component specified by kActionOpenCustomHandler was found and opened.

Writing a Custom Action Handler Component

Any component type, whether it is a media handler, a codec, or your own component type, may be extended to handle custom actions. To extend your component, you implement the ExecuteWiredAction routine described below. If you are writing a component that is being used only to handle custom actions, you should use the component type 'wire'.

EXTERN_API( ComponentResult )
CallComponentExecuteWiredAction (ComponentInstance  ci,
                                QTAtomContainer actionContainer,
                                QTAtom  actionAtom,
                                QTCustomActionTargetPtr target,
                                QTEventRecordPtr event);

All the state passed to you in this routine is only valid for the duration of the execution of your custom action, which should be completed when you return from this routine.

The actionContainer contains all of the actions that are being executed in response to the current QTEvent. This atom container should not be edited, only read from, since other actions that have yet to be executed may be contained within it.

The actionAtom specifies the kActionAtom that concerns you, since it contains all of the atoms describing the action type and parameters to your components custom action that are to be executed.

The target parameter specifies the movie elements that you need in order to execute your action.

struct QTCustomActionTargetRecord {
    Movie                           movie;
    DoMCActionUPP                   doMCActionCallbackProc;
    long                            callBackRefcon;
    Track                           track;
    long                            trackObjectRefCon;
    Track                           defaultTrack;
    long                            defaultObjectRefCon;
    long                            reserved1;
    long                            reserved2;
};
typedef struct QTCustomActionTargetRecord QTCustomActionTargetRecord;
typedef QTCustomActionTargetRecord * QTCustomActionTargetPtr;

The movie field is the movie that is or contains the movie element, which is the target of the action.

The doMCActionCallbackProc and callBackRefcon specify the movie controller and refCon for the target movie’s movie controller. They may be used with the CallDoMCActionProc routine to invoke the movie controller functionality, including the new mcActionFetchParameterAs.

The track field is the track that is or contains the movie element, which is the target of the action.

The trackObjectRefCon field is the refCon or ID of the movie element that is the target of the action. If the target is a sprite, this will be the sprite ID.

The defaultTrack is the track that contains the movie element that handled the QTEvent and generated the custom action. For example, if a hypertext element of a text track generated a SetSpriteVisible action for a sprite in a sprite track, the defaultTrack would be the text track, while the sprite track would be the target track.

The defaultObjectRefCon is the refCon or ID of the movie element that handled the QTEvent and generated the custom action.

The event specifies information about what QTEvent generated the custom action.

The Action Being Executed

If your component defines multiple custom action types, then you can determine the action type by looking at the kWhichAction atom, which is a child atom of this kActionAtom. The data of this kWhichAction atom is a long defining the action (and needs byte flipping for Windows).

Fetching the Parameters

The parameters to your custom action may be the result of a wired expression. You don’t have to be concerned about analyzing the kActionParameter child atoms: the wired expression evaluation machinery has been made accessible via a movie controller action called mcActionFetchParameterAs.

You fill out a QTFetchParameterAsRecord and pass it to mcActionFetchParameterAs as in:

CallDoMCActionProc(resolvedTarget->doMCActionCallbackProc,
                    resolvedTarget->callBackRefcon,
                    mcActionFetchParameterAs,
                    &fetchAs, &handled );
 
struct QTFetchParameterAsRecord {
    QTAtomSpec                      paramListSpec;
    long                            paramIndex;
    long                            paramType;
    long                            allowedFlags;
    void *                          min;
    void *                          max;
    void *                          currentValue;
    void *                          newValue;
    Boolean                         isUnsignedValue;
};

You set the container and atom of the paramListSpec to the actionContainer and actionAtom passed to ExecuteWiredAction().

You set the paramIndex to the index of the parameter you wish to fetch. If you allow a variable number of parameters, you may count how many child atoms of type kActionParameter the actionAtom has.

You set the paramType to one of the parameter types from the following enumeration:

enum {
    kFetchAsBooleanPtr                  = 1,
    kFetchAsShortPtr                    = 2,
    kFetchAsLongPtr                     = 3,
    kFetchAsMatrixRecordPtr             = 4,
    kFetchAsModifierTrackGraphicsModeRecord = 5,
    kFetchAsHandle                      = 6,
    kFetchAsStr255                      = 7,
    kFetchAsFloatPtr                    = 8,
    kFetchAsPointPtr                    = 9,
    kFetchAsNewAtomContainer            = 10,
    kFetchAsQTEventRecordPtr            = 11,
    kFetchAsFixedPtr                    = 12,
    kFetchAsSetControllerValuePtr = 13,
    kFetchAsRgnHandle                   = 14,   /* flipped to native*/
    kFetchAsComponentDescriptionPtr = 15,
    kFetchAsCString                     = 16
};

You set the allowedFlags to flags from the following enumeration, or 0 fetching without constraints.

enum {
    kActionFlagActionIsDelta        = 1L << 1,
    kActionFlagParameterWrapsAround  = 1L << 2,
    kActionFlagActionIsToggle       = 1L << 3
};

If allowedFlags is not set to 0, you set the min, max, current, and isUnsignedValue fields to appropriate values based on the data type being fetched.

The newValue field returns the result of the parameter.

For scalar and structure types, you pass a pointer to the appropriate data type and it will be filled in.

For Handle, RgnHandle, and Cstring, you pass in a new empty handle that will be resized as needed, you are responsible for disposing the handle when done.

For kFetchAsNewAtomContainer, you pass in a pointer to a non-allocated QTAtomContainer. On return, this container will contain the contents of the single child atom of the parameter atom and all of its children. This lets you pass arbitrary data that is not evaluated in an atom container as a parameter.