Chaining Filters for Complex Effects

Every Core Image filter produces an output CIImage object, so you can use this object as input to another filter. For example, the sequence of filters illustrated in Figure 1-1 applies a color effect to an image, then adds a glow effect, and finally crops a section out of the result.

Core Image optimizes the application of filter chains such as this one to render results quickly and efficiently. Each CIImage object in the chain isn’t a fully rendered image, but instead merely a “recipe” for rendering. Core Image doesn’t need to execute each filter individually, wasting time and memory rendering intermediate pixel buffers that will never be seen. Instead, Core Image combines filters into a single operation, and can even reorganize filters when applying them in a different order will produce the same result more efficiently. Figure 1-2 shows a more accurate rendition of the example filter chain from Figure 1-1.

Notice that in Figure 1-2, the crop operation has moved from last to first. That filter results in large areas of the original image being cropped out of the final output. As such, there’s no need to apply the color and sharpen filters to those pixels. By performing the crop first, Core Image ensures that expensive image processing operations apply only to pixels that will be visible in final output.

Listing 1-2 shows how to set up a filter chain like that illustrated above.

Listing 1-2  Creating a filter chain

func applyFilterChain(to image: CIImage) -> CIImage {

    // The CIPhotoEffectInstant filter takes only an input image

    let colorFilter = CIFilter(name: "CIPhotoEffectProcess", withInputParameters:

        [kCIInputImageKey: image])!

    // Pass the result of the color filter into the Bloom filter

    // and set its parameters for a glowy effect.

    let bloomImage = colorFilter.outputImage!.applyingFilter("CIBloom",

                                                             withInputParameters: [

                                                                kCIInputRadiusKey: 10.0,

                                                                kCIInputIntensityKey: 1.0

        ])

    // imageByCroppingToRect is a convenience method for

    // creating the CICrop filter and accessing its outputImage.

    let cropRect = CGRect(x: 350, y: 350, width: 150, height: 150)

    let croppedImage = bloomImage.cropping(to: cropRect)

    return croppedImage

}

Listing 1-2 also shows a few different convenience methods for configuring filters and accessing their results. In summary, you can use any of these methods to apply a filter, either individually or as part of a filter chain:

• Create a CIFilter instance with the filterWithName: initializer, set parameters using the setValue:forKey: method (including the kCIInputImageKey for the image to process), and access the output image with the outputImage property. (See Listing 1-1.)

• Create a CIFilter instance and set its parameters (including the input image) in one call with the filterWithName:withInputParameters: initializer, then use the the outputImage property to access output. (See the colorFilter example in Listing 1-2.)

• Apply a filter without creating a CIFilter instance by using the imageByApplyingFilter:withInputParameters: method to a CIImage object. (See the bloomImage example in Listing 1-2.)

• For certain commonly used filter operations, such as cropping, clamping, and applying coordinate transforms, use other CIImage instance methods listed in Creating an Image by Modifying an Existing Image. (See the croppedImage example in Listing 1-2.)

Using Special Filter Types for More Options

Most of the built-in Core Image filters operate on a main input image (possibly with additional input images that affect processing) and create a single output image. But there are several additional types of that you can use to create interesting effects or combine with other filters to produce more complex workflows.

• A compositing (or blending) filters combine two images according to a preset formula. For example:

  • The CISourceInCompositing filter combines images such that only the areas that are opaque in both input images are visible in the output image.

  • The CIMultiplyBlendMode filter multiplies pixel colors from both images, producing a darkened output image.

  For the complete list of compositing filters, query the CICategoryCompositeOperation category.

• A generator filters take no input images. Instead, these filters use other input parameters to create a new image from scratch. Some generators produce output that can be useful on its own, and others can be combined in filter chains to produce more interesting images. Some examples from among the built-in Core Image filters include:

  • Filters like CIQRCodeGenerator and CICode128BarcodeGenerator generate barcode images that encode specified input data.

  • Filters like CIConstantColorGenerator, CICheckerboardGenerator, and CILinearGradient generate simple procedural images from specified colors. You can combine these with other filters for interesting effects—for example, the CIRadialGradient filter can create a mask for use with the CIMaskedVariableBlur filter.

  • Filters like CILenticularHaloGenerator and CISunbeamsGenerator create standalone visual effects—combine these with compositing filters to add special effects to an image.

  To find generator filters, query the CICategoryGenerator and CICategoryGradient categories.

• A reduction filter operates on an input image, but instead of creating an output image in the traditional sense, its output describes information about the input image. For example:

  • The CIAreaMaximum filter outputs a single color value representing the brightest of all pixel colors in a specified area of an image.

  • The CIAreaHistogram filter outputs information about the numbers of pixels for each intensity value in a specified area of an image.

  All Core Image filters must produce a CIImage object as their output, so the information produced by a reduction filter is still an image. However, you usually don’t display these images—instead, you read color values from single-pixel or single-row images, or use them as input to other filters.

  For the complete list of reduction filters, query the CICategoryReduction category.

• A transition filter takes two input images and varies its output between them in response to an independent variable—typically, this variable is time, so you can use a transition filter to create an animation that starts with one image, ends on another, and progresses from one to the other using an interesting visual effect. Core Image provides several built-in transition filters, including:

  • The CIDissolveTransition filter produces a simple cross-dissolve, fading from one image to another.

  • The CICopyMachineTransition filter simulates a photocopy machine, swiping a bar of bright light across one image to reveal another.

  For the complete list of transition filters, query the CICategoryTransition category.

Processing Still Images in UIKit and AppKit

UIKit and AppKit provide easy ways to add Core Image processing to still images, whether those images appear in your app’s UI or are part of its workflow. For example:

• A travel app might present stock photography of destinations in a list, then apply filters to those images to create a subtle background for each destination’s detail page.

• A social app might apply filters to user avatar pictures to indicate mood for each post.

• A photography app might allow the user to customize images with filters upon capture, or offer a Photos app extension for adding effects to pictures in the user’s Photos library (see Photo Editing in App Extension Programming Guide).

In iOS and tvOS, you can apply Core Image filters anywhere you work with UIImage objects. Listing 1-3 shows a simple method for using filters with an image view.

Listing 1-3  Applying a filter to an image view (iOS/tvOS)

class ViewController: UIViewController {

    let filter = CIFilter(name: "CISepiaTone",

                          withInputParameters: [kCIInputIntensityKey: 0.5])!

    @IBOutlet var imageView: UIImageView!

    func displayFilteredImage(image: UIImage) {

        // Create a Core Image image object for the input image.

        let inputImage = CIImage(image: image)!

        // Set that image as the filter's input image parameter.

        filter.setValue(inputImage, forKey: kCIInputImageKey)

        // Get a UIImage representation of the filter's output and display it.

        imageView.image = UIImage(CIImage: filter.outputImage!)

    }

}

In macOS, use the initWithBitmapImageRep: method to create CIImage objects from bitmap images and the NSCIImageRep class to create images you can use anywhere NSImage objects are supported.

Processing Video with AV Foundation

The AVFoundation framework provides a number of high level utilities for working with video and audio content. Among these is the AVVideoComposition class, which you can use to combine or edit video and audio tracks into a single presentation. (For general information on compositions, see Editing in AVFoundation Programming Guide.) You can use an AVVideoComposition object to apply Core Image filters to each frame of a video during playback or export, as shown in Listing 1-4.

Listing 1-4  Applying a filter to a video composition

let filter = CIFilter(name: "CIGaussianBlur")!

let composition = AVVideoComposition(asset: asset, applyingCIFiltersWithHandler: { request in

    // Clamp to avoid blurring transparent pixels at the image edges

    let source = request.sourceImage.clampingToExtent()

    filter.setValue(source, forKey: kCIInputImageKey)

    // Vary filter parameters based on video timing

    let seconds = CMTimeGetSeconds(request.compositionTime)

    filter.setValue(seconds * 10.0, forKey: kCIInputRadiusKey)

    // Crop the blurred output to the bounds of the original image

    let output = filter.outputImage!.cropping(to: request.sourceImage.extent)

    // Provide the filter output to the composition

    request.finish(with: output, context: nil)

})

When you create a composition with the videoCompositionWithAsset:applyingCIFiltersWithHandler: initializer, you supply a handler that’s responsible for applying filters to each frame of video. AVFoundation automatically calls your handler during playback or export. In the handler, you use the provided AVAsynchronousCIImageFilteringRequest object first to retrieve the video frame to be filtered (and supplementary information such as the frame time), then to provide the filtered image for use by the composition.

To use the created video composition for playback, create an AVPlayerItem object from the same asset used as the composition’s source, then assign the composition to the player item’s videoComposition property. To export the composition to a new movie file, create an AVAssetExportSession object from the same source asset, then assign the composition to the export session’s videoComposition property.

Processing Game Content with SpriteKit and SceneKit

SpriteKit is a technology for building 2D games and other types of apps that feature highly dynamic animated content; SceneKit is for working with 3D assets, rendering and animating 3D scenes, and building 3D games. (For more information on each technology, see SpriteKit Programming Guide and SceneKit Framework Reference.) Both frameworks provide high-performance real-time rendering, with easy ways to add Core Image processing to all or part of a scene.

In SpriteKit, you can add Core Image filters using the SKEffectNode class. To see an example of this class in use, create a new Xcode project using the Game template (for iOS or tvOS), select SpriteKit as the game technology, and modify the touchesBegan:withEvent: method in the GameScene class to use the code in Listing 1-5. (For the macOS Game template, you can make similar modifications to the mouseDown: method.)

Listing 1-5  Applying filters in SpriteKit

override func touchesBegan(touches: Set<UITouch>, withEvent event: UIEvent?) {

    for touch in touches {

        let sprite = SKSpriteNode(imageNamed:"Spaceship")

        sprite.setScale(0.5)

        sprite.position = touch.location(in: self)

        sprite.run(.repeatForever(.rotate(byAngle: 1, duration:1)))

        let effect = SKEffectNode()

        effect.addChild(sprite)

        effect.shouldEnableEffects = true

        effect.filter = CIFilter(name: "CIPixellate",

                                 withInputParameters: [kCIInputScaleKey: 20.0])

        self.addChild(effect)

    }

}

Note that the SKScene class itself is a subclass of SKEffectNode, so you can also apply a Core Image filter to an entire SpriteKit scene.

In SceneKit, the filters property of the SCNNode class can apply Core Image filters to any element of a 3D scene. To see this property in action, create a new Xcode project using the Game template (for iOS, tvOS, or macOS), select SceneKit as the game technology, and modify the viewDidLoad method in the GameViewController class to use the code in Listing 1-6.

Listing 1-6  Applying filters in SceneKit

// Find this line in the template code:

let ship = rootNode.childNode(withName: "ship", recursively: true)!

// Add these lines after it:

let pixellate = CIFilter(name: "CIPixellate",

                         withInputParameters: [kCIInputScaleKey: 20.0])!

ship.filters = [ pixellate ]

You can also animate filter parameters on a SceneKit node—for details, see the reference documentation for the filters property.

In both SpriteKit and SceneKit, you can use transitions to change a view’s scene with added visual flair. (See the presentScene:transition: method for SpriteKit and the presentScene:withTransition:incomingPointOfView:completionHandler: method for SceneKit.) Use the SKTransition class and its transitionWithCIFilter:duration: initializer to create a transition animation from any Core Image transition filter.

Processing Core Animation Layers (macOS)

In macOS, you can use the filters property to apply filters to the contents of any CALayer-backed view, and add animations that vary filter parameters over time. See Filters Add Visual Effects to OS X Views and Advanced Animation Tricks in Core Animation Programming Guide.

Rendering with an Automatic Context

If you don’t have constraints on how your app interoperates with other graphics technologies, creating a Core Image context is simple: just use the basic init or initWithOptions: initializer. When you do so, Core Image automatically manages resources internally, choosing the appropriate or best available CPU or GPU rendering technology based on the current device and any options you specify. This approach is well-suited to tasks such as rendering a processed image for output to a file (for example, with the writeJPEGRepresentationOfImage:toURL:colorSpace:options:error: method).

Take care when using this approach if you intend to render Core Image results in real time—that is, to animate changes in filter parameters, to produce an animated transition effect, or to process video or other visual content that already renders many times per second. Even though a CIContext object created with this approach can automatically render using the GPU, presenting the rendered results may involve expensive copy operations between CPU and GPU memory.

Real-Time Rendering with Metal

The Metal framework provides low-overhead access to the GPU, enabling high performance for graphics rendering and parallel compute workflows. Such workflows are integral to image processing, so Core Image builds upon Metal wherever possible. If you’re building an app that renders graphics with Metal, or if you want to leverage Metal to get real-time performance for animating filter output or filtering animated input (such as live video), use a Metal device to create your Core Image context.

Listing 1-7 and Listing 1-8 show an example of using a MetalKit view (MTKView) to render Core Image output. (Important steps are numbered in each listing and described afterward.)

Listing 1-7  Setting up a Metal view for Core Image rendering

class ViewController: UIViewController, MTKViewDelegate {  // 1

    // Metal resources

    var device: MTLDevice!

    var commandQueue: MTLCommandQueue!

    var sourceTexture: MTLTexture!                         // 2

    // Core Image resources

    var context: CIContext!

    let filter = CIFilter(name: "CIGaussianBlur")!

    let colorSpace = CGColorSpaceCreateDeviceRGB()

    override func viewDidLoad() {

        super.viewDidLoad()

        device = MTLCreateSystemDefaultDevice()            // 3

        commandQueue = device.newCommandQueue()

        let view = self.view as! MTKView                   // 4

        view.delegate = self

        view.device = device

        view.framebufferOnly = false

        context = CIContext(mtlDevice: device)             // 5

        // other setup

    }

}

1. This example uses a UIViewController subclass for iOS or tvOS. To use with macOS, subclass NSViewController instead.

2. The sourceTexture property holds a Metal texture containing the image to be processed by the filter. This example doesn’t show loading the texture’s content because there are many ways to fill a texture—for example, you could load an image file using the MTKTextureLoader class, or use the texture as output from an earlier rendering pass of your own.

3. Create the Metal objects needed for rendering—a MTLDevice object representing the GPU to use, and a command queue to execute render and compute commands on that GPU. (This command queue can handle both the render or compute commands encoded by Core Image and those from any additional rendering passes of your own.)

4. Configure the MetalKit view.

5. Create a Core Image context that uses the same Metal device as the view. By sharing Metal resources, Core Image can process texture contents and render to the view without the performance costs of copying image data to and from separate CPU or GPU memory buffers. CIContext objects are expensive to create, so you do so only once and reuse it each time you process images.

MetalKit calls the drawInMTKView: method each time the view needs displaying. (By default, MetalKit may call this method as many as 60 times per second. For details, see the view’s preferredFramesPerSecond property.) Listing 1-8 shows a basic implementation of that method for rendering from a Core Image context.

Listing 1-8  Drawing with Core Image filters in a Metal view

public func draw(in view: MTKView) {

    if let currentDrawable = view.currentDrawable {              // 1

        let commandBuffer = commandQueue.commandBuffer()

        let inputImage = CIImage(mtlTexture: sourceTexture)!     // 2

        filter.setValue(inputImage, forKey: kCIInputImageKey)

        filter.setValue(20.0, forKey: kCIInputRadiusKey)

        context.render(filter.outputImage!,                      // 3

            to: currentDrawable.texture,

            commandBuffer: commandBuffer,

            bounds: inputImage.extent,

            colorSpace: colorSpace)

        commandBuffer.present(currentDrawable)                   // 4

        commandBuffer.commit()

    }

}

1. Obtain a Metal drawable texture to render into and a command buffer to encode rendering commands.

2. Configure the filter’s input parameters, including the input image sourced from a Metal texture. This example uses constant parameters, but remember that this method runs up to 60 times per second—you can use this opportunity to vary filter parameters over time to create smooth animations.

3. Tell the Core Image context to render the filter output into the view’s drawable texture. The bounds parameter tells Core Image what portion of the image to draw—this example uses the input image’s dimensions.

4. Tell Metal to display the rendered image when the command buffer finishes executing.

This example shows only the minimal code needed to render with Core Image using Metal. In a real application, you’d likely perform additional rendering passes before or after the one managed by Core Image, or render Core Image output into a secondary texture and use that texture in another rendering pass. For more information on drawing with Metal, see Metal Programming Guide.

Real-Time Rendering with OpenGL or OpenGL ES

Core Image can also use OpenGL (macOS) or OpenGL ES (iOS and tvOS) for high-performance, GPU-based rendering. Use this option if you need to support older hardware where Metal is not available, or if you want to integrate Core Image into an existing OpenGL or OpenGL ES workflow.

• If you draw using OpenGL ES (in iOS or tvOS), use the contextWithEAGLContext:options: initializer to create a Core Image context from the EAGLContext you use for rendering.

• If you draw using OpenGL (in macOS), use the contextWithCGLContext:pixelFormat:colorSpace:options: initializer create a Core Image context from the OpenGL context you use for rendering. (See the reference documentation for that method for important details on pixel formats.)

In either scenario, use the imageWithTexture:size:flipped:colorSpace: initializer to create CIImage objects from OpenGL or OpenGL ES textures. Working with image data that’s already in GPU memory improves performance by removing redundant copy operations.

To render Core Image output in OpenGL or OpenGL ES, make your GL context current and set a destination framebuffer, then call the drawImage:inRect:fromRect: method.

CPU-Based Rendering with Quartz 2D

If your app doesn’t require real-time performance and draws view content using CoreGraphics (for example, in the drawRect: method of a UIKit or AppKit view), use the contextWithCGContext:options: initializer to create a Core Image context that works directly with the Core Graphics context you’re already using for other drawing. (In macOS, use the CIContext property of the current NSGraphicsContext object instead.) For information on CoreGraphics contexts, see Quartz 2D Programming Guide.