What’s new with text and text interactions

♪ ♪ James: Hello. I'm James Magahern from UIKit, and I'll be your guide through what's new with text and text interactions. Text plays a crucial role in every application. It's the primary way to consume information and communicate.

Now we're giving you even more tools to create a powerful text experience in your app, whether you're starting from scratch or from high-level abstractions. I'll be going over a bunch of great improvements to crafting text experiences and how to take advantage of them in your apps, including some changes to system selection UI, adding text item actions and menus to UITextViews, lists and bullets in TextKit 2, dictation UI on macOS, and some important updates for internationalization. We'll begin by talking about some changes to selection UI. New on all our platforms is a completely redesigned text cursor.

We now display an inline, interactive switcher when changing input languages, more ergonomic selection handles when doing range selection, and we also have a completely new loupe to make it easier to place the cursor in large bodies of text. If your app is using UITextViews or UITextFields, then you'll get all this new UI automatically. Additionally, we made sure that clients of UITextInteraction also get the new selection UI. However, if you have a highly customized UI for displaying text, it can be challenging to keep up with these changes in your own implementation, especially if you aren't able to adopt UITextInteraction. That's why in iOS 17, we've added UITextSelectionDisplayInteraction, which just provides the selection UI without the gestures included with UITextInteraction.

UITextSelectionDisplayInteraction is a new type of UIInteraction that can be installed on any UIView. You must also provide an object that implements the UITextInput protocol, which defines various methods for the interaction to obtain selection state from your view. UITextSelectionDisplayInteraction will then do all the heavy lifting for you, providing the cursor view and cursor accessories, as well as the range highlight and selection handles. All of these views are replaceable too, so you can customize the behavior if you need to. Here's a quick look at an example of how to do this in code. First, create a selection display interaction and provide it with your document object that implements the UITextInput protocol. This can be the same as your view, if applicable. Next, simply add the interaction to the view you want to display the selection UI inside of. This can be a container view or the view that renders your document. Whenever the selection state changes, just make sure to call setNeedsSelectionUpdate on the interaction, and the interaction will take care of updating all the views to reflect your new selection state. In addition to UITextSelectionDisplayInteraction, we've also added a new API for displaying a loupe. It can be used on any view, and UITextSelectionDisplayInteraction or UITextInput isn't required to use it. It's recommended to use a gesture recognizer, such as UIPanGestureRecognizer, to drive the loupe's updates. Here's how to manage a loupe session in code. When using a pan gesture recognizer, in the callback, grab the gesture's location and selection widget, in this case, the cursor view. Next, if the gesture is beginning, create a new UITextLoupeSession by calling begin(at:), providing the loupe's starting location, selection widget, and coordinate space. Then when the gesture moves around, call move(to:) on the existing loupe session to update its location. Lastly, when the gesture recognizer ends, make sure to call invalidate on the loupe session to dismiss it. Next I'd like to go over text item actions and how you can use them to display menus inside of text views. Text item interactions in UITextView are now much more customizable using new APIs on UITextViewDelegate.

These new additions make it possible to modify the primary action for text items or present a menu like the candidate menu shown here in the Translate app. Previously, UITextView allows developers to disable item interactions through the shouldInteractWith APIs on UITextViewDelegate for links and attachments. In iOS 17, you now have the ability to customize item interactions for text items in text views. UITextViewDelegate has gained new methods to customize both the primary action, or provide a menu for specific text items. Here's a quick review of what text items actually are and how they integrate with your app.

Text items represent the content that support item interactions. This includes text attachments, represented by NSTextAttachment, and links, represented by NSLinkAttributeName. UITextItem now also supports tagging custom ranges of text for interaction. To tag a range for interaction, use the UITextItemTagAttributeName on the attributed string representing the range of text you wish to make interactive. With these new APIs, it is now possible to change the default tap or menu behavior when a text item is interacted with. For instance, you might want to redirect known links to existing views within the app or add custom menus to parts of the text. To continue to suppress or disable the primary action or menu, simply return nil for the menuConfiguration or primaryAction delegate methods.

Here's a small code example demonstrating how to show a menu in response to the user tapping on a link inside of your UITextView. Just implement the menuConfigurationFor:defaultMenu: method from UITextViewDelegate and return a menu configuration containing the menu you wish to display. You may also provide a preview as part of the menu configuration to show inline with the rest of the context menu.

List and bullet support have also been added to TextKit 2. I'll go over how you can take advantage of this in your application. In addition to supporting multiple different kinds of bulleted items, roman numeral, alphabetical, and decimal ordering is also supported. Each one will automatically be localized based on the device or app's configured locale, so worldwide internationalization is built right in. Here's a quick example of how you can get started with text lists using attributed strings. Use the textLists property on NSParagraphStyle to define which paragraph has an ordered list applied to it. The system will automatically number the items based on line terminating characters like newlines, and UITextView will automatically propagate the paragraph style based on typing attributes. There are some big changes in the way dictation works on macOS Sonoma. We have some new APIs to help you make sure your custom text elements work well with the new design. This is how the new dictation indicator behaves on macOS 14. Like on iOS, dictation has a trailing glow effect while speaking, and a microphone indicator appears when at rest.

Notice that after scrolling, the indicator next to the cursor will stick to the edge of the scroll view, and offer a button to return to the current position in the document. If you're using a standard text control in AppKit like NSTextView, you'll get this behavior right out of the box. If your app offers a customized implementation for showing the insertion point, there's a little bit of work you'll need to do to support this new behavior.

Similar to UITextSelectionDisplayInteraction for iOS, there's now a new API for macOS that also allows you to remain consistent with the system's selection UI. Instead of drawing your own cursor, adopt NSTextInsertionIndicator, which is a customizable NSView subclass that you can place inside of your document's view to indicate the cursor position. Its appearance is customizable. You can change the color and size. By default, it will follow the current accent color. It makes supporting the dictation effects really easy. There's just a little bit of configuration required to allow the system to insert the appropriate views into your hierarchy. Here's how to get started using NSTextInsertionIndicator on macOS. Simply create an NSTextInsertionIndicator view and add it as a subview to whatever view is displaying the contents of your document. Once you've done that, set effectsViewInserter to a block that inserts views provided by your system into your document's view hierarchy. The system will automatically take care of positioning the effect views and making sure they get updated as the insertion indicator moves around. When your custom text view resigns responder status, make sure to hide the cursor by setting displayMode to hidden.

As text is inserted, updates to the frame will automatically be animated with a glow effect if dictation is used as the input source. If you need to disable the glow effect, remove the showEffectsView option from automaticModeOptions. When the user changes input modes, an effect view is displayed beneath the cursor in horizontal text to show the language selection UI. If you need to override the placement of this UI, you can specify preferredTextAccessoryPlacement in your implementation of NSTextInputClient. The system checks the value of this property to determine where to draw the accessories. When the text cursor is scrolled offscreen during dictation, the system will also display a scroll-away indicator to indicate the relative position of the cursor and offer an affordance to get back to the dictation point.

To make sure this behavior is supported in your custom text view, make sure to adopt NSTextInputClient and implement the selectionRect and documentVisibleRect properties. Notify the system about scrolling beginning and ending by calling textInputClientWillStartScrollingOrZooming and willEndScrollingOrZooming. Making sure your app works well in every language is crucial to providing an outstanding text experience. We've made some important changes to standard text controls, enhancing their readability and ergonomics across a broader range of languages. Supporting dynamic type in your apps goes a long way towards improving the layout of your UI in every language. Something important to consider when laying out text elements in your app is the fact that many languages can have variable line heights in addition to layout direction and font styles. A common issue encountered when dealing with variable line heights between languages is the problem of clipped text. Not only does this look bad, but it can also drastically impact readability and significantly hinder the overall user experience of your app. These issues may be difficult to notice, especially if you aren't able to test your app in every language. We've made some improvements to standard text elements, like UITextField and UILabel, to address this problem automatically in most cases. However, there are some best practices that you should still make sure to follow to ensure the system can prevent clipping of these elements. To prevent these common clipping issues, all our platforms will now automatically adjust the line height in UILabel or UITextField to accommodate languages with highly dynamic line heights. This new behavior takes place only in specific cases. First, keep in mind that line heights are now dependent on the device's configured preferred languages. Second, this adjustment affects the entire user interface, increasing line heights for all text elements, even those that display scripts with fixed line heights, like English, which wouldn't normally require the extra space. This is intentional to ensure visual consistency across all text elements. Lastly, this automatic adjustment only applies to text elements that use text styles. Custom fonts will continue to use their fixed line heights. This behavior isn't entirely new, as it has been in effect for several releases. However, in iOS 17 it's much more dynamic, and the exact adjustment depends on both the text style and language used.

To take advantage of this behavior, adopt Text Styles by creating a UIFont explicitly with the preferredFont(forTextStyle:) method and assign it to a text element, like UILabel. Next, avoid setting clipsToBounds on text elements. Ascenders and descenders for languages like Thai and Hindi will often protrude beyond the line height bounds. This is generally acceptable in most layouts, since there's usually extra space around neighboring elements, but it does mean that setting clipsToBounds will result in those text elements being clipped. UIKit has also been updated to prevent unnecessary use of this setting where it was previously enabled by default. Lastly, ensure that your UIs are responsive to changes in height and that all other controls remain aligned, as text elements can now grow vertically. We also made substantial enhancements to line-breaking behavior for languages such as Chinese, German, Japanese, and Korean. These improvements apply different rules based on what kind of text style you're using in your UI and which language is being displayed. For instance, previously, a word within Korean text might have been split between lines. Now UIKit prevents such splitting for title text styles, ensuring a more seamless reading experience. To benefit from this new behavior in your app, similarly adopt Text Styles. So that's everything you need to know to create a powerful text experience in your app on iOS 17.

If you're using a custom text view in your app, make sure to use the system selection UI to take advantage of all the new features in the OS. Use text items to create better interactions with links in your text views. Leverage the powerful new features of TextKit 2 to provide a more rich text editing experience. And adopt Text Styles in common text elements to ensure a great user experience for everyone, regardless of what language or preferred text size. That's all for now. Thanks for watching. Be sure to rate five stars and leave a comment down below. ♪ ♪