WWDC 2021: Optimize for variable refresh rate displays

Adaptive-Sync

Most Apple displays have fixed refresh rates
Variable refresh-rate displays are iPad Pro w/ ProMotion and Macs w/ external Adaptive-Sync displays

With fixed-rate display, the refresh rate is constant, and each frame is always presented for same amount of time (e.g. 1/60ms per-frame lifetime on 60Hz displays)
Also, at fixed-rate always need to display a frame, even if its just repeating the previous frame

With Adaptive-Sync, frames can be displayed for different amounts (e.g. for 40Hz-120Hz Adaptive-Sync display, each frame can be displayed between 8ms-25ms) However, if for slowest frame (40 Hz = 25ms) no new frame, then macOS will force refresh from existing frame and will be unavailable for short amount of time (8ms)

Free benefit from Adaptive-Sync: if on fixed-rate 120Hz display you miss a frame, then you have to wait for 8ms before you can display a new frame (since only refreshes at 120Hz = 8ms), so noticeable hitch
On adaptive-sync, you can show the late frame right away after 1ms

Previously, used to advise that if you could deliver frames at slower rate than refresh rate to sometimes slow down to keep in sync with multiple of refresh rate
Now you can deliver your frames at whatever rate within the supported display range (e.g. 40Hz-120Hz)

Supported Macs: all Apple Silicon Macs and recent Intel Macs

App must be running in full-screen mode on supported display to do Adaptive-Sync

Metal call: [commandBuffer presentDrawable:drawable atTime:t]
Can use this to let users adjust the FPS for your game

ProMotion on iPad Pro

All iPad Pros since 2017 shipped with 120Hz displays - but system may lower refresh rate, e.g. in low power mode

Fixed 60Hz display will be smooth as long as frame intervals is multiple of 60 (e.g. 30Hz, 20Hz), but the display will still need to refresh/repeat the previous frame at 60Hz - so even though the display is smooth/coherent at 30Hz, you use same amount of power as 60Hz

ProMotion displays at 120Hz are compatible with any frame rate compatible with 60Hz (since 60 is multiple of 120)

Main advantage of ProMotion is it can adjust the refresh rate dynamically - so no need to waste power repeating previous frame like fixed-rate

When is 120Hz refresh rate not available?
Accessibility settings - Limit frame rate
Thermal state
Low power mode (new in iPadOS 15)

Display link - timer synchronized with display refresh rate for your app to drive custom animations and render loops
CVDisplayLink - Core Video (macOS)
CADisplayLink - Core Animation (iOS, macOS Catalyst)

Focus of talk is on CADisplayLink, but similar conceptually to CVDisplayLink

CADisplayLink gives an application the maximum amount of time to complete its work Regular NSTimer is very likely to be out of phase with the display refresh rate, so your app will get reduced time slice to produce next frame

CADisplayLink.preferredFramesPerSecond - can adjust display link rate to lower than that of display refresh rate
CADisplayLink will also automatically adjust its timer rate to the underlying display refresh rate

4 best practices for driving your own drawing/render loop:

1. Query display refresh rate at runtime instead of hardcoding it

   Maximum frame rate: [[UIScreen mainScreen] maximumFramesPerSecond] (e.g. will always return 120 on ProMotion displays)

   Current frame rate: round(1 / displayLink.duration)

   You almost always should use the current frame rate since the system may not run at the maximum frame rate (e.g. low power mode)

2. Use actual frame rate of CADisplayLink

   CADisplayLink will adjust itself in response to hardware changes

   Example: requesting 40 fps on 120Hz display gives each frame 24ms
   However, if display then changes to 60Hz, the display link will adjust to 30Hz (since 40 isn't multiple of 60)

3. Use targetTimestamp to gracefully handle changes in display refresh rates

   timestamp - when callback is scheduled to be invoked targetTimestamp - when next frame will be composited by Core Animation

   Main problem with timestamp is that at the transition point from 120Hz to 60Hz, when interpolating each frame in an animation, at the transition point you will calculate by 8ms of progress but present frame in 16ms

   

   targetTimestamp gives you the timestamp of the next frame, which is what you really care about when preparing the next frame

   In general, want to use targetTimestamp instead of timestamp

4. Handle missed display link callbacks

   (targetTimestamp - timestamp) is theoretical expected amount of time between display link callbacks, but not guaranteed (e.g. higher priority thread scheduled on CPU)

   Display callbacks can also be skipped if run loop is busy

   Instead of assuming you always have the full 8ms, use CACurrentMediaTime() to compute the actual amount of time you have

   You also have to handle the case where the display link callback is skipped because your run loop was busy Need to keep track of previousTargetTimestamp