QUIC Transport with Network.framework: Beyond TCP and UDP

by omcmacosnetworkingquicremote-desktopswift

QUIC Transport with Network.framework: Beyond TCP and UDP

QUIC is the transport protocol that makes modern remote desktop streaming possible. It gives you the reliability of TCP (for control messages) and the speed of UDP (for video frames) in a single multiplexed connection with built-in TLS 1.3 encryption.

Apple's Network.framework provides native QUIC support since macOS 14 and iOS 17. Astropad Workbench uses QUIC as its primary transport (via the Rust quinn-proto library), but for a Swift-first approach, Network.framework is the way to go.

Why QUIC for Remote Desktop?

| Feature | TCP | UDP | QUIC | |---------|-----|-----|------| | Reliability | Yes | No | Configurable per-stream | | Multiplexing | No (head-of-line blocking) | Manual | Built-in streams | | Encryption | Separate TLS | Manual | Built-in TLS 1.3 | | Connection migration | No | No | Yes (WiFi↔cellular) | | 0-RTT reconnection | No | N/A | Yes | | Datagrams (unreliable) | No | Yes | Yes (QUIC datagrams) |

For remote desktop, you need:

  • Reliable streams for control (codec params, session config, clipboard sync)
  • Reliable streams for input events (mouse, keyboard — can't lose these)
  • Unreliable datagrams for video frames (losing a frame is better than waiting for retransmission)

QUIC gives you all three over a single connection.

Server Setup (Host Mac)

import Network

class StreamServer {
    private var listener: NWListener?
    private var sessions: [StreamSession] = []
    
    func start(port: UInt16 = 9876) throws {
        // Create QUIC parameters with TLS
        let quicOptions = NWProtocolQUIC.Options()
        quicOptions.alpn = ["liquid-v1"]  // Application-Layer Protocol Negotiation
        
        // For development: use a pre-shared key or self-signed cert
        // For production: use a proper TLS certificate
        sec_protocol_options_add_pre_shared_key(
            quicOptions.securityProtocolOptions,
            pskData as __DispatchData,
            pskIdentity as __DispatchData
        )
        
        let parameters = NWParameters(quic: quicOptions)
        
        listener = try NWListener(using: parameters, on: NWEndpoint.Port(rawValue: port)!)
        
        listener?.newConnectionHandler = { [weak self] connection in
            self?.handleNewConnection(connection)
        }
        
        listener?.stateUpdateHandler = { state in
            switch state {
            case .ready:
                print("Server listening on port \(port)")
            case .failed(let error):
                print("Server failed: \(error)")
            default:
                break
            }
        }
        
        listener?.start(queue: .main)
    }
    
    private func handleNewConnection(_ connection: NWConnection) {
        let session = StreamSession(connection: connection)
        sessions.append(session)
        session.start()
    }
}

Client Setup (Viewer)

class StreamClient {
    private var connection: NWConnection?
    
    func connect(host: String, port: UInt16 = 9876) {
        let quicOptions = NWProtocolQUIC.Options()
        quicOptions.alpn = ["liquid-v1"]
        
        // Match server's PSK or TLS config
        sec_protocol_options_add_pre_shared_key(
            quicOptions.securityProtocolOptions,
            pskData as __DispatchData,
            pskIdentity as __DispatchData
        )
        
        let parameters = NWParameters(quic: quicOptions)
        
        connection = NWConnection(
            host: NWEndpoint.Host(host),
            port: NWEndpoint.Port(rawValue: port)!,
            using: parameters
        )
        
        connection?.stateUpdateHandler = { state in
            switch state {
            case .ready:
                print("Connected to host")
                self.openStreams()
            case .failed(let error):
                print("Connection failed: \(error)")
            default:
                break
            }
        }
        
        connection?.start(queue: .main)
    }
}

Channel Design

We multiplex multiple logical channels over a single QUIC connection:

Control Stream (Bidirectional, Reliable)

Used for session negotiation and ongoing control:

  • Codec parameters (SPS/PPS/VPS for decoder init)
  • Display info (resolution, refresh rate)
  • Keepalive pings (latency measurement)
  • Session teardown

Input Stream (Client→Host, Reliable)

Carries user input events:

  • Mouse move, click, scroll
  • Keyboard press/release
  • Apple Pencil pressure/tilt
  • Clipboard data

Video Datagrams (Host→Client, Unreliable)

Carries encoded video frames:

  • Each datagram is one NAL unit (or fragment thereof)
  • Lost frames are simply skipped — the next keyframe recovers
  • No retransmission delay = lowest latency

Opening QUIC Streams

// Open the control stream (bidirectional)
func openControlStream() {
    let controlStream = connection?.createStream(direction: .bidirectional)
    controlStream?.stateUpdateHandler = { state in
        if case .ready = state {
            self.sendSessionConfig(on: controlStream!)
        }
    }
    controlStream?.start(queue: .main)
}

Sending Video via Datagrams

// Send encoded frame as QUIC datagram
func sendVideoFrame(data: Data, frameId: UInt32, isKeyframe: Bool, timestamp: UInt32) {
    // Build frame header
    var header = Data(capacity: 11)
    header.append(contentsOf: withUnsafeBytes(of: frameId.bigEndian) { Array($0) })
    header.append(isKeyframe ? 0x01 : 0x00)  // flags
    header.append(contentsOf: [0x00, 0x01])   // fragment 0 of 1
    header.append(contentsOf: withUnsafeBytes(of: timestamp.bigEndian) { Array($0) })
    
    let packet = header + data
    
    // Fragment if larger than max datagram size (~1200 bytes)
    if packet.count > 1200 {
        sendFragmented(data: data, frameId: frameId, isKeyframe: isKeyframe, timestamp: timestamp)
    } else {
        connection?.send(content: packet, contentContext: .datagram,
                        isComplete: true, completion: .idempotent)
    }
}

func sendFragmented(data: Data, frameId: UInt32, isKeyframe: Bool, timestamp: UInt32) {
    let maxPayload = 1200 - 11 // header size
    let totalFragments = (data.count + maxPayload - 1) / maxPayload
    
    for i in 0..<totalFragments {
        let start = i * maxPayload
        let end = min(start + maxPayload, data.count)
        let fragment = data[start..<end]
        
        var header = Data(capacity: 11)
        header.append(contentsOf: withUnsafeBytes(of: frameId.bigEndian) { Array($0) })
        header.append(isKeyframe ? 0x01 : 0x00)
        header.append(UInt8(i))                    // fragment index
        header.append(UInt8(totalFragments))        // total fragments
        header.append(contentsOf: withUnsafeBytes(of: timestamp.bigEndian) { Array($0) })
        
        let packet = header + fragment
        connection?.send(content: packet, contentContext: .datagram,
                        isComplete: true, completion: .idempotent)
    }
}

Receiving Datagrams

func receiveDatagrams() {
    connection?.receiveMessage { [weak self] data, context, isComplete, error in
        if let data = data, context?.protocolMetadata(definition: .quic) != nil {
            self?.handleReceivedDatagram(data)
        }
        // Continue receiving
        self?.receiveDatagrams()
    }
}

func handleReceivedDatagram(_ data: Data) {
    guard data.count >= 11 else { return }
    
    let frameId = data[0..<4].withUnsafeBytes { $0.load(as: UInt32.self).bigEndian }
    let flags = data[4]
    let fragIndex = data[5]
    let totalFrags = data[6]
    let timestamp = data[7..<11].withUnsafeBytes { $0.load(as: UInt32.self).bigEndian }
    let payload = data[11...]
    
    let isKeyframe = (flags & 0x01) != 0
    
    if totalFrags == 1 {
        // Complete frame
        decoder.decode(Data(payload), isKeyframe: isKeyframe, timestamp: timestamp)
    } else {
        // Fragment — reassemble
        reassembler.addFragment(frameId: frameId, index: Int(fragIndex),
                               total: Int(totalFrags), data: Data(payload),
                               isKeyframe: isKeyframe, timestamp: timestamp)
    }
}

What Astropad Does

From binary analysis, Astropad's networking is more sophisticated:

  • liquid_net::runtime::datagram — Custom datagram handling with packet_reader, packet_sender, transmitter, receiver, and pulse (keepalive)
  • liquid_net::router — Message routing across channels: Video, StateSyncEvent, ClipboardSync, MultiplayerInputCoordination, Diagnostics, and more
  • liquid_net::sockets::encrypted_udp — Additional encryption layer beyond QUIC's TLS
  • liquid_net::throttle — Bandwidth throttling
  • liquid_net::runtime::network_quality — Real-time quality monitoring

They use quinn-proto 0.11.9 (Rust QUIC) rather than Network.framework, likely for cross-platform consistency and finer control over congestion algorithms.

Latency Measurement

Use the control stream for ping/pong latency measurement:

func measureLatency() {
    let sendTime = mach_absolute_time()
    sendPing(on: controlStream)
    
    // In pong handler:
    let receiveTime = mach_absolute_time()
    let rtt = machTimeToMilliseconds(receiveTime - sendTime)
    let oneWayLatency = rtt / 2.0
}

Connection Migration

One of QUIC's killer features: if the client switches from WiFi to cellular, the connection survives. Network.framework handles this automatically — the NWConnection maintains its state across network transitions.

Part 4 of the "Building a Remote Desktop from Scratch" series. Based on reverse engineering analysis of Astropad Workbench 1.1.0.

← All notes