Injecting Apple Pencil Pressure into macOS via CGEvent

by omcapple-pencilcgeventinput-injectioniosmacostablet

Injecting Apple Pencil Pressure into macOS via CGEvent

One of Astropad's killer features is Apple Pencil support with full pressure and tilt sensitivity. When you draw on your iPad, the Mac app receiving the input sees a pressure-sensitive tablet event — as if you had a Wacom tablet connected.

This post covers how to capture Apple Pencil input on iOS, transmit it over the network, and inject it into macOS as a native tablet event using CGEvent.

The Pipeline

iPad (Apple Pencil)
    → UITouch events (pressure, altitude, azimuth)
    → Serialize as InputEvent
    → Send over QUIC reliable stream
    → macOS host receives InputEvent
    → Create CGEvent with tablet properties
    → Post to system event tap
    → Photoshop/Procreate sees a tablet event

iOS: Capturing Apple Pencil Input

class PencilInputView: UIView {
    var onPencilInput: ((PencilEvent) -> Void)?
    
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        handleTouches(touches, phase: .began)
    }
    
    override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
        handleTouches(touches, phase: .moved)
    }
    
    override func touchesEnded(_ touches: Set<UITouch>, with event: UIEvent?) {
        handleTouches(touches, phase: .ended)
    }
    
    private func handleTouches(_ touches: Set<UITouch>, phase: TouchPhase) {
        guard let touch = touches.first else { return }
        
        // Check if this is an Apple Pencil touch
        guard touch.type == .pencil else { return }
        
        let location = touch.location(in: self)
        
        // Normalize coordinates to 0.0-1.0
        let normalizedX = Float(location.x / bounds.width)
        let normalizedY = Float(location.y / bounds.height)
        
        // Pressure: 0.0 (no pressure) to 1.0 (max pressure)
        let pressure = Float(touch.force / touch.maximumPossibleForce)
        
        // Tilt: altitude angle (0 = parallel to screen, π/2 = perpendicular)
        let altitudeAngle = Float(touch.altitudeAngle)
        
        // Azimuth: rotation angle around the perpendicular axis
        let azimuthAngle = Float(touch.azimuthAngle(in: self))
        
        let event = PencilEvent(
            x: normalizedX,
            y: normalizedY,
            pressure: pressure,
            altitudeAngle: altitudeAngle,
            azimuthAngle: azimuthAngle,
            phase: phase,
            timestamp: touch.timestamp
        )
        
        onPencilInput?(event)
    }
}

Apple Pencil Pro: Roll Angle

On Apple Pencil Pro (iPad Pro M4+), you also get barrel roll:

if #available(iOS 17.5, *) {
    let rollAngle = Float(touch.rollAngle)  // Rotation around pencil's long axis
    event.rollAngle = rollAngle
}

High-Frequency Coalesced Touches

Apple Pencil samples at 240Hz, but touchesMoved fires at screen refresh rate (120Hz). To get the full 240Hz data:

override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
    guard let touch = touches.first else { return }
    
    // Get all coalesced touches since last callback (up to 240Hz)
    let coalescedTouches = event?.coalescedTouches(for: touch) ?? [touch]
    
    for coalescedTouch in coalescedTouches {
        handleTouch(coalescedTouch)
    }
}

Wire Format

struct PencilEvent: Codable {
    let x: Float          // 0.0-1.0 normalized
    let y: Float          // 0.0-1.0 normalized
    let pressure: Float   // 0.0-1.0
    let altitudeAngle: Float  // radians
    let azimuthAngle: Float   // radians
    let rollAngle: Float?     // radians (Pencil Pro only)
    let phase: UInt8      // 0=began, 1=moved, 2=ended
    let timestamp: Double
}

Serialize with a compact binary format (not JSON) for minimal latency:

[4B] x | [4B] y | [4B] pressure | [4B] altitude | [4B] azimuth | [1B] phase = 21 bytes per event

At 240Hz, that is 5KB/s — negligible bandwidth. Send over the QUIC reliable stream.

macOS: Injecting Tablet Events via CGEvent

This is the key part. macOS has a hidden tablet event system in CGEvent that most developers don't know about.

Mouse Events with Tablet Pressure

func injectPencilEvent(_ event: PencilEvent, displayWidth: Int, displayHeight: Int) {
    // Map normalized coordinates to display pixels
    let x = CGFloat(event.x) * CGFloat(displayWidth)
    let y = CGFloat(event.y) * CGFloat(displayHeight)
    let point = CGPoint(x: x, y: y)
    
    // Determine event type based on phase
    let eventType: CGEventType
    switch event.phase {
    case 0: eventType = .leftMouseDown
    case 1: eventType = .leftMouseDragged
    case 2: eventType = .leftMouseUp
    default: eventType = .mouseMoved
    }
    
    // Create the mouse event
    guard let cgEvent = CGEvent(
        mouseEventSource: nil,
        mouseType: eventType,
        mouseCursorPosition: point,
        mouseButton: .left
    ) else { return }
    
    // SET TABLET SUBTYPE — This is the magic
    cgEvent.setIntegerValueField(
        .mouseEventSubtype,
        value: Int64(CGEventMouseSubtype.tabletPoint.rawValue)
    )
    
    // SET PRESSURE (0.0 to 1.0)
    cgEvent.setDoubleValueField(
        .tabletEventPointPressure,
        value: Double(event.pressure)
    )
    
    // SET TILT
    // Tablet tilt X: derived from azimuth + altitude
    let tiltX = cos(Double(event.azimuthAngle)) * cos(Double(event.altitudeAngle))
    let tiltY = sin(Double(event.azimuthAngle)) * cos(Double(event.altitudeAngle))
    cgEvent.setDoubleValueField(.tabletEventTiltX, value: tiltX)
    cgEvent.setDoubleValueField(.tabletEventTiltY, value: tiltY)
    
    // SET TABLET BUTTON (pen tip = button 1)
    if event.phase == 0 || event.phase == 1 {
        cgEvent.setIntegerValueField(.tabletEventPointButtons, value: 1)
    }
    
    // Post the event
    cgEvent.post(tap: .cghidEventTap)
}

The Key CGEvent Fields

| Field | Type | Range | Purpose | |-------|------|-------|---------| | .mouseEventSubtype | Int64 | .tabletPoint | Tells apps this is a tablet event | | .tabletEventPointPressure | Double | 0.0 - 1.0 | Pen pressure | | .tabletEventTiltX | Double | -1.0 - 1.0 | Horizontal tilt | | .tabletEventTiltY | Double | -1.0 - 1.0 | Vertical tilt | | .tabletEventPointButtons | Int64 | bitmask | Which pen buttons are pressed | | .tabletEventRotation | Double | 0.0 - 360.0 | Barrel rotation | | .tabletEventTangentialPressure | Double | -1.0 - 1.0 | Tangential pressure (airbrush) |

Proximity Events

When the pen approaches or leaves the tablet surface:

func injectProximityEvent(entering: Bool) {
    guard let event = CGEvent(source: nil) else { return }
    event.type = entering ? .tabletProximity : .tabletProximity
    event.setIntegerValueField(.tabletProximityEventEnterProximity, value: entering ? 1 : 0)
    event.setIntegerValueField(.tabletProximityEventPointerType, value: 1) // Pen
    event.post(tap: .cghidEventTap)
}

Required Permissions

CGEvent posting requires Accessibility permission:

func checkAccessibilityPermission() -> Bool {
    let trusted = AXIsProcessTrusted()
    if !trusted {
        // Prompt user to grant permission
        let options = [kAXTrustedCheckOptionPrompt.takeUnretainedValue(): true] as CFDictionary
        AXIsProcessTrustedWithOptions(options)
    }
    return trusted
}

App Store Considerations

Important: If you plan to distribute via the Mac App Store, CGEvent posting with .cghidEventTap can trigger App Store rejection under guideline 2.4.5. Reddit user reports confirm this:

"Got rejected twice for my keyboard layout switcher app. I use CGEventTap (.listenOnly) to monitor keystrokes and CGEvent.post() to inject the corrected text."

For App Store distribution, you may need to use Accessibility APIs (AXUIElement) instead. For direct distribution (DMG, Homebrew), CGEvent works fine.

What Astropad Does

From our binary analysis:

  • InputOSInjector.swift — The Swift class that handles CGEvent injection
  • SystemKeyEventTap.swift — Uses CGEvent.tapCreate for global key interception
  • liquid_input::os::mac::cursor_monitor — Rust module tracking cursor position
  • multiplayer_input_protocol — Wire format with current_cursor and cursor_info messages
  • The app requires Accessibility permission during onboarding (AccessibilityPermission.swift, PermissionHelper.swift)

Part 9 of the "Building a Remote Desktop from Scratch" series. Based on reverse engineering analysis of Astropad Workbench 1.1.0.

← All notes