Reverse Engineering Astropad Workbench: How LIQUID Protocol Works Under the Hood

by omcastropadmacosremote-desktopreverse-engineeringrustswift

Astropad Workbench - Reverse Engineering Research

Table of Contents

  1. Product Overview
  2. Binary Analysis
  3. Internal Architecture
  4. LIQUID Protocol Deep Dive
  5. Networking & Connectivity
  6. Screen Capture Pipeline
  7. Video Encoding Pipeline
  8. Input Handling
  9. State Synchronization
  10. Security Model
  11. Server Infrastructure
  12. Open Source Alternatives
  13. Building a Clone - Technical Approach
  14. Reddit Community Insights

Product Overview

Astropad Workbench (v1.1.0, build 799) is a remote desktop app for Mac, iPad, and iPhone. Built for the "AI era" - monitoring AI agents, remote workflows, and creative work.

Pricing

| Plan | Monthly | Annual | Limits | |------|---------|--------|--------| | Free | $0 | $0 | 20 min/day | | Unlimited | $10/mo | $50/yr | Unlimited |

System Requirements

  • Mac (host): macOS 15+ (Apple Silicon strongly preferred; Intel limited to baseline LIQUID codec)
  • iPhone/iPad (client): iOS 26+ / iPadOS 26+
  • No Windows support

Key Features

  • LIQUID proprietary streaming engine (shared with Luna Display & Astropad Studio)
  • Retina-grade "perceptually lossless" streaming
  • AES-256 end-to-end encryption
  • Global relay network (11 regions), no port-forwarding needed
  • Apple Pencil support with pressure sensitivity
  • Voice input/dictation
  • Multi-display consolidation into single virtual screen
  • Mini-map navigation

Binary Analysis

Build Environment (from Info.plist)

Build Date:     2026-04-07T16:40:03Z
Xcode:          26.11 (17B100)
SDK:            macOS 26.1 (25B74)
Architecture:   Universal (x86_64 + arm64)
Bundle ID:      com.astropad.workbench
Project Name:   Stargate (build 799)
Principal Class: Astropad_Workbench.StargateApplication
Update Feed:    https://downloads.astropad.com/workbench/mac/sparkle.xml

Linked Frameworks (key ones)

| Framework | Purpose | |-----------|---------| | ScreenCaptureKit | Screen capture (SCStream, SCContentFilter) | | VideoToolbox | Hardware H.264/HEVC encoding/decoding | | Metal | GPU rendering, tile processing | | CoreGraphics | CGEvent input injection | | Network.framework | QUIC/UDP networking | | CoreAudio / AVFAudio | Audio capture & streaming | | IOSurface | Zero-copy GPU buffer sharing | | CoreImage | Image processing | | IOKit | Hardware interaction | | Speech | Speech recognition for dictation | | Vision | Computer vision (likely for UI detection) | | Security | TLS/certificate management | | CoreLocation | Location services | | Sparkle | Auto-update framework | | Sentry | Crash reporting |

Third-Party Frameworks

  • Sparkle 2.7.0 - Auto-updates via https://downloads.astropad.com/workbench/mac/sparkle.xml
  • Sentry - Crash reporting & diagnostics

Internal Architecture

Project Codename: Stargate

The internal project name is "Stargate" - visible in build paths, class names, and the CI runner path:

/Users/admin/actions-runner/_work/Stargate/Stargate/

Language Stack

  • Swift (macOS app layer, SwiftUI UI)
  • Rust (LIQUID engine core, networking, codec, image processing)
  • Objective-C (bridging layer, especially for ScreenCaptureKit FFI)

Rust Core: liquid-next

The LIQUID engine is a Rust workspace located at AstroDeps/liquid-next/ containing these crates:

LIQUID Engine Crates (from binary symbols)

| Crate | Purpose | |-------|---------| | liquid_codec | Tile-based encoding/decoding, H.264/HEVC support, video stats | | liquid_net | Networking layer: QUIC, STUN/TURN, encrypted UDP, routing | | liquid_screencap | Screen capture via ScreenCaptureKit (macOS backend) | | liquid_input | Input handling: cursor monitoring, window tracking (macOS) | | liquid_image_processing | wgpu-based GPU image processing: diffing, scaling pipelines | | liquid_manager | Session management: client/server loops, screen selection | | liquid_rate_control | Adaptive bitrate: integrated & Kalman filter approaches | | liquid_statesync / liquid_statesync_core | State sync between peers via FFI to Swift | | liquid_reporter | Diagnostics & telemetry | | liquid_selection / liquid_selection_types | Content selection (windows, displays) | | liquid_metrics | Performance metrics collection | | liquid_web_server | Local web server (endpoints for configuration) |

Private Astropad Crates (from crates.astropadcloud.com)

| Crate | Version | Purpose | |-------|---------|---------| | astro-auth | 0.14.1 | JWT-based auth (HS256/ES256/EdDSA), shared sessions, token refresh | | astro-key-store | 0.9.4 | Peer identity, certificate management, keychain storage | | astro-peer-identity | 0.14.0 | Peer identification & authorization | | astro-identity | 0.10.1 | User identity management | | astro-virtual-display | 0.1.0 | Virtual display creation/management (CGVirtualDisplay private API) | | astro-diagnostics | 0.3.1 | Diagnostic reporting, tracing | | astro-reporting | 0.4.1 | Azure-hosted reporting endpoint | | astro-sysinfo | 0.10.7 | System information collection | | astro-remote | 0.10.2 | Remote connection management | | astro-monitor | 8.1.0 | Display monitoring | | astro-edid | 2.1.0 | EDID display metadata | | astro-hermes-uploader | 0.1.0 | Hermes analytics upload | | kv_db | 0.2.0 | Key-value database | | scoped-thread-pool | 1.0.4 | Thread pool management |

Additional Rust Components

| Component | Purpose | |-----------|---------| | apple_api/src/sc_kit/ | Rust FFI bindings for ScreenCaptureKit | | astro-clipboard/ | Cross-platform clipboard sync | | astro-sync-quinn/ | QUIC-based synchronization (client + server) | | segmented_payload/ | Payload segmentation/aggregation | | tokio_utils/ | Async utilities (debouncing) |

Microservices Architecture (from binary paths)

The app uses an internal microservice/actor pattern:

| Service | Location | Purpose | |---------|----------|---------| | video_encoder | services/video_encoder/ | Screen capture → tile encoding → H.264/HEVC | | video_decoder | services/video_decoder/ | H.264/HEVC decoding on client | | video_network | services/video_network/ | Video frame transport | | event_network | services/event_network/ | Event stream transport | | event_renderer | services/event_renderer/ | Event rendering on client | | event_sender | services/event_sender/ | Event dispatch | | clipboard_sync | services/clipboard_sync/ | Bidirectional clipboard sync (local/remote/rx/files) | | statesync | services/statesync/ | State synchronization protocol | | multiplayer_input_host | services/multiplayer_input_host/ | Input injection on host (cursor, keyboard) | | multiplayer_input_client | services/multiplayer_input_client/ | Input capture on client | | multiplayer_input_protocol | services/multiplayer_input_protocol/ | Input protocol (cursor info, service errors) | | virtual_display_client | services/virtual_display_client/ | Virtual display management (client-side) | | virtual_display_server | services/virtual_display_server/ | Virtual display management (server-side) |

Swift App Layer (from debug paths)

Stargate/Stargate/
├── API/
│   └── BenchAPIClient.swift          # API client for bench.astropad.com
├── AppDelegate.swift                  # Main app delegate
├── AppDelegate+ClientRequestedContent.swift
├── AppDelegate+DecoderWindowController.swift
├── AppDelegate+Diagnostic.swift
├── AppDelegate+MeetingDetector.swift
├── AppDelegate+SleepTracking.swift
├── AppState/
│   └── ConnectedPeer.swift
├── AX/
│   └── NSRunningApplication+AX.swift  # Accessibility API integration
├── ChromeWindows/
│   ├── ClientChromeWindow.swift
│   ├── Views/ClientToolbarView.swift
│   └── WorkbenchChromeWindow.swift
├── Client/
│   ├── Decoder.swift                  # Video decoder
│   ├── Decoder+EventProcessing.swift
│   ├── Decoder+H26x.swift            # H.264/H.265 specific
│   ├── Decoder+LiquidClient.swift     # LIQUID client integration
│   ├── Decoder+OutputSize.swift
│   ├── Decoder+SharedContentState.swift
│   ├── Decoder+StateSync.swift
│   ├── DecoderProcessor.swift
│   ├── DecoderWindowController.swift
│   ├── DecoderWindowController+FullScreen.swift
│   ├── H264x.swift                    # H.264 handling
│   ├── ClientMultiplayerState.swift
│   └── SystemKeyEventTap.swift        # System-wide key event interception
├── LIQEvent+Extensions.swift
├── LiquidRuntime.swift                # Rust↔Swift bridge
├── Onboarding/
│   ├── OnboardingPermissions.swift
│   ├── OnboardingUI.swift
│   ├── OnboardingWindow.swift
│   └── Permissions/
│       ├── AccessibilityPermission.swift
│       └── PermissionHelper.swift
├── SelectionUI/
│   ├── ContentSelectorController.swift
│   └── DropAreasWindow.swift
├── Server/
│   ├── InputOSInjection/
│   │   └── InputOSInjector.swift      # CGEvent-based input injection
│   └── SharedContent/
│       ├── FloatingControlsWindow.swift
│       ├── MultiPlayerViewController.swift
│       ├── SharedContentController.swift
│       └── SharedContentTracker.swift
├── Utils/
│   ├── ConnectionRequestManager.swift
│   ├── Image+Util.swift
│   ├── ImageLoader.swift
│   ├── NetworkObserver.swift
│   └── SleepPrevention.swift
└── Workbench/
    ├── AvailableContent.swift
    └── Diagnostic/
        └── ReportGeneratedView.swift

MeetingDetection/
├── ContactSelectorView.swift
├── DetectedMeeting.swift
└── WebBrowserTracker.swift

Swift System Classes (from Obj-C metadata)

StargateApplication          # NSApplication subclass
StargateState                # Main app state (observable)
├── DebugSystem              # Debug overlay & tools
├── InputSystem              # Input event handling
├── ClientViewSystem         # Client view management
├── ClipboardSyncSystem      # Clipboard synchronization
├── MeetingContactsRepo      # Meeting detection contacts
├── VirtualDisplaySystem     # Virtual display lifecycle
├── CapturedContentSystem    # Captured screen content
├── CaptureSettingsSystem    # Capture configuration
└── CapturableContentSystem  # Available content enumeration
StargateDefaults             # UserDefaults wrapper

LIQUID Protocol Deep Dive

Architecture

LIQUID is a tile-based rendering protocol — NOT a traditional video codec:

  1. Tile Division: Screen divided into independent tiles (typically 64x64 or 128x128 pixels)
  2. Change Detection: Only changed tiles are processed and transmitted (via wgpu diffing pipeline)
  3. Independent Tiles: No inter-frame prediction (no P/B-frames for tile data)
  4. GPU-Accelerated Diffing: Uses wgpu compute shaders for pixel diffing (all_pixels_diffing, checkboard_diffing)

Dual Codec System

From binary analysis, LIQUID supports two parallel encoding modes:

TileCodecOnly     - Pure tile-based (LIQUID native, lowest latency)
H26xCodecOnly     - H.264/HEVC full-frame (for motion-heavy content)

The system dynamically switches between these based on content:

  • Static/drawing content → Tile codec (raw pixel tiles via UDP)
  • Full-screen video/motion → H.264/HEVC via VideoToolbox
  • Both can run simultaneously for different screen regions

Key Internal Types (from binary strings)

// Codec types
enum CodecMode { TileCodecOnly, H26xCodecOnly }

// Frame handling
struct H264EncodedFrame { frame_id, segment_num, total_segments, tile_info }
struct H264EncodedFrameSegment { segment_num, total_segments, tile_info_dead_bit_count, tile_info, encoded_frame_data }

// Tile system
struct TileInfo { ... }  // Tile metadata with dead bit count
struct TileBuffer { ... }  // Tile pixel data

// Encoder config
struct EncoderConfig {
    target_size: TargetSize,
    max_available_size: Size,
    scale: f64,
    drop_frames_when_busy: bool,
    load_style: LoadStyle,  // NetworkDriven or Static
    scheduler: TileScheduler,
    starting_frame_id: u64,
    debug_overlay_config: DebugOverlayConfig,
}

// Rate control
struct DebugOverlayConfig { flags: DebugOverlayFlags }

Encoding Pipeline (from module paths)

ScreenCaptureKit (SCStream)
  → liquid_screencap::capture::capture_macos::sc_kit::stream
  → IOSurface buffer
  → liquid_image_processing::wgpu::diffing_pipeline (GPU tile diff)
  → liquid_image_processing::wgpu::scaling_pipeline (resolution adapt)
  → liquid_codec::encoder::tile_scheduler (tile selection)
  → liquid_codec::encoder::compressor (tile compression)
  → liquid_codec::video_encoder::apple (H.264/HEVC via VideoToolbox)
  → liquid_codec::render_proxy::frame_aggregator (frame assembly)
  → segmented_payload (packet segmentation)
  → liquid_net::runtime::datagram (UDP send)

Latency

  • Local network: ≤ 16ms end-to-end (claimed by Astropad blog)
  • The tile approach inherently has lower latency than full-frame encoding since only changed regions are sent

Networking & Connectivity

Transport Protocol: QUIC (via Quinn)

The binary contains quinn-proto 0.11.9 - a Rust QUIC implementation. This is the PRIMARY transport:

QUIC is used for:
- Reliable stream transport (control, state sync, clipboard)
- Unreliable datagrams (video tiles, audio)
- Built-in encryption (TLS 1.3)
- Connection migration
- 0-RTT reconnection

Signaling: MQTT (via mqtts://)

Peer discovery and signaling uses MQTT over TLS:

Production: mqtts://signal.astropad.com:1883
Cloud:      mqtts://signal.astropadcloud.com

NAT Traversal

  • STUN: Used for public IP discovery (STUN Error: strings present)
  • TURN: Relay fallback when direct connection fails (TURN Error: strings present)
  • QUIC/TURN task: Combined QUIC-over-TURN for relay scenarios

Connection Types (from binary)

ListenerQUIC            - QUIC server listener
ListenerUSB             - USB direct connection
RestoreLastConnection   - Reconnect to last peer
ClientQR                - QR code pairing
ClientManual            - Manual IP entry
ClientUSB               - USB client connection
ClientDiscovery         - Automatic peer discovery

Network Quality Monitoring

liquid_net::runtime::network_quality  // Real-time quality assessment
liquid_net::runtime::datagram::pulse  // Heartbeat/keepalive
liquid_net::throttle                  // Bandwidth throttling
liquid_rate_control::integrated       // Integrated rate control
liquid_rate_control::kalman           // Kalman filter for rate estimation

Encrypted UDP

Beyond QUIC's built-in TLS, there's an additional layer:

liquid_net::sockets::encrypted_udp::encryption_keys  // Custom UDP encryption

Message Channels (from router)

Message          - Generic messages
Fragment         - Fragmented large messages
Analytics        - Telemetry data
StateSyncEvent   - State synchronization events
Video            - Video frame data
Connectivity     - Connection state updates
Diagnostics      - Debug information
AppSpecific      - Custom app messages
StateSyncStargateClient  - Stargate-specific client state
StateSyncStargateServer  - Stargate-specific server state
MultiplayerInputCoordination - Input event coordination
ClipboardSync    - Clipboard data sync

Screen Capture Pipeline

macOS Backend

liquid_screencap::capture::capture_macos::sc_kit    # ScreenCaptureKit wrapper
liquid_screencap::capture::capture_macos::backend    # Capture orchestration
liquid_screencap::capture::capture_macos::display_stream  # Display stream management
liquid_screencap::capture::capture_macos::img_processing  # Post-capture processing
liquid_screencap::io_surface                         # IOSurface buffer handling

ScreenCaptureKit Integration

  • Uses SCStream for screen capture
  • SCContentFilter for window/display selection
  • Custom stream::delegate and stream::output for receiving frames
  • filters::windows_wrappers for window-level capture
  • shareable_content.rs for enumerating capturable content

Image Processing (GPU via wgpu)

liquid_image_processing::wgpu::diffing_pipeline    # GPU-accelerated change detection
  - all_pixels_diffing      # Full pixel comparison
  - checkboard_diffing      # Checkerboard pattern diffing (optimization)
liquid_image_processing::wgpu::scaling_pipeline    # Resolution scaling
liquid_image_processing::wgpu::resources::mapped_buffer  # GPU↔CPU buffer mapping
liquid_image_processing::pool                       # Buffer pool management

Video Encoding Pipeline

H.264/HEVC via VideoToolbox

liquid_codec::video_encoder::apple    // Apple VideoToolbox encoder
liquid_codec::video_encoder::data_limits  // Bitrate control

// Metrics tracked:
codec.h26x_bytes    // H.264/HEVC byte count
codec.tile_bytes    // Raw tile byte count
codec.keyframes.regular    // Regular keyframes
codec.keyframes.recovery   // Recovery keyframes (error recovery)

Tile Encoding

liquid_codec::tiles::tile_processor    // Individual tile compression
liquid_codec::tiles::compressed_tile   // Compressed tile format
liquid_codec::tiles::tilebuffer        // Tile buffer management

liquid_codec::encoder::tile_scheduler  // Decides which tiles to encode
liquid_codec::encoder::compressor      // Tile compression engine
liquid_codec::encoder::encoder_inner   // Core encoding logic

Frame Assembly

liquid_codec::render_proxy::frame_aggregator  // Assembles tiles into frames
liquid_codec::render_proxy::fps               // FPS tracking

liquid_codec::frame::frame_grid              // Tile grid layout

// Segmentation for network transport
segmented_payload::segmentation   // Break frames into network packets
segmented_payload::aggregation    // Reassemble on receiver
segmented_payload::segment        // Individual segment handling

Decoding (Client Side)

Decoder.swift               // Main decoder
Decoder+H26x.swift          // H.264/HEVC decoding via VTDecompressionSession
Decoder+LiquidClient.swift  // LIQUID tile reassembly
DecoderProcessor.swift      // Frame processing pipeline
H264x.swift                 // H.264 specific utilities

Input Handling

Server-Side (Host Mac)

InputOSInjector.swift          // CGEvent-based input injection

liquid_input::os::mac            // macOS input module
liquid_input::os::mac::cursor_monitor     // Cursor position tracking
liquid_input::os::mac::multiplayer_state  // Multi-user input state
liquid_input::os::mac::window_tracking    // Window focus tracking
liquid_input::os::mac::unsupported_events // Handling unsupported input
liquid_input::types::raw_type_conversions // Input type normalization

Client-Side (iPad/iPhone)

multiplayer_input_client/     // Client input capture
multiplayer_input_protocol/   // Wire format for input events
  - current_cursor            // Cursor position messages
  - cursor_info               // Extended cursor metadata (per-platform)
  - service_error             // Error handling

Input Method: CGEvent (confirmed from linked frameworks)

  • Uses ApplicationServices.framework for CGEventPost, CGWarpMouseCursorPosition
  • SystemKeyEventTap.swift for intercepting system key events via CGEvent.tapCreate
  • Requires Accessibility permission

Apple Pencil

  • Apple Pencil acts as "precise cursor controller" (from App Store listing)
  • Pressure sensitivity transmitted via the multiplayer input protocol
  • kCGTabletEventPointPressure (0.0-1.0) injected on host via CGEvent

State Synchronization

StateSync System

A sophisticated bidirectional state sync between client and server:

liquid_statesync_core::ffi::statesync  // Core state sync FFI
  - capturable_content    // What can be captured
  - client_view           // Client viewport state
  - clipboard_sync        // Clipboard state
  - debug                 // Debug state
  - input                 // Input state
  - meeting_contacts_repo // Meeting contacts
  - virtual_display       // Virtual display config

liquid_statesync_core::store        // State store (persistent)
liquid_statesync_core::store::flusher  // Periodic state flush
liquid_statesync_core::observe::notifier  // Change notifications
liquid_statesync_core::ffi::subscriber   // State change subscribers

Network Protocol

services/statesync/
├── lib.rs              // Service entry point
├── network_message.rs  // Wire format for state sync messages
└── protocol.rs         // State sync protocol definition

Security Model

Encryption

  • QUIC TLS 1.3 for transport encryption (via quinn-proto)
  • AES-256-GCM for additional data encryption (OpenSSL-based, from string analysis)
  • CHACHA20_POLY1305 also available as cipher option
  • Encrypted UDP layer for datagram encryption

Authentication

  • JWT tokens with multiple algorithm support (HS256, HS384, HS512, ES256, ES384, RS256, EdDSA)
  • Shared sessions with claims: exp, iss, sub, owner_id, peer_id
  • Peer authorization: Authorize, AuthorizeTemporary, Deny, DenyPersistent
  • Certificate-based peer identity via astro-key-store
  • Keychain integration for secure key storage

Peer Identity Flow

1. Generate certificate & key pair (astro-key-store)
2. Store in macOS Keychain
3. Exchange peer identities via MQTT signaling
4. Verify certificates during QUIC handshake
5. Authorize/deny peers based on stored trust
6. Detect MITM: "Peer is authorized with a different hash, did they partially reset or MITM?"

Server Infrastructure

API Endpoints

| URL | Purpose | |-----|---------| | https://bench.astropad.com | Production API | | https://bench.astropad.com/api/v1 | REST API v1 | | https://sbench.astropad.com/api/v1 | Staging API | | mqtts://signal.astropad.com:1883 | MQTT signaling (production) | | mqtts://signal.astropadcloud.com | MQTT signaling (cloud) | | https://hermes.astropad.com | Analytics/telemetry | | https://astropad-44f7fcc4d74ab.flex.countly.com | Countly analytics | | https://astropadwest-reporting-64356.azurewebsites.net/api/reports | Azure reporting | | https://downloads.astropad.com/workbench/mac/sparkle.xml | Update feed |

Environment Flavors (from binary)

Apollo      - Internal (shared code name with Luna Display?)
Bench       - Workbench production
Beta        - Beta testing
Dev         - Development
LocalDev    - Local development
PR          - Pull request testing
Staging     - Staging environment
Custom      - Custom deployment

Build Infrastructure

  • GitHub Actions runner at /Users/admin/actions-runner/_work/Stargate/Stargate/
  • Private Cargo registry at crates.astropadcloud.com
  • Xcode 26.11 (Xcode 17B100)

Open Source Alternatives

For Building Something Similar

| Project | Approach | macOS Host | iOS Client | Latency | Stars | |---------|----------|------------|------------|---------|-------| | Lumen (Sunshine fork) | ScreenCaptureKit + VideoToolbox + virtual displays | Yes (Apple Silicon) | Moonlight iOS | ~1ms encode | 112+ | | Sunshine | Cross-platform game streaming host | Yes (limited) | Moonlight iOS | <30ms | 11.8k | | RustDesk | Rust-based remote desktop | Yes | Yes | 30-60ms | 73k | | Deskreen | WebRTC browser-based | Yes | No (Safari limitations) | 100-200ms | 17k |

Key Takeaway from Reddit

Lumen (github.com/trollzem/Lumen) is the most promising open-source base:

  • Fork of Sunshine specifically for Apple Silicon
  • Already uses ScreenCaptureKit + VideoToolbox (same APIs as Astropad)
  • System audio via ScreenCaptureKit (no BlackHole needed)
  • Virtual display via CGVirtualDisplay (same private API as Astropad)
  • 1ms encode latency on M4 Mac Mini
  • Missing: Apple Pencil support, tile-based encoding, adaptive codec switching

Building a Clone - Technical Approach

Phase 1: Core Screen Streaming (MVP)

1. Screen Capture

// Use ScreenCaptureKit (same as Astropad)
let content = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: true)
let filter = SCContentFilter(display: content.displays.first!, excludingApplications: [], exceptingWindows: [])

let config = SCStreamConfiguration()
config.width = 1920
config.height = 1080
config.minimumFrameInterval = CMTime(value: 1, timescale: 60)
config.showsCursor = true
config.capturesAudio = true
config.sampleRate = 48000

let stream = SCStream(filter: filter, configuration: config, delegate: self)
try await stream.startCapture()

2. Video Encoding

// Hardware H.264/HEVC via VideoToolbox
VTCompressionSessionCreate(nil, width, height, kCMVideoCodecType_HEVC, nil, nil, nil, callback, nil, &session)
VTSessionSetProperty(session, kVTCompressionPropertyKey_RealTime, kCFBooleanTrue)
VTSessionSetProperty(session, kVTCompressionPropertyKey_AllowFrameReordering, kCFBooleanFalse)
// Encode CVPixelBuffer from ScreenCaptureKit
VTCompressionSessionEncodeFrame(session, pixelBuffer, pts, duration, nil, nil, nil)

3. Input Injection

// CGEvent for mouse/keyboard injection
let mouseEvent = CGEvent(mouseEventSource: nil, mouseEventType: .mouseMoved, mouseCursorPosition: point, mouseButton: .left)
mouseEvent?.post(tap: .cghidEventTap)

// Tablet pressure via CGEvent
let tabletEvent = CGEvent(mouseEventSource: nil, mouseEventType: .mouseMoved, mouseCursorPosition: point, mouseButton: .left)
tabletEvent?.setDoubleValueField(.tabletEventPointPressure, value: pressure)
tabletEvent?.post(tap: .cghidEventTap)

4. Networking (QUIC)

Use Quinn (Rust) or Network.framework (Swift) for QUIC transport:

  • Reliable streams for control, state sync, clipboard
  • Unreliable datagrams for video frames and audio
  • Built-in TLS 1.3 encryption

Phase 2: LIQUID-like Tile Engine

GPU Tile Diffing

// Metal compute shader for tile change detection
kernel void detectChangedTiles(
    texture2d<float, access::read> previousFrame [[texture(0)]],
    texture2d<float, access::read> currentFrame  [[texture(1)]],
    device uint* changedTiles [[buffer(0)]],
    uint2 gid [[thread_position_in_grid]]
) {
    // Compare tile regions, mark changed tiles
    float4 prev = previousFrame.read(gid);
    float4 curr = currentFrame.read(gid);
    if (distance(prev, curr) > threshold) {
        atomicOr(&changedTiles[tileIndex], 1);
    }
}

Adaptive Codec Selection

if (changed_tile_ratio > 0.7) {
    // Lots of motion → switch to H.264/HEVC full-frame
    useH26xCodec()
} else {
    // Mostly static → tile-based streaming
    sendChangedTilesOnly()
}

Phase 3: NAT Traversal & Global Connectivity

Architecture

1. MQTT signaling server (peer discovery & SDP-like exchange)
2. STUN servers (public IP discovery)
3. TURN/relay servers (fallback for restrictive NATs)
4. QUIC transport (multiplexed streams + datagrams)

Services to Use

| Component | Option 1 | Option 2 | |-----------|----------|----------| | Signaling | Self-hosted MQTT (Mosquitto) | AWS IoT Core | | STUN | coturn (free) | Google STUN (stun.l.google.com:19302) | | TURN relay | coturn cluster on AWS/GCP | Cloudflare Realtime | | QUIC | Quinn (Rust) | Network.framework (Swift) |

Phase 4: Apple Pencil & Advanced Features

Apple Pencil Pressure

// On iOS client:
func pencilInteraction(_ interaction: UIPencilInteraction, didUpdate data: UIPencilInteractionData) {
    let pressure = data.pressure     // 0.0 - 1.0
    let altitude = data.altitudeAngle
    let azimuth = data.azimuthAngle
    // Send to host via QUIC
}

// On macOS host:
let event = CGEvent(mouseEventSource: nil, ...)
event?.setDoubleValueField(.tabletEventPointPressure, value: pressure)
event?.post(tap: .cghidEventTap)

Virtual Display

// Astropad uses private CGVirtualDisplay API (same as Lumen)
// Alternative: BetterDisplay open-source approach

Clipboard Sync

// macOS: NSPasteboard observation
// iOS: UIPasteboard monitoring
// Sync via QUIC reliable stream with UTType serialization

Reddit Community Insights

Key Findings from Reddit

  1. EasyCanvas is repeatedly recommended as a cheaper Astropad Studio alternative ($12 one-time vs $80/yr)
  2. Jump Desktop supports Apple Pencil pressure sensitivity for remote desktop
  3. Lumen (Sunshine fork for macOS) achieves 1ms encode latency with VideoToolbox
  4. Parsec is widely used for Mac remote desktop but lacks 4:4:4 on free tier
  5. Splashtop (v3.4.2+) supports stylus pressure events
  6. CGEventTap usage can cause App Store rejection under guideline 2.4.5 (important for distribution)
  7. Disabling AWDL (sudo ifconfig awdl0 down) fixes audio/video jitter when streaming from Mac
  8. Community sentiment: Astropad is technically excellent but overpriced

Competitive Landscape

| Product | Price | Pencil Pressure | Latency | Platform | |---------|-------|----------------|---------|----------| | Astropad Workbench | $50/yr | Yes | ~16ms local | Mac→iPad/iPhone | | Astropad Studio | $80/yr or $150 lifetime | Yes | ~16ms local | Mac/Win→iPad | | EasyCanvas | $12 one-time | Partial (hover added 2025) | Low | Mac/Win→iPad | | Jump Desktop | $18 one-time | Yes | Medium | Cross-platform | | Parsec | Free/$8/mo | No | ~15ms | Cross-platform | | Splashtop | $5/mo+ | Yes (v3.4.2+) | Medium | Cross-platform | | Luna Display | $130 hardware | Yes | ~16ms (hardware) | Mac→iPad | | Duet Display | $4/mo+ | Yes | Medium | Cross-platform |

Next Steps for Reverse Engineering

Immediate

  1. Network traffic analysis: Use Wireshark/mitmproxy to capture MQTT signaling and QUIC streams
  2. Binary decompilation: Use Hopper/Ghidra on the arm64 slice for deeper code analysis
  3. Runtime analysis: Use frida or dtrace to trace API calls during a live session
  4. class-dump: Extract Objective-C headers from the Swift/ObjC binary

Deeper Analysis

  1. LIQUID wire format: Capture and analyze UDP datagrams to understand tile encoding format
  2. MQTT protocol: Subscribe to signaling topics to understand peer discovery
  3. QUIC streams: Map stream IDs to service channels
  4. Rate control: Understand the Kalman filter-based adaptive bitrate algorithm

Building a Prototype

  1. Start with Lumen as a base (already has ScreenCaptureKit + VideoToolbox + virtual displays)
  2. Add tile-based diffing using Metal compute shaders
  3. Replace Sunshine's protocol with QUIC (Quinn)
  4. Add Apple Pencil pressure/tilt injection via CGEvent tablet events
  5. Build MQTT signaling for peer discovery
  6. Deploy coturn for STUN/TURN relay

Research compiled: April 9, 2026 Binary analyzed: Astropad Workbench 1.1.0 (799), built 2026-04-07

← All notes