fpv labs - infrastructure for physical intelligence

i joined fpv labs in nov 2025 as a mobile engineer. the job was not a typical app surface with a few forms and screens. the phone sat in the middle of a much harder workflow: record the world, move large datasets, and keep the session usable when devices, sensors, and networks behave badly.

that changed the standard for "done". a flow was only good if it survived field conditions, not just the simulator.

the first push was reliability. some modules were unstable enough to drag the app crash rate toward ~6%, which is unacceptable when the app is part of a recording workflow instead of a casual consumer session.

i refactored the shakier parts of the codebase and tightened error handling across ios and android, pushing the crash rate to under 1%. the benefit wasn't only cleaner logs. it meant operators could trust that a recording session would stay alive when something unexpected happened.

capture sessions get heavy fast, so i built a cross-platform upload pipeline for 10gb+ files with resumable uploads, background execution, and failure recovery.

the goal was simple: if an upload gets interrupted, the app should recover instead of making someone start from zero. that turned uploads from a best-effort feature into infrastructure the team could rely on.

i also built a high-fidelity ar data capture system on top of arkit and arcore, recording video, pose, point cloud, depth, and imu streams from the same session.

on top of capture, i designed a structured recording pipeline that packages sessions into mcap datasets compatible with ros2. the system keeps sampling deterministic at roughly 30 fps and handles session lifecycle cleanly, so what comes out is useful for downstream robotics and physical intelligence workflows, not just raw sensor noise.

these recordings happen in public spaces. so every session is anonymized on device before it can be uploaded, faces pixellated in every rgb frame of the mcap and in the thumbnail.

the part i actually care about is that it isn't a setting. no upload row can exist without the blur marker on disk, enforced inside the one function that creates uploads and inside every recovery path that could sneak around it. an unblurred session is not uploadable. there is no code path where forgetting is possible.

the rewrite is built to survive being killed. read the mcap linearly, blur the face topic frames with vision and coreimage, pass everything else through verbatim, write to a temp file, re-scan the temp to verify record counts and decode the first and last frame, then swap with an atomic replace. the marker gets written strictly last.

that ordering gives you a clean failure matrix. kill the app anywhere before the swap and the original is untouched, with an inert temp file that the next launch sweeps. kill between the swap and the marker and recovery re-blurs an already blurred file, which is a visual no-op and re-verifies anyway. every interruption lands somewhere recoverable.

decode, detect, pixellate, encode runs 30-45 ms per frame serial, so it fans out across three workers behind a reorder gate and lands around 0.3-1x the recording duration. memory stays bounded at six frames in flight.

there is also an inline mode that blurs during capture on the pre-encode image, so frames hit disk already anonymized and the whole rewrite is skipped along with its 2x transient disk cost. privacy stays structural there too: a frame is either scanned before it is written or, under backpressure, never written at all. if vision fails on a frame it still gets written but counted, and the session only earns its marker when that count is zero. anything else falls back to the full pass, which is idempotent over already blurred frames.

hand pose is the signal the recorder exists to capture. vision detects 21 joint skeletons and we lift them into 3d through the recording's own depth.

the rule that shaped the whole feature is the inverse of blur: hand pose never gates an upload. a detection failure, a skip, an unsupported session, all land on done with no poses and the recording uploads anyway. even the naming defends it. the gate is called isPostProcessEnabled and not isRequired, so nobody wires it into upload eligibility by accident a year from now.

it runs post capture by default, which means a recording can gain poses it was never recorded for, and a fix to the resolver can be applied to footage already sitting on disk.

it takes two passes over the file. pass one detects and writes nothing. pass two copies every record and merges the poses in by timestamp. the reason is emission order rather than pairing: by the time an rgb frame's depth partner arrives, its timestamp is already behind the writer's forward-only cursor, so the poses could never be written in order.

pass one buffers small structs, about 1 kb a frame, instead of serialized bytes. twenty minutes of footage is ~12 mb that way and ~236 mb the other way, and the second number gets you killed by the os on a phone.

the sharp edge is that detection parallelizes and chirality resolution does not. the resolver tracks hands across frames with decayed scores and hysteresis, so it has to run serially in timestamp order. splitting the core into a stateless detect stage and a serial resolve stage was the only way to fan out the expensive half.

both this and blur chain onto a single global rewrite queue, so two whole-file vision passes can never run at once and double peak memory. hand pose always runs after blur, enforced in two places.

the oracle is an offline script that scores a session against wilor. record the same scene twice, once inline and once post hoc, and compare. post hoc chirality agreement lands at inline's ~99.8%.

debugging a capture used to mean finishing the session, exporting it, and opening the file somewhere else. that loop is too slow when an operator is standing in a warehouse wondering if depth is even working.

so i built live mcap streaming. a native server on the phone speaks the foxglove websocket protocol on port 8765 and advertises itself over bonjour, so a laptop on the same network sees the device and can watch topics as they record.

the interesting part was where to tap. i subclassed the mcap writer and mirrored every schema, channel, and message to the server after the base class had written it to disk and released its lock. the writer itself stays untouched, and a test asserts a tapped and an untapped writer produce byte-identical files. streaming can never corrupt the artifact.

our channels are already cdr / ros2msg and json / jsonschema, both of which foxglove accepts natively, so the server forwards the recorded bytes unchanged. no second serialization path to keep in sync.

there is also a small viewer served off port 8766 for when nobody has foxglove installed. it decodes cdr in the browser and mirrors the in-app preview: camera with hand pose overlays, depth colormap, an orbitable 3d scene, plots. its decoders are cross checked against rosbags in ci so the browser and python agree.

ios ships today. the tap seam on android is the same choke point, so the port is mostly a websocket server and an mdns advert.

one camera only sees one side of a task. the fix is two phones recording the same scene, which sounds simple until you ask what timestamp means across two devices.

it splits into two problems that turn out to be independent. start together, and align afterwards.

start is the cheap one. the leader broadcasts a countdown, each device counts down on its own clock, and both flip into recording within roughly 5-25 ms. good enough, and it keeps clock sync off the critical path of scheduling.

alignment is the real work. ios gives you no shared hardware clock between phones, and arkit collaborative sessions are spatial only. they do not sync frame timestamps. so i ran a ping/pong protocol between the devices, cristian style, and recorded every raw measurement into the session sidecar. per sample we keep the offset from the lowest rtt round plus median, stddev, and the rtt spread, so post processing can re-verify the estimate instead of trusting it.

on top of the bursty samples there is a continuous path at 2.5 hz feeding a kalman filter over offset and drift. measurement variance per round is derived from rtt, timeouts and outliers get rejected, and the leader pushes its filter state to the follower.

one fix worth calling out. sync originally ran on the reliable transport and queued behind collaboration traffic, so rtt spiked to 60-200 ms against a 5-8 ms median. moving sync to the unreliable channel bypassed that queue entirely.

each device writes its own mcap. fusion happens offline: one command takes both session folders, remaps the follower onto the leader timeline with a piecewise linear model, merges into a single namespaced mcap, and emits a report with the measured sync gap. target is 1-2 ms alignment and a playback gap under 3 ms on pose.

arkit owns the wide camera and will not drive the ultra wide. you cannot have both. so getting ultra wide rgb and raw lidar depth from the same session meant dropping arkit for that mode and driving the cameras directly with AVCaptureMultiCamSession.

that is a real trade. no live pose, no mesh, no point cloud, since all three come from arkit. those get recovered offline instead. what you gain is raw ultra wide frames and raw lidar depth together, which is what the calibration and reconstruction work downstream actually wanted.

i shipped it as a third capture mode next to arkit mode and ultra wide only, toggled from the recording settings panel and persisted with the rest of the config. it writes into the same mcap file under a dedicated /ultrawide/... namespace, so nothing downstream needed a new reader. channels register lazily, so arkit mode files stay exactly as clean as they were.

sync is the part people get wrong here. wide and depth are genuinely hardware synced through a data output synchronizer, same pipeline, matching presentation timestamps, gap p99 effectively zero. wide and ultra wide are not. they are two free running 30 fps cameras on one session and land within about one frame of each other. that is the cost of the mode, not a bug, so the recorder emits a per callback sync diagnostic topic and a clock probe sidecar, and the offline analyzer turns them into gap histograms with budgets per pair.

imu lives in a different timebase again, and apple does not promise it matches the video clock. so the probe samples both at once and measures the offset and its drift per device per recording, instead of assuming.

runs on iphone 12 pro and later, hidden everywhere else.