17 KiB
Synthetic data generation pipelines for eye tracking
The generation of synthetic eye images has evolved from single-model path-traced renderers into a diverse ecosystem of pipelines spanning photorealistic 3D rendering, real-time rasterization, and deliberately abstract 2D image synthesis. Each pipeline navigates a fundamental tension between anatomical fidelity, rendering speed, and controllability — and none yet offers explicit, independent parameterization of stable inter-individual eye shape features such as orbital bone structure or palpebral fissure geometry. This report provides a detailed technical comparison of seven major generators, covering their rendering methods, randomization axes, label outputs, and acknowledged limitations, structured to support a thesis chapter on the synthesis/generation perspective.
From single scans to morphable models: the photorealistic lineage
The modern pipeline lineage begins with Świrski & Dodgson (2014), who first applied physically-based rendering to generate synthetic eye images for eye tracker evaluation. Their system used Blender's Cycles path tracer with a single public-domain head model, rendering corneal refraction, adjustable LED glints, and eyelid/pupil variation. Ground truth included pupil contour ellipses, glint locations, and gaze vectors. Though limited to one head model (white male), a spherical cornea, and no bright-pupil imaging, this work established the template — Blender-based path tracing with scripted parameter variation — that all subsequent photorealistic pipelines would follow.
SynthesEyes (Wood et al., ICCV 2015) expanded this approach to 10 high-quality head scans from Ten24's 3D Scan Store (0.1 mm geometry, 10K diffuse textures), each manually retopologized in ZBrush to ~9,005 polygons with edge loops following exterior eye muscles. The two-part eyeball model used a transparent refractive outer shell (corneal index n = 1.376) over an opaque inner surface carrying iris and sclera textures. Skin used physically-based subsurface scattering. Rendering at 150 rays per pixel produced photorealistic output at 120 × 80 px in 5.26 seconds per image on an NVIDIA GTX 660. Randomization covered gaze direction (constrained to ±25° pitch, ±35° yaw per MIL-STD-1472G), head pose (40° total via spherical camera placement), four HDR environment maps with random rotation and intensity scaling, four iris colors, three sclera tints, and blend-shape-driven pupil dilation and eyelid pose. The pipeline produced 28 eye-region landmarks (12 eyelid, 8 iris, 8 pupil) in both 2D and 3D, plus gaze vectors and head pose matrices — targeting eye-shape registration and appearance-based gaze estimation. Key limitations included the labor-intensive per-model manual preparation, slow rendering (approximately two months for one million images), only 10 identity models, and no modeling of glasses, makeup, or emotion-related deformations. Cross-dataset evaluation revealed a substantial domain gap (13.91° vs. 6.33° on MPIIGaze).
UnityEyes (Wood et al., ETRA 2016) was explicitly designed to overcome three SynthesEyes bottlenecks: slow rendering, limited identity variation, and manual per-model rigging. By switching to Unity 5.2's real-time rasterizer, it achieved 23 ms per image (3.6 ms rendering, remainder I/O) — a 200× speedup — at a higher resolution of 400 × 300 px. The central innovation was a PCA-based generative morphable eye region model: all 20 head scans were registered to a single 229-vertex topology, and principal component analysis yielded a continuous shape space from which infinite novel eye regions could be sampled (s(α) = μ + U·diag(σ)·αᵀ, α ∼ N(0, I)). The first principal component captured the hooded-versus-protruding eye axis. Eyelid animation became fully procedural — anatomically-inspired geometric rotations with pivot interpolation and shrinkwrapping to the eyeball surface — eliminating manual blend shapes. Corneal refraction was approximated via a fragment shader texture offset rather than true ray tracing, and skin used a pre-integrated scattering shader. UnityEyes randomized eye region shape (PCA), 20 skin textures, iris photo-textures, iris width, pupil size, gaze (±30° default, user-configurable), head pose (±30°), 20 HDR panoramas (vs. 4 in SynthesEyes) with random rotation and exposure, plus a random directional light. Labels included interior margin, caruncle, and iris landmarks, gaze vectors, pupil/iris size, and head pose in JSON format. Over one million images were generated in under 12 hours. Despite lower per-image photorealism, UnityEyes achieved marginally better gaze estimation than SynthesEyes (9.95° vs. 10.09° on MPIIGaze with k-NN) because dataset scale and shape diversity compensated for rendering fidelity — a result that shaped subsequent thinking about the photorealism-versus-scale trade-off.
Anatomical depth: NVGaze and RIT-Eyes push biological fidelity
NVGaze (Kim et al., CHI 2019) from NVIDIA extended the SynthesEyes Blender pipeline with several anatomically critical corrections. The eye model was updated to a 24 mm diameter eyeball with a 7.8 mm corneal apex radius, and crucially introduced the ~5° visual-pupillary axis disparity and a nasal-superior pupil constriction shift (up to 0.25 mm) that occur in real eyes. Materials were calibrated for monochromatic 950 nm infrared imaging, including corneal index n = 1.38. Eyelid kinematics followed the empirical 4:1 upper-to-lower lid travel ratio. Four simulated IR LEDs produced corneal glints, and random camera slip modeled headset movement. Using 10 face models (diverse in gender, age, ethnicity), the pipeline randomized gaze, pupil diameter (2–8 mm), eyelid positions, iris rotation, and skin tone augmentation. At 1280 × 960 px with full path tracing, each image required approximately 30 seconds, making the full 2 million-image dataset equivalent to roughly 3.8 GPU-years — rendered in about one week on an NVIDIA multi-GPU cluster. NVGaze produced two segmentation maps (one standard with skin/pupil/iris/sclera/glints, one without face geometry for pixel-accurate labels under eyelid occlusion), 2D and 3D gaze and pupil coordinates, and blink labels. Ablation studies confirmed measurable improvements from the anatomical corrections: the eye model geometry fix, IR texture adjustment, and pupil center shift each contributed to reducing gaze estimation error (achieving 2.06° ± 0.44° on held-out real subjects with mixed training). Limitations included no myopic eyeball elongation, no crystalline lens, and no Listing's Law compliance for eyeball rotation.
RIT-Eyes (Nair, Kothari et al., SAP 2020; earlier ETRA 2020 abstract) pursued the most comprehensive anatomical modeling of any open pipeline. Built on Blender 2.8 Cycles at 200 rays per pixel, it used 24 head models from 3DScanStore with 8K color maps. Its eye model introduced several features absent from all predecessors: an aspherical cornea parameterized as a spheroid with asphericity Q drawn from three values (−0.130, −0.250, −0.370, spanning the population distribution around the mean of −0.250); a deformable iris with actual pupil aperture (not an opaque disc) causing realistic iris texture deformation during dilation; retinal retroreflection following a Beckmann distribution to simulate the bright-pupil effect when IR source and camera axis are within ~2.25°; an explicit lacrimal caruncle; a tear film with glossy and transparent shader properties; and gaze-coordinated eyelid deformation via Blender's wrapping function with closure approximated as a linear function of vertical eye rotation. Half of all images included reflective eyeglasses (though without refraction — only reflection). Randomization covered gaze (±30° in azimuth and elevation), pupil aperture radius (1–4 mm uniform), corneal asphericity, 9 IR iris textures with random rotation, 25 HDR environment maps (9 indoor, 16 outdoor from HDRI Haven) with ±50% intensity variation and multi-axis rotation, camera distance (2.5–4.5 cm), and eyelid state (including fully closed images). Three dataset variants were rendered (S-NVGaze, S-OpenEDS, S-General) totaling 154,800 images at 640 × 480 or 400 × 640 px, mimicking different real hardware configurations. Labels comprised pixel-level segmentation masks (pupil, iris, sclera, background), 2D/3D eye feature centers, and eye pose. The feature comparison table in the paper shows RIT-Eyes as the only pipeline simultaneously offering aspherical cornea, retroreflection, segmentation masks, both IR and RGB rendering, reflective eyewear, lacrimal caruncle, and variable eyelids — at the cost of real-time capability. Notable limitations include only one sclera texture (causing poor sclera segmentation generalization), no refraction through eyeglasses, no eye makeup, and a persistent sim-to-real gap (mIoU dropping from >95 within synthetic to 73–86 cross-domain). A follow-up, Temporal RIT-Eyes (Chaudhary et al., IEEE TVCG 2022), extended the pipeline to generate temporally contiguous gaze behavior sequences rather than i.i.d. frames.
Departing from photorealism: LEyes and the efficiency argument
LEyes (Byrne et al., Behavior Research Methods 2025) represents a deliberate paradigm shift. Its core hypothesis is that a model localizing pupils and corneal reflections is "nothing more than a model that is good at finding relatively dark or light pupil- or CR-shaped blobs in an image" — rendering photorealistic skin, sclera vasculature, and eyelashes is unnecessary overhead. Built on DeepTrack 2.1, a modular Python library originally developed for digital microscopy, LEyes models image features as 2D Gaussian light distributions without any 3D geometry. The pupil is a randomly oriented dark Gaussian with slight ellipticity (major-to-minor axis ratio 1.0–1.3) and exponentially distributed amplitude. Corneal reflections are bright Gaussians with luminance fixed at 255, truncated to create realistic saturation cores. The iris-pupil boundary emerges from a two-section background of different luminance values, with a randomly placed and oriented dividing line. Images are composited mathematically and discretized to 8-bit output.
The key innovation is on-the-fly generation during training: each epoch produces 1,000 unique images that are shown to the network once and then discarded, eliminating storage requirements entirely. Randomization covers Gaussian amplitude, axis radii, feature positions, orientations, background luminance, noise levels, and spurious reflections. A two-stage training regime tightens parameters in the second stage. Labels include pupil center, CR center, CR-to-illuminator matching, and segmentation masks. At 13 ms inference on a standard CPU, with training feasible on free Google Colab, LEyes targets P-CR (Pupil-Corneal Reflection) eye tracking specifically. The authors contrast their approach against NVGaze's ~3.8 GPU-years for dataset generation and demonstrate competitive or superior performance on OpenEDS 2019/2020 and other real datasets. Limitations include restriction to feature-level P-CR tracking (not appearance-based gaze estimation) and the need to analyze target hardware images to set appropriate parameter distributions.
UnityEyes 2 brings configurability and multi-camera support
UnityEyes 2 (Smith et al., ETRA 2025) is an open-source successor (MIT license, GitHub) to the original UnityEyes, rebuilt for modern eye-tracking hardware diversity. The core advance is full camera model configurability: pinhole perspective cameras with user-specified intrinsics (f_x, f_y, c_x, c_y, w, h) and 6-DOF extrinsics, calibratable from real hardware using standard OpenCV toolboxes. A novel camera array motion center allows multiple cameras to be defined as child transforms of a shared reference point with 6-DOF noise, simulating realistic head-device relative motion. Multi-camera setups generate synchronized views of the same eye state. Configuration is managed through JSON files for reproducible batch generation, with a GUI for rapid prototyping. Rendering speed reaches 85.7 images per second on an M3 Max MacBook Pro (comparable to the original UnityEyes' 82.0 img/s). Labels include 2D pupil center, normalized optical axis vector, and 3D eye globe center. In a demonstrated robotic eye-alignment application with three cameras, UnityEyes 2's camera-specific training reduced mean pixel error by 76% compared to generic UnityEyes training (7.4 ± 5.0 vs. 30.8 ± 38.4 pixels). The system remains a work in progress: custom face/environment distributions, a Python API, and advanced light types are planned but not yet implemented.
The absent axis: inter-individual eye shape as uncontrolled variation
A critical finding across all surveyed pipelines is that no generator explicitly parameterizes stable inter-individual eye shape variation — orbital bone structure, palpebral fissure geometry, eye socket depth, or eyeball size — as independently controllable axes. SynthesEyes and NVGaze select from 10 discrete head scans; RIT-Eyes uses 24 scans; Meta's recent Digital Eye Tracker Prototyper (2025) captures 195 identities via NeRF but cannot generate truly novel shapes. In all cases, anatomical variation is implicitly entangled within discrete 3D head models, making systematic exploration of how specific shape features affect downstream performance impossible. UnityEyes' PCA morphable model comes closest to continuous parameterization — its first principal component captures the hooded-versus-protruding eye distinction — but the shape coefficients lack semantic labels for individual anatomical features.
This gap has been explicitly identified in the literature. Fischer et al. (2023, Journal of Eye Movement Research) demonstrated that corneal asphericity alone significantly influences simulated eye-tracking outcomes, yet most pipelines use a fixed spherical cornea (only RIT-Eyes varies asphericity, and only across three values). The biometrically parameterized SyntEyes model (Rozema et al., 2011/2016) provides population-level distributions for corneal Zernike coefficients and intraocular distances but has never been integrated into an image generation pipeline. Wood et al.'s 3D Morphable Eye Region Model (ECCV 2016) and the more recent DREAM model (2025) offer PCA or tensor-decomposed identity dimensions, but both were designed for analysis-by-synthesis fitting rather than controlled synthetic data generation. The field thus faces a structural limitation: inter-individual shape diversity remains either discretely sampled or entangled, preventing controlled ablation of how specific anatomical features impact model training.
Bridging approaches: GAN refinement and neural rendering
Beyond the core generator pipelines, two complementary paradigms deserve mention. SimGAN (Shrivastava et al., Apple, CVPR 2017) introduced adversarial refinement of UnityEyes output: a refiner network transforms synthetic images to look more realistic while a self-regularization L1 loss preserves annotation fidelity (pupil center shift verified at only ~1.1 ± 0.8 px). This achieved a 21% relative improvement on MPIIGaze over raw synthetic training, with humans unable to distinguish refined from real images (51.7% accuracy in a visual Turing test). Follow-ups including GazeGAN (CycleGAN-inspired) and EyeGAN (segmentation-mask-conditioned) further developed this approach. Meta's Digital Eye Tracker Prototyper (March 2025) represents the cutting edge: NeRF-mesh hybrid representations from 195 real identities captured in a light-dome setup enable novel view synthesis with simulated optical effects (sensor noise, blur, glasses slippage), though diversity is bounded by the captured identity pool.
Conclusion
The evolution from Świrski & Dodgson's single-head renderer to today's ecosystem reveals two competing trajectories. The fidelity trajectory (SynthesEyes → NVGaze → RIT-Eyes) progressively adds anatomical structures — aspherical cornea, pupillary axis disparity, retinal retroreflection — but at escalating computational cost and with persistent sim-to-real gaps. The efficiency trajectory (UnityEyes → UnityEyes 2, LEyes) sacrifices anatomical detail for speed, configurability, and accessibility, achieving competitive or superior downstream performance through dataset scale and hardware-specific adaptation. LEyes' success with purely abstract 2D Gaussians challenges the assumption that photorealism is necessary for feature-level eye tracking. The most consequential open problem across all pipelines is the lack of controlled, semantically labeled inter-individual anatomical variation: without it, the field cannot systematically study how eye shape diversity affects model robustness, nor generate targeted training data for underrepresented morphologies. Integrating biometric eye models like SyntEyes into rendering pipelines — creating generators that can independently dial orbital depth, palpebral fissure width, or corneal asphericity — remains an unrealized but clearly identified next step.