In this article
I implemented five self-supervised learning recipes on 3.43 million anterior-segment OCT B-scans, the cross-sectional images used to examine the front of the eye. Training from the same encoder checkpoint produced markedly different feature geometry: full-field MAE spreads variation across more directions, two patch-limited MAE variants concentrate it, and JEPA produces the most dispersed representation in the recorded panel.
I built the monitoring and profiling workflow to make those changes inspectable alongside the cost of producing them. It also helped reduce two-GPU JEPA update time from 10.246 to 8.787 seconds, preserving the training recipe and passing exact equivalence and resume checks.
This note presents the aggregate measurements saved on 1 October 2026 at 00:15 CEST. The five-recipe article explains the experiment configurations; here I follow the data selection, the observed features, and the execution trace.
One retained corpus for five experiments
I assembled a research corpus of 3,663,454 OCT cross-sections from the MS-39 exports retained in Bronze. This collection was frozen in June 2026. Selecting the pretraining population and setting aside images for separate engineering checks leaves 3,475,579 images for quality scoring.
| Stage | OCT cross-sections |
|---|---|
| Prepared research corpus | 3,663,454 |
| Selected for pretraining, before quality filtering | 3,475,579 |
| Retained after quality filtering, shared by all five recipes | 3,427,130 |
The corpus selection record documents these counts. The histogram below starts at the second row: it shows the quality scores of the 3,475,579 selected images.
I use a logistic quality model built from 31 image- and mask-related variables to rank unsuitable inputs. QC085 is the retained index: the score must be strictly below float32(0.85).
That rule excludes 48,449 scans, or 1.39%, and retains 3,427,130. Compared with the previous 0.95 threshold, it removes another 12,323 scans. All five runs share this retained index, keeping the source population fixed across their different learning objectives.
- Below 0.85: retained
- 0.85–0.95: 12,323 additional exclusions
- 0.95–1.00: 36,126 already excluded
Each bar is a percentage of the 3,475,579 images scored for pretraining. Its height is on a log scale. The quality filter retains 3,427,130 images and excludes 48,449 (1.39%).
Open the full-size figureThe histogram shows where that decision acts: most scores lie near zero, with a smaller high-score tail. The score is an uncalibrated quality ranking, and 0.85 is the chosen cutoff on that scale. This distribution makes the size of the excluded tail visible alongside the retained training population.
Measure the same representation on the same images
The monitor periodically evaluates 256 fixed B-scans from the pretraining population. It reduces each full raster by three, applies no augmentation, and averages patch-token features after the encoder’s final LayerNorm. The result is one 1,024-dimensional vector per image, before the projector or decoder.
I center these vectors across images without L2-normalizing each vector. The squared singular values of this centered matrix describe how its variation is distributed. Writing those values as λ, the two plotted summaries are:
PC1 share = largest λ / sum(λ) Participation ratio = sum(λ)² / sum(λ²)
PC1 share measures how much variation lies in the strongest direction. Participation ratio, or PR, measures the effective number of contributing directions. For variances [1, 1, 1, 1], PC1 is 25% and PR is four. For [7, 1, 1, 1], PC1 rises to 70% and PR falls to about 1.92: four directions still vary, but one dominates.
This is why I track both. They reveal concentration that a scalar training loss can hide. The fixed panel removes changes in sampled images from the comparison, while the shared feature extraction makes the measurement consistent across objectives. With 256 centered observations, the measured rank is at most 255, regardless of the encoder’s 1,024 output coordinates.
The recipes produce different feature spectra
All encoders start from the same ImageNet-pretrained MAE checkpoint. The zero-update reference therefore measures the representation before domain pretraining.
- Full-field MAE
- Matched Global
- Matched CUNEX
- DiffMAE 2D
- JEPA + VISReg
- ImageNet-MAE initialization
Participation ratio
Variance explained by the first principal component
Snapshot: 1 October 2026, 00:15 CEST. JEPA is shown through its recorded 52,500-update diagnostic; the other runs reach 150,000 updates. The grey line marks the ImageNet-MAE initialization. Lines join the recorded measurements.
| Encoder or run | Updates in snapshot | Participation ratio | PC1 share |
|---|---|---|---|
| ImageNet-MAE initialization | 0 domain updates | 6.81 | 32.7% |
| Full-field MAE | 150,000 | 8.21 | 22.5% |
| Matched-Global | 150,000 | 3.38 | 50.9% |
| Matched-CUNEX | 150,000 | 2.47 | 61.8% |
| DiffMAE 2D | 150,000 | 6.56 | 32.7% |
| JEPA + VISReg | 52,500 | 13.59 | 16.3% |
The clearest contrast is between full-field reconstruction and the matched-patch arms. Full-field MAE increases PR from 6.81 to 8.21 and reduces the leading direction’s share. Both matched arms move toward greater concentration; in Matched-CUNEX, one direction accounts for nearly 62% of the measured variation.
DiffMAE ends close to the initialization on these two summaries. JEPA reaches PR 13.59 and PC1 16.3%, the broadest spectrum among the displayed checkpoints. This dated comparison captures JEPA at its recorded 52,500-update diagnostic and the other four arms at their completed 150,000 updates.
Each update presents 64 source images. The full 150,000-update budget is therefore 9.6 million presentations, approximately 2.8 passes over the retained corpus. JEPA processes ten derived views for each source presentation, adding computation within that shared exposure budget.
What I learned from the concentration
The matched arms reveal a shared behavior across two sampling strategies. Matched-Global samples the grid, while Matched-CUNEX uses a segmentation mask; both concentrate variation relative to their initialization. Their common restriction on patch count makes reduced context a plausible explanation to test, alongside the effects of masking and optimization.
JEPA explicitly shapes the distribution of representations through its VISReg regularizer. The regularizer acts after the projector, and its gradients train the encoder as well. The broader spectrum measured before the projector is consistent with that training signal.
PR and PC1 describe how variation is distributed in the features of this fixed training panel. The directions can encode anatomy, acquisition conditions or combinations of both. These measurements identify the representation behavior; the next task comparison asks which distinctions each encoder makes accessible.
Make the global–local recipe practical
JEPA combines two full-field views with eight native local views. Running that workload on two NVIDIA L40S GPUs required preserving the full-batch objective while distributing encoder work. I kept the batch of 64 sources, the 640-representation projector population, the global objective and gradient clipping, and BF16 precision unchanged.
Profiling exposed an input-preparation delay that appeared as a communication stall. Rank zero spent about 1.48 seconds in all-gather, the operation that collects work from both GPUs, waiting for the other rank to finish preparing its input and views on the CPU.
I changed the coordinator to prepare the next native batch and its first views while the GPUs computed the current update. This preserved source order and random-number state while overlapping preparation with useful GPU work.
The matched, unprofiled ABBA comparison ran the original and optimized variants in A–B–B–A order on the same workload, retaining ten update measurements per variant:
| Measurement | Original | With input prefetch |
|---|---|---|
| Mean time per optimizer update | 10.246 s | 8.787 s |
| Observed range | 10.024–10.511 s | 8.618–8.935 s |
That is 14.2% less time per update, or a 1.166× speedup. The timing includes input waiting, forward and backward computation, validity checks, gradient clipping, and AdamW. Periodic diagnostics and checkpoint saves are outside this timing window. After resuming the training run, a separate 20-update window averaged 8.799 seconds.
I verified the computation before deploying the change: inputs, features and loss scalars matched exactly, as did 303 gradient tensors and 1,221 model/optimizer-state tensors. The two ranks agreed, and the resume comparison was bit-identical. The faster implementation preserved the experiment’s numerical behavior in these qualification checks.
Compute dominates the optimized step
In a 26 September Nsight Compute capture, I inspected eight BF16 matrix-multiplication launches from a checkpoint replay. Measured compute throughput was 78.66–89.70% of the relevant peak, compared with 54.83–61.31% memory throughput. ML Training Monitor classified all eight sampled kernels as primarily compute-limited.
The 30 September optimized two-GPU trace then showed where the full update spent its time. GPU kernels occupied about 90% of each rank’s measured step, including about 7.9 seconds per GPU outside the NCCL communication library. Rank zero’s all-gather duration was about 36 milliseconds. Both GPUs were simultaneously idle for only 146 milliseconds of the 8.893-second aligned capture.
The kernel counters and execution timeline show a GPU-compute-dominated optimized step. At this point, the trace contained no other large isolated avoidable delay. The profiles explain where the time goes, and the matched unprofiled runs measure the speedup.
The next research question
These experiments have established different representation trajectories and a faster, equivalent implementation of the global–local recipe. I can now carry the encoders into a common patient-separated task comparison.
The prepared downstream pools contain 562 development patients with 23,891 B-scans and 140 reserved patients with 6,142 B-scans. Task-specific eligibility and labels will determine the final counts. A frozen-encoder probe trains a small predictor on fixed features; fine-tuning also updates the encoder. Comparing both, with checkpoint selection on development data, will connect the geometry measured here to performance on the selected tasks.
Data and measurement record
The aggregate snapshot and definitions contain the histogram counts, spectral curves, checkpoint positions and measurement scopes used in these figures. They were saved on 30 September 2026 at 22:15:56 UTC and remain a dated record as training continues.
The profiling summary records the separately dated Nsight kernel capture and two-GPU qualification. Hardware-counter definitions are documented in the Nsight Compute profiling guide.
The workflow is available as ML Training Monitor. The figures and timings above come from the recorded experiments. A separate 48-second software demonstration uses labeled synthetic telemetry.