Thermal camera hub · follows Seeing millikelvin heat with the P3

Watching heat flow through the M5 keyboard

Switch the M5's processors on for a minute at a time and the P3 films the heat working its way out. It reaches the right side of the view in about 20 seconds and the left keys only after 40–50, and it keeps arriving for 25 seconds after the load stops. The CPU and the GPU push it out in different places. Under CPU load a strip down the middle of the view cools while everything around it warms; under GPU load that same strip warms first.

Thermal Master P3 hung from the M5 MacBook's screen, looking down at the keyboard, in calibrated mode, with the shutter fired 5 s before every load start; 0.72 mm per pixel. Thu 15 Oct 2026, 09:04–11:12. Five loads: all 16 cores, the 6 performance cores, the GPU, the 12 efficiency cores, and none. Each ran 60 s on and 120 s off, six cycles. The first cycle is dropped and the next five averaged (three for the performance cores).

20 s
until the right side of the view reaches half its rise (all cores)
40–50 s
until the left keys do
+25 s
the keyboard keeps warming after the load stops
5 mK
noise per 1 s picture, from 36 mK per frame
All 16 cores, 60 s on then 120 s off, five cycles averaged, with a no-load run on the same schedule subtracted. Red is warmer than just before the start, blue cooler, grey a change under 3× the noise. The scale (±2 K) is set by the keyboard. The four spots at the top saturate it: they react to any load within a second and are not CPU heat (see the overnight report).

Heat sweeps across the keyboard from right to left

Map of when each pixel reaches half its final rise under all-core load: the right side in about 20 s, the middle in 30–35 s, the left keys in 40–50 s, with arrows pointing from right to left and a hatched strip down the middle that cools
When heat arrives: each pixel coloured by the time it reaches half its final rise, during the 60 s of all-core load. Contours every 10 s. The arrows follow the front from earlier to later. Hatched blue: pixels that cool instead.

Timing every pixel turns the movie into one map. The right half of the view gets there in about 20 s, the middle in 30–35 s, and the left keys in 40–50 s. On the left, each keycap reaches its half-rise about 10 s after the metal around it: plastic keys over air are slower than the aluminium frame between them. Last night's run, analysed without the control subtraction, shows the same right-to-left order, so it is not an artefact of the correction.

The strip down the middle goes the other way. About 10 s into the load it starts cooling, by up to 2.5 K by the end of the minute, while the keys on both sides warm.

Two space-time plots: heat along a row and along a column through the earliest-warming point, against time; the black line, when each point reaches half its rise, rises from 20 s on the right to 40–50 s on the left; a vertical blue band marks the strip that cools
Heat against position (across) and time (up), along a row and a column through ★, the earliest-warming point. The black line marks when each point reaches half its rise. The blue band is the strip that cools.

The CPU and the GPU heat it from different places

The same 60 s schedule with five different loads. The first 90 s of the averaged cycle, each panel on its own scale.

CPU load (all 16 cores, or the 6 performance cores alone) cools the middle strip and warms the right side first. GPU load warms that strip first, reaching half its rise in 10–15 s, and nothing cools. The heat enters the keyboard at different places for the two, so the GPU and the CPU have different paths to the surface. Why the strip cools under CPU load is not known yet. Airflow from the fans is the obvious suspect, but fan speed was not logged.

The 6 performance cores give about 80% of what all 16 give, so the efficiency cores contribute little. Alone, the 12 efficiency cores give about a quarter of the all-core heat. That is too little to separate from the camera's own drift: their row looks like the no-load row.

Five arrival-time maps side by side: all cores and performance cores warm the right side first with the middle strip cooling; the GPU warms the middle strip first; efficiency cores and no load show only a uniform 16 s drift
When heat arrives, for each load (time to half the final rise, 0–60 s; grey: cools or too weak). The efficiency-core and no-load maps show only the camera's own drift, which settles with a ~16 s time constant after each shutter.
Load (cycles averaged)Median rise at 20 sMedian rise at 60 sMiddle stripWarms first
All 16 cores (5)115 mK660 mKcools, by up to 2.5 Kright side, ~20 s
6 performance cores (3)147 mK532 mKcoolsright side, ~19 s
GPU (5)167 mK453 mKwarms firstmiddle strip, 10–15 s
12 efficiency cores (5)56 mK175 mKnot separable from the camera's drift
No load: the camera's own drift (5)139 mK148 mKmedian |drift| after the shutter; subtracted from the rows above
Grid of rise maps at 5, 15, 30 and 60 s after the load starts, one row per load; all cores and performance cores show the left keys warming and the middle strip cooling, the GPU shows the strip warming, efficiency cores and no load show broad drift patterns
Rise at 5, 15, 30 and 60 s after the load starts, one row per load, each row on its own scale. The no-load row is the pattern the camera itself makes after its shutter: the corners warm relative to the centre.

Pulling the heat out of the noise

The same first minute of all-core load, five ways, on one colour scale, with each panel's own noise. The sixth panel is the arrival map from above.
Noise per picture, all-core run (mK)
one frame (40 ms), one cycle
36
1 s average, one cycle
16
5 cycles averaged, control subtracted
11
+ 1.5-pixel blur
4.9
+ low-rank (SVD)
4.5
Random noise of each picture, from second differences between consecutive 1 s pictures, so a steady rise does not count as noise.

Averaging 25 frames into a second takes 36 mK to 16. Averaging five cycles takes it to 11, not 7, because subtracting the no-load run adds that run's own noise. A 1.5-pixel blur (about 1 mm) halves it again. Low-rank denoising, which keeps only the patterns that stand above pure noise in a singular-value decomposition, adds little here: the heat movie needs over 100 components.

Chart over the 180 s cycle: the camera's own drift with no load stays under about 35 mK for 10 s after the shutter, then jumps to about 140 mK median and 400 mK at the 95th percentile; the all-core, performance-core and GPU rises climb to 450–830 mK median and 1.7–2.6 K at the 95th percentile, peaking around 70–90 s; the efficiency-core rise stays inside the drift band
Each load's rise (median and 95th percentile over the image) against the camera's own drift with no load (grey band). The load is on in the shaded band. All-core heat peaks around 85 s, 25 s after the load stops.

The real limit is not the noise but the camera. After its shutter fires, the camera's own image drifts with no load at all: 25–35 mK in the first 10 s, then about 140 mK (median; about 400 mK in the corners) from 20 s on. Subtracting a no-load run on the same schedule removes the repeatable part. What it cannot remove varies from cycle to cycle by 100–150 mK. So within 10 s of a shutter, changes of a few tens of mK are real. After that, only changes of a few hundred mK are, which the CPU and GPU easily give and the efficiency cores do not.

Off-laptop articulated arm + live AppleSMC silicon & fan telemetry (16 Oct)

Two of the open items from the first campaign — mounting the P3 off the laptop so the camera body does not rock or warm with the screen lid, and logging the M5's internal cooling fans, power rails, and silicon die sensors in step with the 25 Hz thermal stream — were run on Fri 16 Oct 2026. The P3 was clamped to a rigid articulated desk arm angled down across the M5 keyboard and streamed over `/stream.raw.z` (`zlib`-compressed `P3FR` frames, `0 dropped frames` across 11,303 frames) in un-halved 14-bit raw NUC bypass mode (`nuct_bypass=1`, `+38.622 mK` per 14-bit NUC count = `+9.6555 mK` per wire `uint16` LSB — `1.62×` finer quantization than calibrated `1/64 K` mode).

+9.66 mK
wire LSB in 14-bit raw NUC mode (vs 15.63 mK in 1/64 K mode)
11.1 mK
extracted surface noise (from 40.6 mK single-frame raw)
9.15 s vs 13.59 s
half-rise time: aluminium chassis web vs ABS keycap tops
1,342 → 3,656 RPM
AppleSMC twin blower ramp cooling the upper keyboard strip
Live 14-bit raw NUC stream coupled to AppleSMC internal telemetry (`arm-cpu-lockin`, 194 s, 4,851 frames at 25.0 Hz). Left: single 40 ms 14-bit frame (`σ = 40.6 mK`). Middle: extracted surface temperature wave (`σ = 11.1 mK`, scale `±1,439 mK` overlaid on the greyscale structural keyboard image). Right: non-sudo `AppleSMC` silicon die temperatures (`Tp*` CPU max `69.3 → 99.8 °C`, `Tf*` finstack `63.1 → 74.8 °C`), system power (`PSTR` `35.1 → 96.5 W`, heatpipe `PHPC` `18.5 → 60.8 W`), twin cooling fans (`F0Ac`/`F1Ac` `1,342/1,451 → 3,031/3,282 RPM`), and P3 external keyboard zone temperature rises.
Photo of the Thermal Master P3 mounted on a rigid articulated arm looking down at the MacBook Pro M5 keyboard
Physical setup. The Thermal Master P3 is held over the right side of the M5 MacBook Pro by an articulated clamp arm, isolating the camera from screen-hinge vibration and chassis conduction while keeping the G/H/J SoC core, R/T/Y/U upper keys, hinge exhaust, and left palm rest in the field of view.
Why raw 14-bit NUC (`nuct_bypass=1`) matters

In default calibrated mode (`1/64 K`), the P3 ASIC right-shifts the 14-bit NUC array by 1 bit (`NUC >> 1`, halving sensitivity to 8,192 bins) before indexing the `NUC-T` lookup table and left-shifting by 2 (`<< 2`, leaving bits `0..1` zero and creating a `62.5 mK` staircase).

Setting register 0x0126 / 0x0048 = 1 (nuct_bypass=1) bypasses the `>> 1` shift and lookup table, placing the full 14-bit post-NUC radiance count (`<< 2`) directly on the wire. Cross-correlating the warm SoC keyboard core (`u16 ≈ 20,778`) against the cool palm rest (`u16 ≈ 20,100`) confirms that warmer target surfaces produce higher 14-bit NUC counts at +38.622 mK / count (+9.6555 mK / wire uint16 LSB), whereas camera-body FPA warming moves raw counts in the opposite direction.

All three runs streamed lossless `zlib`-compressed raw frames over HTTP (`tools/p3_observe.py`) with 0 dropped frames and 0 hardware `cnt3` gaps.

Six-stage sub-LSB signal extraction (`40.6 mK → 11.1 mK`)

Progressive extraction of the M5 keyboard thermal wave across a 55 s lock-in window (scale `±1,412 mK`). Notice Panel 2 (`1 s Boxcar, No Drift Fix`): without cold-anchor common-mode drift subtraction, post-shutter FPA drift makes the entire left side of the laptop appear to cool by `−600 mK` (blue). Subtracting the real-time median of the unheated palm-rest cold anchor (Panel 3) immediately restores the true physical heating map.
StageTechnique appliedRandom noise σ (`arm-cpu-lockin`)Random noise σ (`arm-gpu-metal`)What it removes
1. Single 14-bit frameRaw `u16 - base_u16` at 25 Hz (`40 ms` integration)40.60 mK44.74 mKStatic spatial fixed-pattern offset (`base_u16`)
2. 1 s boxcar averageAverage 25 consecutive frames per 1.0 s bin (`1/√25`)20.57 mK (+drift)23.38 mK (+drift)High-frequency readout + shot noise (`> 1 Hz`)
3. Cold-anchor drift fixSubtract per-bin median of unheated Palm Rest ROI (`rows 170..235, cols 5..55`)25.85 mK27.73 mKGlobal camera FPA drift (`150–400 mK` post-shutter wander)
4. Multi-cycle lock-inPhase-locked average across N workload cycles (`3×` CPU, `2×` GPU)13.51 mK22.03 mKUncorrelated temporal noise (`1/√N`) and non-synchronous room drafts
5. Spatial Gaussian (`1.5 px`)Reflect-padded 2D Gaussian kernel (`σ = 1.5 px ≈ 1.1 mm`, narrower than a `19 mm` keycap)11.13 mK19.51 mKPixel-to-pixel uncorrelated spatial variance while preserving keycap edges
6. Marchenko–Pastur SVDKeep only singular values exceeding a synthetic noise movie threshold (`k = 55` components)11.09 mK19.59 mKResidual high-rank spatiotemporal noise

Aluminium chassis web vs ABS keycaps, and CPU vs GPU thermal footprints

Three-column comparison of CPU All-Core 14-bit lock-in, GPU Metal 14-bit lock-in, and Sustained 45 s Calibrated Step Wave, with surface temperature rise curves comparing the aluminum chassis web against ABS keycap tops
Spatial footprint and structural conduction across all three runs. Top row: peak extracted surface temperature rise overlaid on the keyboard. Bottom row: time traces for the machined Aluminium Chassis Web (35% coolest baseline pixels in the keyboard grid, i.e. the low-emissivity metal lattice between keys), ABS Keycap Tops (65% warmest baseline pixels), SoC Core (`G/H/J`), and Upper Keys (`R/T/Y/U`).

Segmenting the keyboard grid by baseline apparent radiance cleanly separates the high-emissivity ABS plastic keycaps (ε ≈ 0.92) from the anodized aluminium chassis web between the keys. Under 16-core CPU load (`arm-cpu-lockin`), the aluminium web reaches half its rise in 9.15 s, whereas the ABS keycaps sitting on scissor mechanisms over an air gap lag behind at 13.59 s (`+4.44 s` keycap thermal lag). Under GPU Metal compute (`arm-gpu-metal`), total system power peaks at 110.4 W (`PHPC = 69.2 W`), driving a +2,388 mK (`+2.39 K`) rise on the keycaps and warming the upper keyboard row (`R/T/Y/U`, `+2,039 mK`) right alongside the SoC core (`G/H/J`, `+2,033 mK`).

Internal silicon die (`AppleSMC`) to external surface (`P3`) transfer function

Six-panel transfer function showing AppleSMC system power PSTR, heatpipe power PHPC, cooling fan RPM F0Ac/F1Ac, internal CPU and GPU die temperatures, finstack temperature, and external P3 keyboard surface temperature rise for CPU and GPU runs
Simultaneous internal (`AppleSMC` / `IOHID`) and external (`Thermal Master P3`) telemetry. Top: system power `PSTR` (solid purple), package/heatpipe power `PHPC` (dashed red), and twin cooling fans `F0Ac`/`F1Ac` (dotted green, RPM). Middle: internal `CPU Die Max` (`Tp*`), `GPU Cluster Max` (`Tg*`), and `Finstack/Heatpipe` (`Tf*`). Bottom: drift-corrected external P3 surface temperature rise across four keyboard zones.

Logging `AppleSMC` at 2 Hz (`experiments/sensitivity/mac_sensors.py`, reading 321 SMC keys + 17 `IOHID` PMU sensors with zero `sudo`) answers the question from the first campaign about why part of the upper keyboard cools during CPU runs:

  • Sub-second internal silicon jump vs 9–14 s chassis diffusion delay: When 16-core CPU load starts (`t = 15 s`), `PSTR` jumps from `35 W` to `96.5 W` and internal `CPU Die Max` hits `99.8 °C` within `2.0 s`. External keyboard surface temperature does not begin rising until `t + 3.5 s` and reaches `50%` at `9.15 s` (aluminium web) to `13.59 s` (keycaps), continuing to climb for `15–20 s` after the CPU load turns off.
  • Twin blower fan ramp (`1,342 → 3,656 RPM`) cools the upper keyboard & hinge intake: For the first `15 s` of Cycle 1 (`t = 15..30 s`), the M5's twin fans remain at their idle floor (`1,342 / 1,451 RPM`) while the heatpipe finstack (`Tf*`) warms from `63 °C` to `72 °C`. At `t = 30 s`, SMC ramps the fans linearly to `3,031 / 3,282 RPM` (`3,393 / 3,656 RPM` in `arm-cal-wave`). As forced airflow increases through the upper chassis and hinge exhaust, the Upper Keys (`R/T/Y/U`) actively cool from `+980 mK` down to `+580 mK` between cycles even while the SoC Core (`G/H/J`) ratchets upward to `+1,301 mK`.

Next

  1. Done (16 Oct): Off-laptop articulated arm mount & live `AppleSMC` fan/power/die logging. Mounting the P3 on a rigid desk arm eliminated screen-lid rock, and `experiments/sensitivity/mac_sensors.py` now records `F0Ac`/`F1Ac` fan RPM, `PSTR`/`PHPC` power, and 321 `AppleSMC` temperature keys in step with every P3 recording.
  2. Done (16 Oct): Remote real-time sub-LSB observation client (`tools/p3_observe.py`). Streams 25 Hz `/stream.raw.z` over USB, Wi-Fi/Tailscale, or direct Cellular LTE (`--cell`) or 12 Hz Spacesheep telemetry (`--spacesheep`), applying EMA + cold-anchor common-mode drift subtraction + spatial Gaussian smoothing in real time.
  3. Blackbody reference patch with contact thermometer in the field of view. Adding an electrical-tape-coated aluminium block with a `±0.1 °C` contact sensor (e.g. TMP117) inside the camera's FOV will tie the relative millikelvin wave (`±11 mK`) to an absolute NIST-traceable Kelvin anchor.
  4. Transfer to the AI Foundry lab (`ET-SoC-1` cards). Use the exact `arm_campaign.py` + `p3_observe.py` lock-in pipeline at `60–65 mm` working distance looking at the back of an ET-SoC-1 PCB during inference bursts.
Methods, code and caveats

Recording (`15 Oct` bezel campaign & `16 Oct` articulated-arm campaign). The 15 Oct campaign used calibrated mode (`1/64 K`), automatic shutter off, and a manual shutter 5 s before every load start (`60 s on / 120 s off`, 6 cycles at 25 fps). The 16 Oct articulated-arm campaign (`experiments/sensitivity/arm_campaign.py`) recorded three uninterrupted 25.0 Hz runs over `/stream.raw.z`: arm-cpu-lockin (14-bit raw NUC `nuct_bypass=1`, `25 s on / 30 s off`, 3 cycles, 4,851 frames), arm-gpu-metal (14-bit raw NUC `nuct_bypass=1`, `25 s on / 30 s off`, 2 cycles, 3,476 frames), and arm-cal-wave (16-bit calibrated `1/64 K`, `45 s on / 50 s off`, 2,976 frames), while `experiments/sensitivity/mac_sensors.py` sampled `AppleSMC` (`IOServiceGetMatchingService("AppleSMC")`, selector `2`) and `IOHIDEventSystemClient` at 2 Hz without `sudo`.

Analysis (`flow.py` & `arm_visualize.py`). In 14-bit raw NUC mode (`nuct_bypass=1`), wire `uint16` values are `14-bit_NUC << 2` with target radiance sensitivity `+38.622 mK / count` (`+9.6555 mK / wire LSB`). Frames are averaged into 1.0 s bins, referenced to the pre-workload cold baseline (`base_u16`), and common-mode FPA drift is removed per bin using the median of the unheated palm-rest cold reference ROI (`rows 170..235, cols 5..55`). Phase-locked cycle folding is followed by a `1.5 px` 2D Gaussian blur and Marchenko–Pastur SVD thresholding against a synthetic noise movie of matching spatial/temporal variance.

Remote observation (`tools/p3_observe.py`). Connects to the phone over USB/Wi-Fi/Tailscale (`/stream.raw.z`), direct Cellular LTE (`--cell`), or Spacesheep (`--spacesheep`, decrypting `priv_enc` via `AES-256-GCM` when the operator key is present) to display and log live sub-LSB ROI temperatures (`--csv`) alongside local `AppleSMC` hardware sensors (`--mac-sensors`).

Code and the runs' settings, logs and results are in the repo yaroslavvb/thermal-android, under experiments/sensitivity/ and tools/p3_observe.py; the multi-GB raw `.p3rec` recordings stay on the M5.