Star trail photographs are built by stacking hundreds of long exposures. Aircraft and satellites cross those exposures and leave bright dashed streaks that survive into the final image. Star Trail CleanR detects the streaks with a neural network, then repairs each one with a technique we call Star Bridge: it fills the damaged pixels with real sky borrowed from the frames on either side, slid into place to follow the motion of the stars. This note documents the full method, including the parts that failed first: a point tracker that slid off star streaks, a brightness pass that crushed repairs to black, a cleanup that quietly deleted stars, and an edge-of-sequence case found only in a late review. On a real 89-frame sequence, every one of 1,265 repairs was filled with genuine photographed sky, and zero pixels were synthesized.
1The problem
A star trail image is a time-lapse compressed into a single frame. The photographer locks a camera on a tripod and shoots continuously, anywhere from a few minutes to all night: dozens to hundreds of exposures, ten seconds to a few minutes each. Stacking software then keeps, for every pixel position, the brightest value that ever occurred there. Stars sweep slowly across the sky, so each star paints an arc. This brightest-pixel rule is called a lighten-max stack, and it is the standard method in tools such as StarStaX and Startrails.exe [3, 4].
The rule has a side effect. Anything bright that crosses even one frame is permanent. An airliner crossing the field for forty seconds leaves a dashed line of strobes and navigation lights. A satellite leaves a thin unbroken pencil stroke. Over a two-hour session near a Southern California flight path it is normal for dozens of frames to carry trails, and every one of them survives into the finished picture. The traditional remedy is manual: open each affected frame and clone the trail out by hand, or throw out a whole busy stretch of the session and stack only the clean remainder. Abandon the first hour of a six-hour shoot because it was too busy to clean, and the arcs simply come out shorter. With satellite constellations growing quickly, the manual approach is losing.
2Why you cannot just erase
Detection itself is a separate problem, solved upstream by a tiled YOLO segmentation network. The companion note, Finding the trails, describes it. Once a trail is found, the obvious repairs all fail in instructive ways.
Paint it black. Filling the trail pixels with zero is smarter than it sounds. Black is the dimmest possible value, so in a deep lighten-max stack a black patch always loses to the real sky from the other frames: the fill is effectively invisible, and for a long time this was exactly what we shipped. It even let us delete an expensive background-inpainting step, which roughly doubled the pipeline speed. But the trick only works when many frames vote on every pixel. The moment the app grew timelapse and single-frame output, black patches became visible holes. Black only disappears inside the stack; on a single frame it is a hole.
Content-aware fill. Synthesizing plausible sky texture hides the trail, but it erases any real star that sat under the mask and invents structure that was never in the sky. Worse for a star trail: each frame is filled on its own, so the invented patches never line up frame to frame. When the frames are stacked, those mismatches bleed through as mottled patches, and a star arc crossing a filled region breaks up instead of drawing one smooth, continuous trail. For a photograph that people treat as a record of a real night, we consider that disqualifying.
Average or median stacking. A rejection stack (a median, say) automatically drops anything that appears only briefly, which is how deep-sky pipelines remove satellites from a set of aligned exposures. But a star trail is the opposite kind of image: it is lighten-stacked so that each star's motion piles up into an arc, and that motion is the whole point of the picture. A median treats that motion exactly as it treats a satellite. At any given pixel a star passes through in only one or two of the dozens of frames, so the stack votes it away as an outlier, and what is left is bare sky and static foreground with no arcs at all. It removes the one thing a star trail cannot lose.
All three failures point to the same fact: the correct pixels for the damaged region already exist. They were photographed seconds earlier and seconds later, in the neighboring frames, just shifted by a few pixels of star motion. So the job is to fetch the right pixels the neighbor frames already hold.
3The core idea
Star Bridge fills the masked trail pixels of frame N with sky borrowed from frames N−1 and N+1. Between the two neighbors the stars move a small amount, typically 1 to 60 pixels depending on focal length and exposure spacing. If we can measure that motion locally, we can slide each neighbor so its stars land exactly where frame N's stars sit, and the borrowed patch becomes indistinguishable from the sky it replaces. Every filled pixel is a real photographed pixel; stars inside the repair region are real stars in their correct positions.
Long trails complicate this. Each trail the detector marks is one connected shape, a component, and the stars drift by different amounts across a wide frame, so a single component 3,000 pixels long cannot use one motion estimate. Every component is therefore split into segments of at most 500 pixels (measured at a 6,000 by 4,000 reference resolution and scaled with frame size), and each segment gets its own local measurement inside a window padded 120 pixels around it. A short trail is a single segment; a long one becomes several, which is why a sequence has more segments than trails.
4Measuring how far the stars moved
Our first tracker was the classic Lucas-Kanade sparse optical flow [2]: find corner-like features, follow them between frames. It worked on crisp inland skies and failed exactly where repairs matter most. During a multi-minute exposure each star smears into a short streak. A streak has no corner for the tracker to grip, and near a bright twilight horizon Lucas-Kanade would lock onto noise and report motion that was simply wrong. At one diagnosed break it reported a shift of (−10, +26) pixels when the true motion, verified by hand, was (+25, +19). A repair slid by that estimate drags a copy of the sky sideways through the fill region.
The whole repair hinges on one number: how far the stars drifted between the neighbor before and the neighbor after. Get it right and the borrowed sky slides into perfect registration; get it wrong and you smear a copy of the sky across the gap. In place of Lucas-Kanade, the drift is now measured two separate ways and cross-checked, because a single method that is confidently wrong is the worst outcome of all:
- Follow the stars. Find the bright star dots in both neighbor frames and measure how far they moved. Every star in the patch moved by the same amount, so their measurements pile onto one answer while the lone trail streak is outvoted. Even a single clean star gives a reading; at one hard break, this method recovered the shift from just one star.
- Match the whole patch (phase correlation). Instead of picking out individual stars, compare the two patches as a whole and find the single slide that makes them line up, down to a fraction of a pixel, with a score for how sure it is. Because it reads the whole texture rather than points, it still works where there are no clean stars to track, the exact case that defeated Lucas-Kanade. (The Kuglin and Hines frequency-domain method [1], via OpenCV.)
Each method returns an actual shift: a left-right and up-down amount in pixels. They agree when those two amounts land within a few pixels of each other, and that agreement is what makes the number trustworthy, because two unrelated methods rarely land on the same wrong answer. On one hard patch, following the stars gave (+25, +19) and phase correlation gave (+25.6, +18.6), the same drift measured two ways, so the repair slides by it. A shift is also trusted when just one method is strongly sure on its own (a phase score of at least 0.6, or at least four agreeing stars), and any shift outside a plausible range of 1 to 60 pixels is thrown out.
When neither test passes, the repair does not guess. It borrows the same neighbor sky but drops it in without sliding, an unshifted paste. This is safer than it sounds, and the cause of the failure is the reason why: both methods come up empty exactly when there is little to lock onto, which means the patch is nearly featureless sky with few or no stars in it to begin with. In practice these fallbacks cluster low in the frame, near the horizon, where haze and light pollution thin the stars out: in the Figure 1 sequence about two-thirds of them sit in the band just above the foreground. With almost no star structure there, an unshifted paste has almost nothing to misplace; any faint star it holds lands a few pixels off, a small blur far safer than sliding by a wrong amount and smearing the sky sideways. The rule is deliberate: when unsure, do not slide.
On the 89-frame sequence in Figure 1, the trails were split into 1,265 small segments and the shift was measured for each. Where they landed:
- Both methods agreed: 507
- Following the stars alone was enough: 551
- Phase correlation alone was enough: 3
- Too few stars to track and no clear texture to lock onto, so the sky was pasted unshifted: 204
That is 1,061 trusted measurements out of 1,265, or 84 percent. The other 204 (16 percent) fell back to a plain unshifted paste.
5Making the patch invisible
With a trusted shift, both neighbors are warped to meet in the middle: the earlier frame is pushed forward by half the measured motion and the later frame is pulled back by half, so both land on frame N's moment in time. The shipped fill then borrows from one neighbor, the one whose sky color best matches frame N at that spot, rather than averaging both; averaging dims faint stars by half, and keeping them at full brightness matters more than the sub-pixel positional gain.
Color needs its own care. Sky color genuinely changes between frames: twilight fades, light pollution pulses, smoke and airglow drift. Borrowing a patch without correcting for this produced visibly two-toned repairs. Every borrowed pixel is therefore pre-leveled: we measure the median color of a thin collar of sky hugging the trail in frame N and in each neighbor (excluding every masked pixel), and nudge each neighbor's contribution onto frame N's local sky before it is pasted. A final ring-leveling pass then matches the finished patch to the sky immediately beside it, and absorbs the small frame-to-frame drift that remains.
Each neighbor also carries its own trail mask, so contaminated pixels are handled per pixel: where only one neighbor is dirty, the other supplies the fill; where both are dirty (a crossing), the repair reaches out to farther frames for clean sky (Section 9) before falling back to pasting the color-closest raw neighbor, pre-leveled, rather than synthesizing anything.
6Keeping the foreground still
Everything above treats the window as sky. Real trails cross tree trunks, ridgelines, and buildings, and the slide that aligns the stars displaces the foreground by the same amount. The result was a family of artifacts: stepped horizon lines, bright wedges, and dark smudges wherever a repair window touched the ground.
The fix that finally worked judges each pixel by one thing: how far it moved between the frames. The two clean neighbors are compared directly: where they agree within a small tolerance, nothing moved there between the frames, so the scene is still (ground, trunk, deep sky) and the pixel is kept unshifted, exactly in place. Where they differ, something moved (a star), and the slid fill is kept so the star lands correctly. Because the test is motion, it protects light-colored rocks and buildings exactly as well as black silhouettes, with no foreground mask required. One guard rides on top: a small bright blob that the slide adds over a still background is a real star at its correct position and is not overwritten; large bright regions are sky glow dragged over the foreground and are routed back to still.
The routing initially left one small residue: isolated near-black dots where a razor-sharp foreground edge jittered by a pixel between the neighbors. A final cleanup replaces any near-black dot whose local median is much brighter, which erases the speckle and cannot touch real dark regions or stars.
Broad foreground is the easy case for the motion test. A ridge, a trunk, a wall: both neighbors see the same still scene, so the pixel is held in place. Thin detail is harder. A Joshua tree spine is only a few pixels wide, the mask is drawn a little fatter than the streak, and an aircraft trail lasts many frames, so over any given spine the trail is usually crossing in the neighbor frames too. With no clean neighbor to hold still against, the routing fills the spine with sky and erases it along with the trail.
The fix leans on the same fact from a different angle. A spine is the same dark pixel in every frame while the trail is bright and moving, so the darkest value each pixel takes across a few neighbor frames is the spine, with the trail thrown away. Only the pixels clearly darker than the local sky are rebuilt this way; the sky and its stars stay with the slide. The spines come back sharp, no reconstruction required.
7The black rectangle bug
One artifact resisted every explanation for weeks: occasional solid black patches, sharp-edged, near foreground crossings. The natural suspects, in order, were the fill source, the shift measurement, and the masks. We ruled out all three. The cause was the most benign-looking step in the pipeline: the final ring-leveling.
Ring-leveling matches the finished patch to the sky in a thin collar around the trail. When the trail crosses dark foreground, part of that collar is not sky at all; it is trunk. The measured "difference" between patch and collar was then enormous, and the leveling obediently dragged the whole repair down toward black (or, with a bright collar, blew it out white). The step was doing its job perfectly against the wrong reference.
The cure was to cap the leveling with a measurement we already had: the per-neighbor color offsets from Section 5 tell us exactly how much the sky actually drifted between the frames at this spot. The final nudge is now clipped to that measured drift plus a three-level margin. Genuine sky drift passes through untouched; a large demanded correction with near-zero measured drift means the collar is not sky, and the request is refused. The cap adapts by itself to bright twilight and dark night because the measured drift does.
The cap stopped the black, but a quieter version of the same problem was left behind. A collar that is part trunk still reads a little dark, and the leveling was still allowed its small nudge, so a repair beside a trunk or a Joshua tree could come out a couple of shades darker than the sky around it. It was faint on a single frame but showed as a dull band once you looked for it. The real fix is to stop the collar from reading the trunk at all: before it measures, the dark foreground pixels are dropped and only the sky is kept. With a sky-only reference the leveling has nothing dark to pull toward, and the repaired patch lands on the surrounding sky.
8The cleanup that was deleting stars
For months the pipeline carried a plausible-sounding final check: after filling, re-scan the patch for pixels that still look like trail (warm-toned and bright, the color signature of aircraft lights) and overwrite them. The logic has a hole you can see immediately once it is stated: a warm star reads identically. Wherever a trail crossed near a warm-toned star, the cleanup deleted the star that the slide had just placed correctly, leaving hollow notches in bright arcs and holes in single frames.
Before removing it we measured it. Across a full sequence, roughly 98 percent of what the cleanup removed was a correctly placed star; under 2 percent was genuine trail residue, and that residue is already handled by the per-pixel mask logic of Section 5. The cleanup is now disabled. The general lesson cost us enough to state plainly:
9When there is nothing to borrow
The busiest failure is a crossing. Over a flight path, many aircraft and satellites cross the same patch of sky across consecutive frames, so the spot being repaired can carry a trail in both immediate neighbors at once. There is then no clean sky right next door to borrow. It is dense traffic: many aircraft and satellites crossing the same patch across consecutive frames, the normal condition near a busy flight path.
Because the trail is moving, the fix is to reach farther. Where both immediate neighbors are dirty, the repair searches outward, first N±2, then N±3, up to N±4, for the nearest frame that is clean at each pixel. It slides that frame to line the stars up on frame N exactly as the normal bridge does, and borrows the real sky from it. In the crossing above, reaching a single frame further recovers every pixel with genuine photographed sky.
Only when even those farther frames are dirty, or the shift measurement failed, or a frame simply has no usable neighbor, is there truly nothing to borrow. The give-up ladder, in order:
- Paste the color-closest raw neighbor, pre-leveled to the local sky. No slide, so borrowed stars sit a few pixels off their true position, but every pixel is still real photographed sky.
- Paint local sky. With no neighbor at all, the trail streak alone is overwritten with the local sky's median color plus matched grain, and the seam is feathered over three pixels. Compact star-like blobs inside the mask are left untouched. This is the only synthesized fill in the system, and it synthesizes only textureless sky.
- Black survives in the code as a defensive guard that requires a frame with no neighbor and a window so trail-filled that fewer than twenty sky pixels exist to sample a color from. We have never observed it fire on a real photograph, and the run logs now prove the negative: it is instrumented, and it has logged zero occurrences.
The history matters for anyone building a similar system. Black fill was originally the correct choice, for the stack-transparency reason in Section 2, and it was replaced because what the app produced changed: timelapse and single-frame output made individual frames visible for the first time. How you should repair a frame depends on what the result will be used for.
10The first and last frame
A fresh review of a cleaned sequence found two small light notches bitten into a Joshua tree trunk, on the first frame of the set. The run log identified both within minutes: two trail repairs whose fill windows touched the trunk, both on frame zero. The cause was structural. The still-versus-moving routing of Section 6 compares the two neighbors, and the first and last frames of a sequence only have one. On those two frames the routing silently could not run, and the single-neighbor slide nicked any foreground it touched. Because the notch is lighter than the silhouette it damages, the lighten-max stack keeps it: a two-frame defect survives into the final image.
The fix reaches one frame further: for the first frame, frames N+1 and N+2 serve as the comparison pair (for the last frame, N−1 and N−2), and the same still-versus-moving routing runs unchanged. A static trunk is identical in both, so it is kept in place; by the second frame the trail has moved off the spot, so both references are clean where it matters. The fix adds no mask and no new machinery.
Two refinements were needed before the fix held, both worth recording:
- The star-protection guard from Section 6 was rescuing small bright slivers of slid sky that landed on thin branch tips, mistaking them for stars. The guard now only protects bright blobs sitting over sky, since no star is ever in front of an opaque trunk.
- The trail was still crossing near the trunk in the second reference frame, and its detection mask, drawn deliberately fat, overhung the trunk and vetoed the still test exactly where protection was needed. Measurement showed the vetoed pixels agreed in brightness within three levels.
Both cases are cut through by a fact every star trail sequence shares: the camera is on a fixed tripod, so a dark silhouette pixel is static by definition and is kept in place, whatever any mask claims.
11Logging every repair
By then the repair logged every decision it made, so when an anomaly turns up, its cause is easy to find: a search of the log pins it down in minutes. Before the log, the only way in was to run the repair, make an educated guess at the cause, change something, and run again; now the answer is in the data. Each repaired segment records which fill method ran, which measurement tier produced the shift, how many stars agreed, the shift itself, and the brightness corrections applied, with a plain-language legend written into the head of every log so the file explains itself. The same instrumentation settled a question we had been answering from memory: whether the synthetic fills ever actually run. The full distribution from the 89-frame sequence of Figure 1:
| Fill method | Segments | Share |
|---|---|---|
| Borrowed neighbor, slid into place | 1,049 | 82.9% |
| Raw neighbor pasted (measurement declared failure) | 195 | 15.4% |
| Edge-frame variants (first/last frame, one neighbor) | 21 | 1.7% |
| Synthesized sky (give-up rung 2) | 0 | 0% |
| Black (defensive guard) | 0 | 0% |
Every repaired pixel on this sequence, 1,265 segments across 831 detected trail components, came from a real photograph.
12Limitations and open work
No repair is perfect, and this one has known edges. The first two are weak points in the current build; the last is a hard case it handles gracefully rather than solving outright.
- Measurement failures. When the two methods cannot agree on how far the stars moved (the 16 percent from Section 4), the repair plays it safe and pastes the neighbor without sliding. The trail still goes away, but the borrowed stars land a few pixels off from where they belong, so a bright arc can look slightly nudged at that spot. Pushing that 16 percent lower is open work on the star tracking, not the fill, and we do not have a committed approach for it yet.
- Neighbor contamination. The repair borrows sky from the frames on either side and trusts that they are clean. If the detector missed a faint trail in one of those neighbors, the borrow copies that missed trail straight into the repair. We see it on dim satellite passes against near-black sky, where the trail is hardest to detect in the first place. This is a limit of the detector, not of the repair, and it is the strongest reason to keep training the detector on more data.
- Crossings in a busy sky. Over a busy flight path, many trails cross the same patch of sky across consecutive frames, so the spot being repaired can carry a trail in both immediate neighbors at once. The repair reaches outward, up to four frames on either side, for the nearest frame that is clean there and borrows its sky, slid to line the stars up (Section 9). It is beaten only when the sky is so busy that the same strip stays covered four frames deep in both directions; then it falls back to pasting the closest-matching neighbor as it is, so the stars there land slightly off. It degrades gently: the stars land slightly off, the trail is still gone.
13References
- [1] C. D. Kuglin and D. C. Hines, "The Phase Correlation Image Alignment Method," Proceedings of the IEEE International Conference on Cybernetics and Society, pp. 163–165, 1975. The frequency-domain displacement measurement used by the cascade, via OpenCV's phaseCorrelate implementation.
- [2] B. D. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," Proceedings of the 7th International Joint Conference on Artificial Intelligence, pp. 674–679, 1981. The classic point-tracking method we tried first and replaced, because star streaks give it nothing to grip.
- [3] M. Enzweiler, StarStaX. Free star-trail stacking software with lighten blending and gap filling. markus-enzweiler.de/software/starstax
- [4] Startrails.exe. Star-trail stacking software with a lighten-screen-blend mode. startrails.de