Finding the trails: detecting aircraft and satellite streaks one photo at a time

1What makes a trail hard to find

A star-trail photo is a stack of dozens or hundreds of long exposures, combined so each star's slow drift across the sky draws a smooth arc. Aircraft and satellites cross some of those exposures and leave bright streaks that survive into the finished stack. Star Bridge repairs a streak once it knows the streak's outline. Finding that outline is the job, and it's harder than it sounds.

A trail can be a hairline a few pixels wide, faint against a bright twilight sky or lost among thousands of stars, and it can sit anywhere in the photo. The detector has to find every one, on any camera and lens, in skies from full dark to twilight, and tell a real trail from a star, a hot pixel, or the foreground.

An airplane makes this harder than it looks, because an airplane doesn't draw a line. Its anti-collision lights flash, so it leaves a row of separate bright dots instead of one continuous streak. A satellite, with no flashing lights, draws the thin unbroken pencil stroke you'd expect. Take any one of the airplane's dots on its own and nothing tells it apart from a star, and the sky in Figure 1 is full of stars.

What separates them is the pattern. The plane's dots fall along a single straight path, evenly spaced, heading a way no star is heading. You see that at a glance. Turning it into instructions is another matter. How straight does the path have to be? How evenly spaced? How faint can a dot get before it stops counting? Every answer has exceptions. So instead of telling the detector what a trail looks like, we show it thousands of them and let it work that out for itself.

2Why you cannot detect on the whole photo

At the heart of the detector is a neural network from the YOLO family [2], an open-source object detector built and maintained by Ultralytics. YOLO is one of the most widely used object detectors around, so we could start from a proven tool instead of building our own. We use the segmentation version of it, for two reasons. It works tile by tile, which is the only way to find a trail in a full-size photo. And it finds the trail's exact pixels, which is what we fit the angled rectangle to, and what the repair ultimately fills. A neural network is a program that looks at an image and answers a question about it. Ours answers one question: where are the trails?

It learns by example, and a person checks every example. The detector we already have takes a first pass and outlines what it thinks are trails. Those outlines go into CVAT, an open-source annotation tool we run locally, and someone works through every one: keeping the good outlines, fixing the loose ones, deleting the mistakes, and drawing in the trails it missed. Only what survives that review is used for training. Thousands of trails, every one looked at by a person before the detector ever learned from it.

Two things we learned there, for anyone doing the same. We reviewed in LabelMe at first and moved to CVAT [3] for one reason: CVAT lets you zoom into a region and keeps that zoom while you step from photo to photo. LabelMe doesn't, and that matters enormously when the thing you're checking is three pixels wide and you have to follow it across a hundred photos.

The second thing: CVAT's default image quality of 70 is too low to review by. A faint trail simply isn't there at 70, so you sign off on a photo that still has a trail in it. We upload at 95.

It's a loop. Each detector marks up the photos that train the next one, and the next one is better, so there's less to correct. What's left to correct is exactly what it still gets wrong, and those turn out to be the most useful examples we have.

Some of the photos are ours. Most were shared by other astrophotographers. More than a dozen of them, from Arizona to England to India, sent us their sequences so the detector could learn from more skies and more cameras than we could ever shoot ourselves. The photos here are Greg Meyer's. Star Trail CleanR is free, and it's built on that generosity.

The training itself happens ahead of time, on a rented graphics card, and we do it again whenever enough new photos come in. What ships inside the app is the finished result, a single file holding everything it learned. When you clean a sequence, that file runs on your own computer, not on a server, and your photos never leave it.

The network is only part of the detector. Around it sits the machinery this note is about: the part that cuts each photo into tiles, feeds them in one at a time, puts the answers back together, and throws out the ones that aren't trails.

It was built and trained around one fixed input size, 640×640 pixels, which is the standard for this family of models. Every image you hand it gets resized to that before it looks at anything.

The work the detector does grows with the number of pixels you feed it, and it has to run on whatever machine you happen to own, over every photo in a sequence, often with no graphics card to help. You can train a detector to take a bigger input, and some people do. Even that wouldn't be enough. One built for double the size would still shrink a 6000-pixel photo by nearly five times, and the trail is only three pixels wide to start with.

A modern camera photo is 6000 by 4000 pixels or larger. To feed the whole thing in, it gets shrunk to fit, nearly tenfold in each direction, and a trail three pixels wide becomes a third of a pixel. A third of a pixel is nothing. Whatever the detector does next, the trail is already gone.

The fix keeps the photo at full size and gives the detector less to look at each time. We cut the photo into a grid of 640×640 tiles [1] and run the detector on every tile, so it sees each patch of sky at full resolution. A trail you could barely see in the whole photo is an obvious streak inside its tile.

Tiles overlap by 128 pixels, a fifth of their width, so a trail landing on a boundary is still whole inside at least one tile. At full size the trail keeps its width, its brightness, and its shape, so there's something left to find.

3Putting the tiles back together

A 6000 by 4000 photo comes to ninety-six tiles once they overlap, and a long trail crosses several of them. Each tile gets judged on its own, so one trail comes back as a handful of separate pieces, each measured from its own tile's corner.

Putting them back together takes two steps. The per-tile results go back onto a full-size canvas and get combined, so a trail seen twice inside an overlap just merges. Then the pieces get grouped: fragments that line up end to end, within seven degrees of the same angle, count as one trail instead of several. That grouping matters for everything after, because the repair works one trail at a time and needs to know where each trail starts and ends.

Cutting every photo into tiles has a cost, and the overlap is where you pay it. Overlap is what keeps a trail from being lost on a boundary. It also means two tiles regularly report the same trail, and deciding what to do about that turned out to be far harder than cutting the photo up in the first place.

Getting it wrong made the detector look blind in places where every single tile was working perfectly well. That took months to understand, and it gets a note of its own.

4What the detector draws

Most object detectors answer with a box whose sides run straight up-and-down and side-to-side, and the thing you're looking for is somewhere inside it. For a trail, a box like that holds far more sky than trail. A trail is a long thin line lying at an angle, and a straight-sided box drawn around a diagonal line is mostly empty corners. The airplane in Figure 1 fills only about a third of the box around it. The other two-thirds is sky it never touched.

So the detector does more than point at a box. It finds which pixels are trail, and we wrap those pixels in a rectangle tilted to lie along the trail. That tilted rectangle is the outline everything else uses. It's drawn a little wider than the streak on purpose, so the repair has room to work. The same outline is what a human reviews, and what gets stored as a training label.

5Telling a trail from everything else

Finding bright things is easy. A night sky is full of them. The detector's real work is separating a trail from everything else that's bright, and it happens in stages.

The foreground goes first. If a photo has a mask marking ground, trees, or buildings, those pixels get blacked out before anything else looks at them, and a tile that's entirely foreground gets skipped without ever reaching the detector. What's left is sky.

The detector then gives every detection a confidence between zero and one, and anything below 0.25 gets dropped. What remains gets judged on shape. A trail is long and thin, so a blob has to be at least twice as long as it is wide to be kept. It also has to be big enough. Anything smaller than about a thousand pixels counts as noise. That threshold is set for a 6000 by 4000 photo, and it scales down if you shot smaller.

That last rule would throw away something worth keeping. An airliner's navigation lights are small, separate red flashes, exactly the sort of thing a size filter deletes. So a detection that's distinctly red survives no matter how small it is. The red dashes running down the middle of Figure 1 are there because of that one exception.

6Why we chose the smaller YOLO model

YOLO comes in five sizes, from nano up to extra-large. They all work the same way. A bigger one just holds more of the internal numbers it adjusts as it learns. That gives it more room to pick up the many ways a trail can look, and it makes the file bigger. Our detector is the small one, about 23 megabytes, and it ships inside the app. The bigger sizes are tempting, because that extra room usually buys more accuracy.

That extra room comes at a cost, and it lands on the person using the app. A bigger detector is a bigger download, it uses more memory while it runs, and it takes longer on every photo. Someone cleaning a four-hundred-photo sequence on a laptop feels all three. What they notice is the download size and the wait.

We stay small because ours hasn't run out of room for improvement. Every time we add photos it gets better. When that stops being true, a bigger one will be worth its cost.

7Where it still goes wrong

Three failures show up often enough that anyone using this will meet them.

A rectangle can't follow a curve. A wide-angle lens bends a long trail near the edges of the photo, and a straight rectangle fitted to a curved streak ends up wider than the trail, covering sky on either side of it. The repair then rebuilds sky that never needed rebuilding.

Bright twilight sky breaks the detector. On a photo still glowing from sunset, with a bright trail lying across a bright gradient, the detector sometimes hands back a mask covering most of a tile instead of a thin streak. These are the photos it's seen least of in training, and it shows.

The detector only knows what it's been shown. Nearly every photo it trained on came from a Canon or a Nikon behind a wide-angle lens. It has never seen a Sony, a Fuji, or an Olympus. That's a hole in the data, and it closes one shared sequence at a time.

8What comes next

Two big parts of this story are still to come. The first is the trouble tiling caused, and how long it took to figure out: trails the detector seemed to miss but had really found, and trails it missed only because of where they fell inside a tile. That gets a note of its own. The second is the photos, the labelling, and where the training runs, which shape a detector more than the choice of model does. That gets another.

The next note is the other half of the job. Once the detector has found a trail and drawn its outline, something has to put the sky back where the trail was. Star Bridge is how we repair it.

9References

1. [1] F. C. Akyon, S. O. Altinuc, and A. Temizel, "Slicing Aided Hyper Inference and Fine-tuning for Small Object Detection," 2022 IEEE International Conference on Image Processing (ICIP), pp. 966–970, 2022. The tiled-inference method used to run the detector over overlapping crops of a full-size photo.
2. [2] Ultralytics YOLO (G. Jocher, A. Chaurasia, J. Qiu), 2023. The open-source object detector our entire detection is built on. github.com/ultralytics/ultralytics
3. [3] CVAT, the Computer Vision Annotation Tool. Every trail outline in the training set was reviewed in it before training. github.com/cvat-ai/cvat