For years, dedicated Star Trek fans have been using AI in an attempt to make a version of the acclaimed series Deep Space 9 that looks decent on modern TVs. It sounds a bit ridiculous, but I was surprised to find that it’s actually quite good — certainly good enough that media companies ought to pay attention (instead of just sending me copyright strikes).

I was inspired earlier this year to watch the show, a fan favorite that I occasionally saw on TV when it aired but never really thought twice about. After seeing Star Trek: The Next Generation’s revelatory remaster, I felt I ought to revisit its less galaxy-trotting, more ensemble-focused sibling. Perhaps, I thought, it was in the middle of an extensive remastering process as well. Nope!

Sadly, I was to find out that, although the TNG remaster was a huge triumph technically, the timing coincided with the rise of streaming services, meaning the expensive Blu-ray set sold poorly. The process cost more than $10 million, and if it didn’t pay off for the franchise’s most reliably popular series, there’s no way the powers that be do it again for DS9, well-loved but far less bankable.

What this means is that if you want to watch DS9 (or Voyager for that matter), you have to watch it more or less at the quality in which it was broadcast back in the ’90s. Like TNG, it was shot on film but converted to video tape at approximately 480p resolution. And although the DVDs provided better image quality than the broadcasts (due to things like pulldown and color depth) they were still, ultimately, limited by the format in which the show was finished.

For TNG, they went back to the original negatives and basically re-edited the entire show, redoing effects and compositing, involving great cost and effort. Perhaps that may happen in the 25th century for DS9, but at present there are no plans, and even if they announced it tomorrow, years would pass before it came out.

So: As a would-be DS9 watcher, spoiled by the gorgeous TNG rescan, and who dislikes the idea of a shabby NTSC broadcast image being shown on my lovely 4K screen, where does that leave me? As it turns out: not alone.

To boldly upscale…

For years, fans of shows and movies left behind by the HD train have worked surreptitiously to find and distribute better versions than what is made officially available. The most famous example is the original Star Wars trilogy, which was irreversibly compromised by George Lucas during the official remaster process, leading fans to find alternative sources for certain scenes: laserdiscs, limited editions, promotional media, forgotten archival reels and so on. These totally unofficial editions are a constant work in progress, and in recent years have begun to implement new AI-based tools as well.

These tools are largely about intelligent upscaling and denoising, the latter of which is of more concern in the Star Wars world, where some of the original film footage is incredibly grainy or degraded. But you might think that upscaling, making an image bigger, is a relatively simple process — why get AI involved?

Certainly there are simple ways to upscale, or convert a video’s resolution to a higher one. This is done automatically when you have a 720p signal going to a 4K TV, for instance. The 1280×720 resolution image doesn’t appear all tiny in the center of the 3840×2160 display — it gets stretched by a factor of 3 in each direction so that it fits the screen; but while the image appears bigger, it’s still 720p in resolution and detail.

A simple, fast algorithm like bilinear filtering makes a smaller image palatable on a big screen even when it is not an exact 2x or 3x stretch, and there are some scaling methods that work better with some media (for instance animation, or pixel art). But overall you might fairly conclude that there isn’t much to be gained by a more intensive process.

And that’s true to an extent, until you start down the nearly bottomless rabbit hole of creating an improved upscaling process that actually adds detail. But how can you “add” detail that the image doesn’t already contain? Well, it does contain it — or rather, imply it.

Here’s a very simple example. Imagine a old TV showing an image of a green circle on a background that fades from blue to red (I used this CRT filter for a basic mockup).

You can see it’s a circle, of course, but if you were to look closely it’s actually quite fuzzy where the circle and background meet, right, and stepped in the color gradient? It’s limited by the resolution and by the video codec and broadcast method, not to mention the sub-pixel layout and phosphors of an old TV.

But if I asked you to recreate that image in high resolution and color, you could actually do so with better quality than you’d ever seen it, crisper and with smoother colors. How? Because there is more information implicit in the image than simply what you see. If you’re reasonably sure what was there before those details were lost when it was encoded, you can put them back, like so:

There’s a lot more detail carried in the image that just isn’t obviously visible — so really, we aren’t adding but recovering it. In this example I’ve made the change extreme for effect (it’s rather jarring, in fact), but in photographic imagery it’s usually much less stark.

Intelligent embiggening

The above is a very simple example of recovering detail, and it’s actually something that’s been done systematically for years in restoration efforts across numerous fields, digital and analog. But while you can see it’s possible to create an image with more apparent detail than the original, you also see that it’s only possible because of a certain level of understanding or intelligence about that image. A simple mathematical formula can’t do it. Fortunately, we are well beyond the days when a simple mathematical formula is our only means to improve image quality.

From open source tools to branded ones from Adobe and Nvidia, upscaling software has become much more mainstream as graphics cards capable of doing the complex calculations necessary to do them have proliferated. The need to gracefully upgrade a clip or screenshot from low resolution to high is commonplace these days across dozens of industries and contexts.

Video effects suites now incorporate complex image analysis and context-sensitive algorithms, ...