Further Thoughts on Motion Tracking

Atronach's Eye may be operating on my wall (going strong after several months!), but I'm still excited to consider upgrades. So when a new motion detection algorithm came to my attention, I decided to implement it and see how it compared to my previous attempts.

MoViD, with the FFT length set to 32, highlighting and detecting a hand I'm waving in front of the camera. The remarkable thing is that I ran this test after dusk, with all the lights in the room turned off. The camera feed is very noisy under these conditions, but the algorithm successfully ignores all that and picks up the real motion.

I learned about the algorithm from a paper presented at this year's GOMACTech conference: "MoViD: Physics-inspired motion detection for satellite image analytics and communication," by MacPhee and Jalai. (I got access to the paper through work. It isn't available online, so far as I can tell, but it is marked for public release, distribution unlimited.) The paper proposes MoViD as a way to compress satellite imagery by picking out changing regions, but it works just as well on normal video. It's also a fairly simple algorithm (if you have any digital signal processing background, otherwise feel free to gloss over the arcane math coming up). Here's the gist:

1. Convert frames from the camera to grayscale. Build a time series of intensity values for each pixel.
2. Take the FFT of each time series, converting it to a frequency spectrum.
3. Multiply by a temporal dispersion operator, H(ω). The purpose is to induce a phase shift that varies with frequency.
4. Take the inverse FFT to convert back to the time domain.
5. You now have a time series of complex numbers at each pixel. Grab the latest frame from this series to analyze and display.
6. Compute the phase of each complex number - now you have a phase value at each pixel. (The paper calls these "phixels." Cute.)
7. Rescale the phase values to match your pixel intensity range.

The result is an output image which paints moving objects in whites and light grays against a dark static background. I can easily take data like this and apply my existing method for locating a "center of motion" (which amounts to calculating the centroid of all highlighted pixels above some intensity threshold).

My main complaint with the paper is its shortage of details about H(ω). It's an exponential of φ(ω), the "spectral phase kernel" ... but the paper never defines an example of the function φ, and "spectral phase kernel" doesn't appear to be a common term that a little googling will explain. After some struggles, I decided to just make something up. How about the simplest function ever, a linear function? Let φ(ω) = kω, with k > 1 so that higher frequencies make φ larger. Done! Amazingly, it worked.

Okay, math over. Let me see if I can give a more conceptual explanation of why this algorithm detects motion. You could say frequency is "how fast something goes up and down over time." When an object moves in a camera's field of view, it makes the brightness of pixels in the camera's output go up and down over time. The faster the object moves, the greater the frequency of change for those pixels will be. The MoViD algorithm is basically an efficient way of calculating the overall quickness of all the patterns of change taking place at each pixel, and highlighting the pixels accordingly.

It may be hard to tell, but this is me, gently tilting my head back and forth for the camera.

My version also ended up behaving a bit like an edge detector (but only for moving edges). See how it outlines the letters and designs on my shirt? That's because change happens more abruptly at visual edges. As I sway from side to side, pixels on the letters' borders abruptly jump between the bright fabric of the shirt and the dark ink of the letters, and back again.

The wonderful thing about this algorithm is that it can be very, very good at rejecting noise. A naive algorithm that only compares the current and previous camera frames, and picks out the pixels that are different, will see "motion" everywhere; there's always a little bit of dancing "snow" overlaid on the image. By compiling data from many frames into the FFT input and looking for periodic changes, MoViD can filter out the brief, random flickers of noise. I ran one test in which I set the camera next to me and held very still ... MoViD showed a quiet black screen, but was still sensitive enough to highlight some wrinkles in my shirt that were rising and falling with my breathing. Incredible.

Now for the big downside: FFTs and iFFTs are computationally expensive, and you have to compute them at every pixel in your image. Atronach's Eye currently runs OpenCV in Python on a Raspberry Pi. Even with the best FFT libraries for Python that I could find, MoViD is slow. To get it to run without lagging the camera input, I had to reduce the FFT length to about 6 ... which negates a lot of the noise rejection benefits.

But there are better ways to do an FFT than with Python. If I were using this on satellite imagery at work, I would be implementing it on an FPGA. An FPGA's huge potential for parallel computing is great for operations that have to be done at every pixel in an image, as well as for FFTs. And most modern FPGAs come with fast multiply-and-add cells that lend themselves to this sort of math. In the right hardware, MoViD could perform very well.

So this is the first time I've ever toyed with the idea of buying an FPGA for a hobby project. There are some fairly inexpensive FPGA boards out there now, but I'd have to run the numbers on whether this much image processing would even fit in one of the cheap little guys - and they still can't beat the price of the eyeball's current brain, a Raspberry Pi 3A . The other option is just porting the code to some faster language (probably C).

Until the next cycle,
Jenny

Inmarsat-6 Satellite Launch!

I've mostly used this blog to talk about my hobbies, but I also have an aerospace industry job -- and it's become so exciting that I have to say something about it, because one of the projects I worked on is finally about to go up. Something of mine is soon to be IN SPACE you guys. (It's possible that one of my earlier projects is already out there ... but it was a military satellite. I wasn't told when or if it was launched, or whether it was successful. So it kinda doesn't count.) Due to a combination of working secret projects like that, and working IRAD projects that never got launched, I have been waiting for this for over nine years.

I don't have any personal photos of the RF box or our labs, and wouldn't be allowed to show you if I did ... but here's one I snagged from the company website. Image credit: SEAKR Engineering

The satellite in question is Inmarsat-6 F1, first of its name (there is an F2). Inmarsat is the operator, and the satellites were manufactured by Airbus ... which subcontracted the design and construction of the RF data processing boxes to my employer, SEAKR Engineering. The I-6 are data carriers, and the RF (radio frequency) processor functions something like a telephone switch, routing streams of data from one frequency to another.

According to Inmarsat, I-6 F1 will be "The world's largest and most sophisticated commercial communications satellite." I never read any marketing copy for this thing when I was part of the team laboring over it back in 2017-2018. It was just my daily work, another program I was assigned to and needed to get to completion. It feels a little surreal to see the language that Inmarsat describes it with now.

Before I say exactly what I did, I need to talk a bit about my field. I'm an Electrical Engineer with a specialty in FPGAs (Field Programmable Gate Arrays). An FPGA is a type of integrated circuit -- a computer chip. But whereas the average chip contains fixed circuitry that was cut and deposited into the layers of material inside, an FPGA contains many independent components (logic lookup tables, flip-flops, small RAMs, etc.) that can be connected by the user, post-manufacturing, to create almost any kind of digital processor. FPGA design consists of inventing the connection pattern for one of these. Imagine building a computer's CPU out of tiny LEGO bricks. Older FPGAs contained antifuses, and the connections were made by placing the FPGA in a programming socket that would permanently burn some of them closed. Newer FPGAs have configuration memory that can be loaded with binary data, and the values in this memory set electronic switches to establish the circuit. Since the memory can be rewritten, the circuits inside such FPGAs can be revised many times.

A visualization of an FPGA interior, with components used by the current design highlighted. Image credit: Xilinx (Figure 2-2 in UG633 v14.5)

FPGAs are valued by electronics designers for their balance between specialization and flexibility. If you need to do some calculations, you could design and order a 100% custom circuit -- an ASIC -- that will run them in the most efficient way possible. But ASICs are very expensive and time-consuming to produce. If you aren't planning to sell very many units (and in the satellite industry, we generally don't), they often aren't worth it. On the other end of the spectrum, you could buy a standard embedded processor and write your own software for it. But such a processor is designed to do a handful of basic math and logic operations; it won't have specialized circuitry to suit your needs, and your calculations might end up being very slow. FPGAs bridge the gap. They are standardized and lack the custom manufacturing costs of an ASIC, but they come to you as clay ready to be shaped. You can turn them into processors that are excellent at doing exactly what you want to do. And their reconfigurability makes them tolerant of design mistakes.

Now I need to say a few words about the bane -- well, one of the banes -- of electronics in space. That would be radiation, in the form of high-energy particles. Examples include cosmic rays (ions that fly in from interstellar space) and protons from the solar wind. These particles hurtle along at such blazing speeds that they can punch straight through an integrated circuit, leaving behind a trail of electric charge in all the wrong places. Then the misplaced charges can do annoying things, like turning transistors on or off, possibly disrupting the IC's function. Generally, the better an IC's performance, the smaller its internal features are and the less charge is needed to disrupt them. These effects are especially problematic in a reconfigurable FPGA ... not only could they change the current state of the circuit, they could strike the configuration memory, thereby rewiring the circuit itself!

Space radiation is such a problem that we test our parts ahead of time to find out how they will malfunction when struck by particles. I spent most of 2018 assisting that effort. This is the business end of the heavy ion beam system we used at the Texas A&M University Cyclotron Institute.

Earth's atmosphere blocks most ionizing radiation, which is a good thing for both ground-level computers and your own sweet cells. In space, shielding a computer is often impractical. The amount of material needed to stop the particles would make a satellite unacceptably large and heavy. An alternative is to design the processor to detect radiation-induced errors and self-correct.

That's where I came in on this project. I was responsible for one side of the two-part error monitor that watches for radiation events in the big, vulnerable FPGAs that are doing all the data processing and routing. I wasn't the first person to work on it, but I was the one who finished it, took it through integration and test, wrung out all the bugs, and answered everyone's questions about it for months afterward. Unfortunately, I can't give any details about exactly how it works, since that would be going deep into trade secret territory. SEAKR is more protective of that error monitor than most of the code we've developed in-house.

After finishing my half of the error monitor, I helped with the performance testing for the high-speed data links between the processing FPGAs. We ran these tests in a thermal chamber, so we could ensure the data wasn't corrupted at the hottest and coldest temperatures we expected the box to suffer in space.

Another photo from TAMU. This is part of the beam system for proton testing.

I did a worst-case analysis report on yet another data interface. A WCA is a set of theoretical calculations that check whether the interface will still work across all possible environmental and internal conditions. For instance, can the electrical signals be expected to always arrive at their destination within the right time window? WCA might be my least favorite part of my job: complex, tedious, and endlessly frustrating. Usually whoever designed the interface pushed it right to the edge of acceptability -- leaving me, the analyst, to consider tiny effects, make strained assumptions about parts that don't come with adequate data, generate sixteen timing diagrams instead of one, and chew on my own fingers. I finally got the horrible report written up to our lead's satisfaction and moved on to another project.

... And then I came back, in late 2020, to solve a nasty little bug that was causing data packets sent to the processor FPGAs to be duplicated or lost under rare circumstances. At this time the program was wrapping up and struggling to get the flight units shipped. I was running low on things to do and the bug had been going neglected, so they threw me at it, even though I had never studied this part of the FGPA code before. As I recall, it took me weeks. (If you've ever chased a bug in software ... FPGA bugs are an order of magnitude worse.) But I found it and fixed it, and some time later we finally got the two RF boxes out the door.

Then I forgot about them ... until about six weeks ago, when I got the company-wide e-mail that the first of the two satellites had been delivered to Japan. It will be launched from JAXA Tanegashima Space Center on a Mitsubishi Heavy Industries rocket, on December 21st as early as 14:33 GMT (assuming all goes well with weather and other contingencies). The launch will be livestreamed. So if you'd like to watch my first major contribution to civilization ascend to the heavens (or gloriously explode? good thing there's a second one!) be at that link the day of.

Until the next cycle,
Jenny