But amid all of this modern Earth observing splendor, one truism remains: No sensor can image the past.
Since July 1972, satellites in the Landsat program have been imaging the landmasses of our planet at a particularly useful scale that shows us both natural and manmade changes. Landsat is the only satellite system that has recorded Earth’s land-surface conditions for over four decades. This longevity—this steady, unblinking gaze from space—has made the Landsat data record an essential foundation for global change research.
The National Science and Technology Council has endorsed the value of Landsat data. In a study the council conducted in 2014 about the value of U.S. observation systems to the American public, they ranked the Landsat program as number three, just behind the much better known GPS and weather radar systems.
Knowing the worth of this specialized knowledge, it is hard to imagine, that nearly two-thirds of the Landsat data collected in the three and a half decades between 1972 and 1999 was, until recently, completely inaccessible. But so it was, until things started to change in 2010.
Expanding history
With Landsats 7 and 8 now active, it’s no surprise that the archive of current U.S. Earth observation data is growing rapidly. Combined, the satellites collect about 1,200 scenes that take up about 1 terabyte of data every day.
What is surprising is that you can now find much, much more historical Landsat data than ever before in the Landsat archive at the U.S. Geological Survey’s Earth Resources Observation and Science Center, or EROS, in Sioux Falls, South Dakota. This is because a massive international collaboration has been afoot allowing EROS to, in essence, image the past.
The Landsat record has added data acquired between 1972 and 2010 that were previously inaccessible. While the newly recovered information from around the globe doesn’t necessarily change history, it expands the record to sharpen our perspective of natural and human changes on the land, especially in some remote areas.
To explain, let’s imagine some rough parallels in records of American history. If several new contemporary accounts of the Plymouth Colony had been found or if a new trove of letters written by George Washington had been uncovered, the new information wouldn’t necessarily change history, but it would certainly strengthen our confidence in understanding the historical conditions and context of those settings.
Now, about this historical Landsat data: Why was it inaccessible? How was it recovered and re-assembled?
Data from space: more difficult than it appears
The latest incarnation in the Landsat series, Landsat 8 (launched in February 2013), operates with all of the advantages of forty-plus years of technological advancement. Every image Landsat 8 collects makes its way directly into the U.S archive at EROS where the data are processed and immediately made available for anyone in the world with an internet connection to download at no charge.
The design life of the recorders was carefully calculated so that they could obtain “near-global landmass coverage” according to the satellites’ project plan. U.S. ground stations could downlink real-time data collected in their line of sight; so a multitude of images were acquired of the continental U.S. But data for areas outside the U.S. that were not within light of sight had to be recorded onboard and later downlinked to U.S. ground stations.
None of the six recorders reached their expected design lives. These massive recorders with many moving parts and large power demands were ill-suited for orbit. So “near-global landmass coverage” downlinked to U.S. ground stations became impossible.
The next generation of Landsat satellites, deliberately avoided onboard recorders, opting instead for the new tracking and data relay satellite system (TDRSS) technology. For various reasons, the system was not fully operational until 1989. By then a private operator was running the satellite mission, and commercial reasons lead to gaps in international data coverage.
There was, however, a saving grace for worldwide historical Landsat data. As part of the U.S. effort to foster goodwill, share data, and encourage peaceful use of outer space, the U.S. offered other countries the opportunity to build International Ground Stations and operate them as International Cooperators, or ICs. The ICs installed suitable antennae, paid a fee to receive and distribute data, and set up their own data archives.
The U.S. provided technical specifications and engineering support to the ICs and initiated regular meetings. The first ground station operators meeting took place at Johnson Space Center in Texas in 1975. Regular meetings have continued ever since, and the list of ICs has evolved into an international community.
From the very beginning, year-after-year, ICs collected and archived their local data. Year-after-year, more ICs joined the community.
Unlocking vintage Landsat views
During the long life of the Landsat program, its earlier data were often too costly and too computationally expensive to allow global studies to be conducted. The full potential of the chronological span of the data had been locked away, as if it were in ice waiting for technology and policy to let it thaw.
That started to change in the early 2000s when a company known as Earthsat proposed a NASA Scientific Data Purchase to create global terrain-corrected Landsat mosaics for three time periods: the 1970s, circa 1990, and circa 2000. The project was dubbed GeoCover.
To create a global mosaic for time periods prior to 1999, data had to be purchased from ICs to fill in the sizeable gaps in the U.S. global archive. Writing about the GeoCover project in 2004, veteran remote sensing scientist Compton Tucker wrote, “Many receiving stations are the best, and frequently the only, source of Landsat data from their reception regions.”
Even with purchases of Landsat data from ICs, finding enough cloud-free data to make a global land mosaic required EarthSat to utilize data from wide timespans. For example, the circa 1990 GeoCover mosaic ended up using data collected over six years—between 1987 and 1993.
At the time, the GeoCover creators found many historic Landsat images unavailable because the magnetic media was degraded. Tucker warned, “This is a continuing problem which must be addressed in a systematic fashion to avoid additional losses of irreplaceable satellite data.”
“An indispensable history of land-surface state” was at risk, according to Tucker.
Gather glimpses and a fuller picture emerges
This warning did not go unheeded. An idea started to emerge.
Nearly all Landsat data in IC archives was unique and not in the U.S. archive, while USGS had the proven know-how to salvage degraded media. USGS Flight System Manager, Steve Covington, started to ask the central question, “Can USGS step up, save the degrading data and merge the entire IC trove of data into its repository?”
Covington’s briefing to the Landsat Science Team about a follow-on to the GeoCover project (the Mid-Decadal Global Land Survey) was the spark that lit a fire. Soon afterward, the National Satellite Land Remote Sensing Data Archive advisory group initiated a study to find out just how many unique images might exist. Their results were published in 2006.
With this information in hand and a keen understanding of the importance of this historical data, the Landsat Science Team formally requested that USGS consolidate the global archive. In 2010, the Landsat Global Archive Consolidation project, known by the unmelodious acronym LGAC, was born.
“Once Steve Covington initiated the idea, the Landsat Science Team really did put the muscle behind this project,” USGS Landsat International Ground Station Network Manager, Steve Labahn related.
However, the request came at an inopportune time. USGS had recently made the monumental decision to make Landsat data downloads free. Furthermore, system engineers were busy working on the new Landsat 8 ground system. Allocating additional resources for a sizeable project like LGAC was a daunting challenge.
“Timing was tricky. No cost data meant income was going way, while we were being asked to take on more work,” Labahn explained. “But nonetheless, it was exciting to think about how deep and complete the archive could become.”
Next, the work turned to enlisting the help of the ICs. Steve Labahn was tasked with establishing formal agreements with each of them.
“Right out of the gate fifty-percent were onboard and the next 25-percent were easily convinced,” Labahn said. “The remaining ICs were more difficult. Some ICs had fallen off the network or had business models that relied on Landsat data sales income.”
At the twice-yearly IC meetings, the number of ICs supporting the LGAC initiative started to grow. The ICs who had not yet signed on started to realize the historical significance of the project for both themselves as well as the global Landsat user community.
Some 35 international ground stations operated by 22 different ICs had collected data during the last 44 years—all had now agreed to participate in LGAC.
The logistics of making it happen
That the ICs had preserved so much historical Landsat data demonstrated a basic, mutual understanding of Landsat’s significance as an historical record of Earth’s surface. The condition of the decades-old data varied from station to station. Some stations had independently transferred their data to modern media.
“Some stations had everything digitally available and could promptly send it in electronic records to EROS,” Jerad Shaw, a contract System Engineer with USGS, explained.
Other stations that were not able to migrate their legacy data to modern media sent EROS their old data tapes. In some cases the old tapes were badly deteriorated, especially the High-Density Digital Tapes that were used early in the Landsat program.
After receiving the data from the ICs, USGS had to move the data into their archive. The challenges fell into two main camps: (1) reading the data off of the physical media, and (2) translating the formatted data into a form that could be ingested and processed into a standard data product.
None of this was easy.
“The entire process was been a learning experience.” Shaw shared.
Recipe for recovery
When USGS received the deteriorating physical media from ICs, the data engineers had to deal with dirt, mold, and physically degraded, or “sticky,” tapes and outdated tape-reader devices.
Fortunately, since USGS had previous experience salvaging its own historic data the specialists knew what to do. All tapes are cleaned with cotton swabs and alcohol to deal with dirt. Baking the tapes gets rid of any excess water that can make them sticky (see side bar: BASIC BAKED TAPE RECIPE). To read the media, they had to get old tape-readers back in to working order. In some cases, ICs sent their old tape-readers and spare parts to help EROS experts coax the old machines back to life.
It was not a quick process.
“Due to the condition of the tapes, we needed to go through a complex procedure to read the data off those tapes. The manual amount of effort needed per tape is quite a time consuming process,” Shaw described.
The time it takes?
“To run through the process of baking, cleaning and ingesting data off of the oldest tapes. we are doing about two and a half tapes a day,” Shaw said.
Despite the USGS team’s expertise, one thing threw even them for a loop—some tapes showed up with mold on them. Mold samples had to be sent out for testing to make sure they were not harmful. Fortunately none were, but research is still ongoing to figure out the best way to remove the mold.
“We’ve learned a lot in this process,” Shaw shared. “There’s data we’re fully reading now, that when we first looked over the initial tapes we were concerned that we’d even have fifty-percent success.”
Data sleuthing
Once the data is read and in an electronic form, it’s almost ready to process. Almost.
There were no data format standards prior to Landsat 7 (launched in 1999); ICs could then store their data in whatever format made most sense for their processing chains. Consequently, most of the data USGS received was in some sort of IC-specific format. To return the data to an unformatted state, EROS has had to figure out how the IC formatting worked.
“Basically, it is some sleuth work that our software engineers have to do to find commonalities with our data formats and work from there to develop methods to convert the data into a usable format,” Shaw explained.
The ICs provided useful documentation about their data formats, giving the USGS software engineers a starting point from which to create data converters.
Better together: A complete archive
As of September 30, 2016, the U.S. archive contained 6,991,828 images—3.9 million of those are historic images ingested thanks to LGAC efforts. To put that into perspective, Landsat 5, a Guinness World Record holder for its 29-year life span, collected 2.5 million images.
Certainly, the LGAC effort to “image the past” has taken determination and dedication. As Landsat Science Team member Mike Wulder wrote, “The recovery rate has been impressive as a result of the skill and ingenuity of the technical experts at EROS.”
“The ICs have all significantly contributed in a really fundamental way,” Labahn at EROS was quick to point out. It is thanks in large part to the camaraderie and respect that has been fostered over the decades at IC meetings that the LGAC project was even possible.
“If they didn’t know us from Adam, I don’t know how we would have pulled this off,” Labahn maintained. “Because we were meeting on a regular basis, we knew each other, and we could have open and frank conversations to make real progress. The project is a triumph for us all.”
Global-scale investigations for a four decade-plus period are now possible because of LGAC. An additional two million LGAC images are still waiting to be ingested.
“The results have been very good as far as our ability to read data off of these tapes and archive it,” Shaw—an unsung hero of this process—admitted. “There’s some data we’ve had to set aside, but we haven’t given up on anything.”
Labahn followed, “We’re shooting for 100%.”
Article by Laura E.P. Rocchio, NASA Landsat Communication and Public Engagement team and Jon Campbell, U.S. Geological Survey. Today, Oct. 11, we salute AmericaView and Earth Observation Day.
References:
Related Reading:
+ Imaging the Past: Expanding the Historical Record of Landsat Imagery; USGS
+ Landsat Global Archive Consolidation (LGAC); USGS Landsat
The Remote Sensing of Environment special issue, “Landsat 8 Science Results” is now available.
