A formation of U.S. Air Force F-35 Lightning IIs perform aerial maneuvers during a combat power exercise over Utah Test and Training Range. The service is using cloud computing and agile software development to deliver software to the field more quickly, which will potentially benefit a wide range of programs and weapon systems
DevOps revolutionizes Air Force technology development.
Steven Wert, recently named the U.S. Air Force’s Program Executive Officer Digital, is on a mission to fundamentally alter the service’s processes for developing and fielding software through development operations, commonly known as DevOps. The methodology merges software development and technology operations functions, allowing the two to work more closely to reduce the time needed to create new systems. Working closely with the end users is key to what Wert describes as a “release cadence” of weeks or months instead of years.
“Agile DevOps allows us to rapidly field software changes. Our traditional acquisition approach did not allow that,” Wert explains. “We would typically take years to deliver software. These techniques allow us to do that in weeks or months.”
Wert received a crash course about DevOps in 2017 during a visit with Dell’s Pivotal Software Incorporated, which provides a cloud computing platform, developer tools and its own development methodology. The meeting was hosted by the then-Defense Innovation Unit Experimental, which has since dropped Experimental from its name. “They showed us how modern commercial software is built this way and is supported this way, so we’ve been migrating across my portfolio for the past 18 months, and we have multiple programs now that have implemented much of this practice,” Wert says. “My charter as [Program Executive Officer] PEO Digital is to advance this stuff across the Air Force. The office is to help migrate this approach Air Force-wide.”
Cloud computing plays a key role in the service’s flight toward software delivery speed. Wert’s office works with several cloud computing vendors, including Pivotal, Red Hat and Amazon Web Services. “Cloud gives us additional advantages. When I talk about cloud, I’m talking about the platform that you’re building apps on. If that can be provided as a platform-as-a-service, then we’re not consuming our resources managing information technology infrastructure,” Wert explains. “That enables even faster speed, but for our team, it means we can focus on applications and capability versus managing infrastructure components.”
Agile DevOps is a fundamental change for the Air Force because it largely streamlines multiple processes, including requirements definition, technology development, and test and evaluation. Wert has been “in the acquisition business a long time” and has seen many changes, including a focus on outsourcing “a lot of responsibility” to industry, but he suggests agile DevOps makes a bigger difference than any other change he has witnessed. “There have been dozens of different acquisition reforms, different preferred contract mechanisms, but largely none of that changed how we actually executed a program,” he says. “This agile DevOps fundamentally changes how we do requirements, how we do contracting, how we do test and evaluation, how we deliver capabilities to the user. That’s huge for the Air Force. It’s revolutionary.”
The military’s traditional method for developing, testing and evaluating software made it difficult for the services to keep up with technology advances. “Actually getting that software across the goal line is nearly impossible. We have a mixed history with being able to deliver software products that way,” Wert points out. “Under this process, because you’re delivering incrementally, the risk of [delays in] delivery becomes greatly reduced.”
Operators sometimes wouldn’t accept software that didn’t work perfectly. “Under the old process, if something wasn’t working quite right, they didn’t expect to see you for three to five years, so they didn’t operationally accept the software,” Wert recounts. “Now, you’re working with an operational community who expects to see you with an update in weeks or months.”
The new process also makes it easier to field technology that meets some requirements rather than waiting for a perfect solution before handing solutions to warfighters. “If there are a few key features that the user can use right now, and you can field them in two weeks, that’s a value that you’re not postponing for three to five years,” Wert elaborates. “In some cases, that translates directly to money. If features you’re providing lead to more efficient mission plans and less fuel consumed, for example, then you’re talking money.”
That was the case with the Mobility Air Force Planning System, which is mission planning software. The program, which started in 2012 using a traditional multiyear development approach, encountered the usual delivery risks and delays before the Air Force used the more streamlined DevOps approach. “We worked with the Air Mobility Command and our contractor to propose fielding version one into production despite some deficiencies in order to work directly with the end users and rapidly field fixes and improvements,” Wert reports. “In the last 10 months, we fielded four additional major versions of that software.”
Those 10 months included more than 500 fixes or improvements. During that same period, the software successfully processed over 39,000 flight plans and 3,800 sorties for Air Mobility Command while providing fuel savings through more efficient routing. “I’ve been reluctant to talk about this as a cost savings measure, but for the money that we’re spending, we’re providing current capability. We’re not spending money to give them something three to five years from now,” Wert emphasizes.
The Medusa program is another example of an effort that already has benefitted from the new approach. The program’s goal is to build a system to detect and counter small unmanned aerial systems (UASs) to protect Air Force bases. “While specifics are a bit sensitive, I can say a traditional approach to do that would be to try and prove out and operationally test a complete system that did everything from detecting to defeating a small UAS. Because those are urgent needs though, and because the bases have nothing and not all the technologies are fully mature, we’re fielding a detection capability. We’re not waiting for a fully integrated, complete solution,” Wert reveals.
Additionally, the Integrated Strategic Planning and Analysis Network (ISPAN) also began in the traditional path but transitioned to and through the DevOps methodology. “When we went to a full deployment situation [last fall], there was some discussion about instability problems that surfaced during operational test. However, the largest piece of that software was 15 releases beyond what was tested.”
Indeed, he lists the test and development process as one of the challenges. “We’re rapidly outpacing how we traditionally have done operational testing. By the time an independent operational test is produced, we’re many versions beyond the system that was actually tested, so we’re working with the operational test community to better integrate test into this process,” Wert says.
He implies that the DevOps community has gained a reputation in some quarters for resisting testing. “Agile DevOps is not anti-test. In fact, we test more,” he insists. “But we have to have a way of doing operational test that not only doesn’t slow us down but also informs what we’re doing in the development effort.”
To resolve the issue, the Air Force has established a couple of experimental efforts. “We’re piloting different ways to do this. We’re embedding some developmental testers and operational testers right in our software factory, for one. We’ve never done this before, and we don’t want to define a standard without any experience doing it. We’re actually experimenting,” he reveals.
Wert had his first meeting with his fellow PEOs in November. Rather than explain how the process worked, he focused on “the significance of this movement and how big an impact it could have.” He said he believes other PEOs will establish DevOps software factories of their own.
Additionally, DevOps could affect programs that are not solely about software development. “Nearly every Air Force weapon system is software controlled, so there’s a lot of software opportunity out there. Thinking this way actually influences some of the things we’re doing that are not particularly software per se,” he says.
The approach is not without challenges, including financial management. With the new process, it’s harder to tell if a system is being fielded or sustained, which determines the source of funding. “You might one day be building a new feature that gives them new capability and another day just fixing issues. It blurs the line between development and sustainment. Certainly, we need a consistent rule on that so as more and more organizations adopt this, we don’t have different approaches at different locations,” Wert proposes.
Another major challenge is that the new approach is essentially Air Force-led coding, but the service has no occupational specialty that includes writing software code. Although Wert’s office is training personnel who happen to have coding experience, those service members soon will return to their original careers. “The downside of that is that we have a quickly rotating workforce, and we don’t get to keep these coders very long. We are trying to figure out how to get a dedicated, organic workforce that is stable over time,” he offers.
Wert adds that DevOps could be the solution to a problem he has been interested in for a long time. “Occasionally, we’ll see part of our operational community making changes to systems. Sometimes those people are referred to as combat coders. This agile DevOps approach makes us the combat coders, and that’s a good step,” he says. “Wherever and however we’re delivering software, we want to be building it this way.”