Static analysis proposed for shell programs

Semantics-driven static analysis is being proposed by a group of researchers as way to ensure that Unix, Linux, and macOS shell programs are safe, bug-free, and work as expected. However, the effort faces unique challenges, due to the shell’s “pervasive dynamicity” and “opaque, polyglot commands.”

The researchers from Brown University, Stevens Institute of Technology, Rice University, and UCLA make their case in a newly published paper, “From Ahead-of- to Just-in-Time and Back Again: Static Analysis for Unix Shell Programs.” The authors stress that shell programming is as prevalent as ever but is quite complex due in part to the structure of shell programs, their use of opaque software components, and their complex interactions with the broader environment. Even when being extremely careful, shell developers discover devastating bugs in their programs only at runtime. At best, shell programs going wrong crash the execution of a long-running task; at worst, they silently corrupt the broader execution environment, affecting user data, modifying system files, and rendering entire systems unusable, the paper notes. The paper then asks if shell users could enjoy the benefits of semantics-driven static analysis before their programs’ execution, as offered by most other production languages? These benefits would extend to users of Linux, the BSD operating systems (FreeBSD, OpenBSD, and NetBSD), macOS, and anywhere the shell is used including containers and Windows Subsystem for Linux.

Shell scripting is very common, as the shell remains the glue that holds modern systems together; modern facilities such as continuous integration and continuous delivery (CI/CD) are often written in shell, said paper co-author Nikos Vasilakis, from Brown University, in an emailed response to questions. Other popular environments used for tasks such as  building software, serving machine learning workloads, and provisioning the cloud are all thin wrappers around scripts, Vasilakis added. However, the shell language does not behave like other languages, he said. This leaves both inexperienced and seasoned users making many mistakes, with these mistakes tending to be catastrophic. “And because the shell is an old language, it lacks many of the facilities we’ve come to expect in modern languages,” Vasilakis said. “What’s more, the shell is used to manipulate programs on files on live systems. Mistakes can cause data corruption, service interruption, irreversible data loss, and leakage of sensitive user information.”

Static analysis is a proven technique for knowing things about a program before it runs, according to Vasilakis. “A good static analysis can detect many bugs before they have the chance to bite,” he said. By being semantics-driven, the analysis targets deeper reasoning than, say, a syntactic linter, Vasilakis explained. Several kinds of analyses are envisioned, operating in tandem to tackle intricacies of a complex environment. For example, an effect analysis targets file system interactions while a type system centered around regular types targets interprocess interactions in the pipe-and-filter computations. “The goal is to provide precise error messages before the execution of a program, similar to what you’d expect from a modern programming language,” Vasilakis said.

The hope is that semantic analysis will discover more and deeper bugs by being able to reason deeply about shell scripts, the programs they invoke, the way they interact, and what they do to the file system. The researchers are currently implementing several systems that tackle parts of their vision, Vasilakis said. “We have to build up our stream reasoning engine, a symbolic execution engine targeting effects, a specification language for Unix and Linux commands, and semantic models so that we can be confident that our analysis is correct,” he said. “Several more papers and public tools will be available very soon.”

For now, everyone using the shell should be aware of shellcheck, a syntactic—rather than semantic—static analysis for shell scripts, Vasilaskas said. “Our hope is that a semantic analysis will help discover more and deeper bugs—by being able to reason deeply about shell scripts, the programs they invoke, the way they all interact, and what they do to the file system.” Some of these mistakes are in the same category as what shellcheck can catch, but others, such as finding misuses of the file system or command composition mistakes several commands “away,” will be new, he said.

In the meantime, Vasilakis suggested interested parties use the try tool, which will not catch bugs in advance but will limit the “blast radius” of mistakes.