Static analysis proposed to ‘rehabilitate’ shell programs

Semantics-driven static analysis is being proposed by a group of researchers as way to ensure that Unix, Linux, and macOS shell programs are safe, bug-free, and work as expected. However, the effort faces unique challenges, due to the shell’s “pervasive dynamicity” and “opaque, polyglot commands,” the researchers wrote.

The researchers from Brown University, Stevens Institute of Technology, Rice University, and UCLA make their case in a newly published paper, “From Ahead-of- to Just-in-Time and Back Again: Static Analysis for Unix Shell Programs.” Six of the authors—Nikos Vasilakis (Brown), Michael Greenberg (Stevens), Evangelos Lamprou (Brown), Lukas Lazarek (Brown), Konstantinos Mamouras (Rice), and Konstantinos Kallas (UCLA)—shed additional light on the research in an emailed response to questions.

In the paper, the authors stress that shell programming is as prevalent as ever but is quite complex due in part to the structure of shell programs, their use of opaque software components, and their complex interactions with the broader environment. Even when being extremely careful, shell programmers discover devastating bugs in their programs only at runtime. At best, shell programs that go wrong crash the execution of a long-running task; at worst, they silently corrupt the broader execution environment, affecting user data, modifying system files, and rendering entire systems unusable, the paper notes. The paper then asks if shell users could enjoy the benefits of semantics-driven static analysis before their programs’ execution, as offered by most other production languages? These benefits would extend to users of Linux, the BSD operating systems (FreeBSD, OpenBSD, and NetBSD), macOS, and anywhere else the shell is used including containers and Windows Subsystem for Linux.

Shell scripting is very common, as the shell remains the glue that holds modern systems together. For example, modern facilities such as continuous integration and continuous delivery (CI/CD) are often written in shell, wrote the researchers in their email. Other popular environments used for tasks such as  building software, serving machine learning workloads, and provisioning the cloud are all thin wrappers around scripts, they added. However, the shell language does not behave like other languages, they said. This leaves both inexperienced and seasoned users making many mistakes, with these mistakes tending to be catastrophic. “And because the shell is an old language, it lacks many of the facilities we’ve come to expect in modern languages,” the researchers said. “What’s more, the shell is used to manipulate programs on files on live systems. Mistakes can cause data corruption, service interruption, irreversible data loss, and leakage of sensitive user information.”

Static analysis is a proven technique for knowing things about a program before it runs, according to the researchers. “A good static analysis can detect many bugs before they have the chance to bite,” they said. By being semantics-driven, the analysis targets deeper reasoning than, say, a syntactic linter, they explained. Several kinds of analyses are envisioned, operating in tandem to tackle intricacies of a complex environment. For example, an effect analysis targets file system interactions while a type system centered around regular types targets interprocess interactions in the pipe-and-filter computations. “The goal is to provide precise error messages before the execution of a program, similar to what you’d expect from a modern programming language,” the researchers said.

The hope is that semantic analysis will discover more and deeper bugs by being able to reason deeply about shell scripts, the programs they invoke, the way they interact, and what they do to the file system. The researchers are currently implementing several systems that tackle parts of their vision. “We have to build up our stream reasoning engine, a symbolic execution engine targeting effects, a specification language for Unix and Linux commands, and semantic models so that we can be confident that our analysis is correct,” they said. “Several more papers and public tools will be available very soon.”

For now, everyone using the shell should be aware of shellcheck, a syntactic—rather than semantic—static analysis for shell scripts, the researchers said. “Our hope is that a semantic analysis will help discover more and deeper bugs—by being able to reason deeply about shell scripts, the programs they invoke, the way they all interact, and what they do to the file system.” Some of these mistakes are in the same category as what shellcheck can catch, but others, such as finding misuses of the file system or command composition mistakes several commands “away,” will be new, they said.

In the meantime, the researchers suggested that interested parties use the try tool, which will not catch bugs in advance but will limit the “blast radius” of mistakes.