Recently I wrote a post on how to write a CLI application using Rust. This is a follow up post exploring how to test CLI applications and integrating tests with Cargo, Rust's build tool.

What

Cargo has builtin support for running both unit and integration tests. It also generates a tests module template when writing a library. This template makes it quick to get up and running with writing unit tests in Rust.

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}

The #[cfg(test)] annotation on the tests module tells Rust to compile and run the test code only when we run cargo test, and not when we run cargo build. This saves compile time when we only want to build the library, and saves space in the resulting compiled artifact since the tests are not included.

Unit tests (tests that test each unit of code in isolation from the rest of the code) require little effort to write and reason about since they are very specific. A quick example for a unit test would to be to test if a function returns the correct value.

// recursive
fn recursive_factorial(n: i64) -> i64 {
    match n {
        0 => 1,
        _ => n * recursive_factorial(n-1),
    }
}

// iterative
fn iterative_factorial(n: i64) -> i64 {
    (1..n+1).fold(1, |acc, x| acc * x)
}

fn main() {
    println!("10!: {}", recursive_factorial(10));
    println!("14!: {}", iterative_factorial(14));
}


#[test]
fn test_recursive_factorial() {
    assert_eq!(recursive_factorial(4), 24);
}

#[test]
fn test_iterative_factorial() {
    assert_eq!(iterative_factorial(4), 24);
}

Integration tests on the other hand are meant to test that many parts of your application work correctly together, and for this reason they require more effort writing. Units of code that work correctly by themselves could have issues when integrated, this makes integration tests important. More on this later.

Why

“If we want to be serious about quality, it is time to get tired of finding bugs and start preventing their happening in the first place.”— Alan Page

Some reasons for writing tests would be:

• Improved Quality of Software
• Finding bugs and defects.
• Ensuring a consistent User Experience, by preventing regression.

How

As I mentioned earlier, Unit tests are quite easy to writing as compare to integration tests. I will therefore outline how to write Integration tests for Rust CLI applications and give two approaches.

CLI applications are normally invoked as commands on terminal application such as iTerm or cmd.exe. Testing CLI app functionality then requires a terminal environment emulation. In Rust we can achieve this by calling our executable via the std::process::Command interface.

This allows us to gain access to our executable's:

• environment variable mapping
• stdin
• stdout
• stderr
• exit status

We can then use these to assert that our application behaves as expected when executed.

We normally write our integration tests in a top level tests directory, next to the src directory. Cargo knows to look for integration test files in this directory.

Below is an example test of an integration tests (for my rget project) using the std::process::Command interface:

// filename: tests/integration.rs

use std::process::Command;

static WITHOUT_ARGS_OUTPUT: &'static str = "error: The following required arguments were not provided:
    <URL>
USAGE:
    rget [FLAGS] [OPTIONS] <URL>
For more information try --help
";

static INVALID_URL_OUTPUT: &'static str = "Got error: failed to lookup address information:";
 
mod integration {
    use Command;
    use WITHOUT_ARGS_OUTPUT;
    use INVALID_URL_OUTPUT;

    #[test]
    fn calling_rget_without_args() {
        let output = Command::new("./target/debug/rget")
            .output()
            .expect("failed to execute process");
    
        assert_eq!(String::from_utf8_lossy(&output.stderr), WITHOUT_ARGS_OUTPUT);
    }
    
    #[test]
    fn calling_rget_with_invalid_url() {
        let output = Command::new("./target/debug/rget")
            .arg("wwww.shouldnotwork.com")
            .output()
            .expect("failed to execute process");
    
        assert!(String::from_utf8_lossy(&output.stderr).contains(INVALID_URL_OUTPUT));
    }
}

The std::process::Command is quite effective at writing our integration tests but becomes quite repetitive when writing many tests and/or writing tests for different OS platforms since the executables differ based on the platform.

Here, the assert_cli crate comes to our rescue. assert_cli allows us to write out integration tests without worrying about the std::process::Command interface by abstracting it away for us.

It also provides an intuitive test-oriented interface that makes it easy to reason and write integration tests.

When we re-write our example integration tests using assert_cli, they become:

// filename: tests/integration.rs

extern crate assert_cli;

static INVALID_URL_OUTPUT: &'static str = "Got error: failed to lookup address information:";

mod integration {
    use assert_cli;
    use INVALID_URL_OUTPUT;

    #[test]
    fn calling_rget_without_args() {
        assert_cli::Assert::main_binary()
            .fails()
            .and()
            .prints_error("error: The following required arguments were not provided:")
            .unwrap();

    }

    #[test]
    fn calling_rget_with_invalid_url() {
        assert_cli::Assert::main_binary()
            .with_args(&["wwww.shouldnotwork.com"])
            .fails()
            .and()
            .prints_error(INVALID_URL_OUTPUT)
            .unwrap();
    }
}

Voila, assert_cli becomes useful in writing integration tests for CLI applications. The crate's goal is to provide you some very easy tools to test your CLI applications. It can execute child processes and validate their exit status as well as stdout and stderr output against your assertions. For more examples and usages, check out its documentation

Recap

Testing is an important component of Software development. Rust provides support for writing both unit tests and integration tests and Cargo automatically runs tests for us.

Integration tests are more complex to write compared to unit tests, moreso for CLI applications that require the emulation of a terminal environment.

Rust provides std::process::Command that makes it simple to call an executable and be able to monitor it's environment, input, output and status.

The assert_cli crate makes it easy to write integration tests by abstracting rust internals and providing a testing oriented interface that makes it easy to read and reason about our tests.