5.5. Testing

Most students try to solve the complicated programs for this course by implementing them, and then manually testing their program. This is, possible…., but incredibly challenging. Proper unit and integration tests will, on the other hand, save you an enormous number of hours in this course.

5.5.1. Unit Tests

As you develop your program, you are going to write many individual functions. For each, think about how to write simple tests to see if the function does what you expect it to. For example, one of the assignments we often hand out is a user level thread scheduler. To get it to work, you need to set up the stack of the thread properly so that when the thread completes it calls a routine to exit. If you write a simple test at the beginning to test what happens after a thread completes, it will take a few minutes to identify and fix any bugs. Many students, instead, spend hours trying to identify the problem manually when they have a running scheduler, with timer interrupts causing threads to switch between themselves.

Lets see some simple examples, we have already shown a script that runs a set of test programs (Listing 5.1), and a makefile (Listing 5.2)to invoke that script on a parser. The first step on developing a parser is to define its interface, and a set of tests against that interface. To do this, we define the data structures and functions that the parser will expose to its clients in a header file shown in Listing 5.3.

Listing 5.3 Header file for parser for shell

#ifndef MYSHELL_PARSER_H
#define MYSHELL_PARSER_H
#include <stdbool.h>

#define MAX_LINE_LENGTH 512
#define MAX_ARGV_LENGTH (MAX_LINE_LENGTH / 2 + 1)

struct pipeline_command {
  char *command_args[MAX_ARGV_LENGTH]; // arg[0] is command, rest are arguments
  char *redirect_in_path;  // NULL or Name of file to redirect in from 
  char *redirect_out_path; // NULL or Name of a file to redirect out to
  struct pipeline_command *next; // next command in the pipeline. NULL if done 
};

struct pipeline {
  struct pipeline_command *commands; // first command
  bool is_background; // TRUE if should execue in background
};

void pipeline_free(struct pipeline *pipeline);

struct pipeline *pipeline_build(const char *command_line);

#endif /* MYSHELL_PARSER_H */

In this header file, we define everything that test programs, and eventually the shell will need to use to call the parser we will develop. The routines pipeline_build returns a pipeline struct with a flag that indicates if the whole pipleline is in the background, and points to a linked list of pipeline_commands. Each pipeline_command contains a boolean indicates if the command is in the background, and an array where the first element is the command and the other elements are arguments to that command. Our first implementation of myshell_parser.c (see Listing 5.4) simply returns error messages.

Listing 5.4 Initial implementation of

#include "myshell_parser.h"
#include "stddef.h"

struct pipeline *pipeline_build(const char *command_line)
{
	// TODO: Implement this function
	return NULL;
}

void pipeline_free(struct pipeline *pipeline)
{
	// TODO: Implement this function
}

Before implementing any functionality we write tests on what we expect a correct implementation to do. For example, Listing 5.5 shows a test that calls the parser pipeline_build with a single command ls, asserts what it expects the state of a correct execution of the parser is.

Listing 5.5 Test of a simple

#include "myshell_parser.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int
main(void)
{
  struct pipeline* my_pipeline = pipeline_build("ls\n");
  
  // Test that a pipeline was returned
  assert(my_pipeline != NULL);
  assert(!my_pipeline->is_background);
  assert(my_pipeline->commands != NULL);
  
  // Test the parsed args
  assert(strcmp("ls", my_pipeline->commands->command_args[0]) == 0);
  assert(my_pipeline->commands->command_args[1] == NULL);
  
  // Test the redirect state
  assert(my_pipeline->commands->redirect_in_path == NULL);
  assert(my_pipeline->commands->redirect_out_path == NULL);
  
  // Test that there is only one parsed command in the pipeline
  assert(my_pipeline->commands->next == NULL);
  
  pipeline_free(my_pipeline);
}

As another example, Listing 5.6 runs a simple example of two commands with a pipe between them.

Listing 5.6 Test of a simple pipe

#include "myshell_parser.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int
main(void)
{
  struct pipeline* my_pipeline = pipeline_build("ls | cat\n");
  
  // Test that a pipeline was returned
  assert(my_pipeline != NULL);
  assert(!my_pipeline->is_background);
  assert(my_pipeline->commands != NULL);
  
  // Test the parsed args
  assert(strcmp("ls", my_pipeline->commands->command_args[0]) == 0);
  assert(my_pipeline->commands->command_args[1] == NULL);
  
  // Test the redirect state
  assert(my_pipeline->commands->redirect_in_path == NULL);
  assert(my_pipeline->commands->redirect_out_path == NULL);
  
  // Test that there are multiple parsed command in the pipeline
  assert(my_pipeline->commands->next != NULL);
  
  // keep going... what should we be testign next?
  pipeline_free(my_pipeline);
}

We have already provided examples of how you can invoke the unit test programs from a shell scripts (Listing 5.1), and shown how that can in turn by automatically invoked by make (Listing 5.2) so that every time you make a change the makefile will automatically re-run all the tests. Now, given the above test files, if we type make with that makefile, the all rule will run the check rule which will recompile any required software and then invoke the shell script to run all the tests. In our case, both test programs assert because the pipeline returned is NULL.

Unit tests are specific to the particular interfaces of the functions you write. We expect students to write their own unit tests, since the internal functions of their application are up to them to design. For the parser, we have defined the functions that we want you to implement, so that we can give you some example of unit tests. However, we would recommend that a good parser implementation should start by implementing a lexer, that converts each syntactical element of the input into a set of tokens. If you do that, you should add the new interface to the .h file and new tests to make sure that your lexing functionality works.

You should never delete your tests. For example, the parser will eventually become the first shell assignment for many courses based on this book. You will keep adding new functionality, and as you do, you will find that you will find bugs in your parser. If your makefile doesn’t keep running all the old tests, you are very likely to be introducing bugs…

5.5.2. Integration Tests

Integration tests use the public interfaces end-to-end. For example, the tests we run with gradescope to automatically test if your programs are correct are all integration tests. We hope you will share your integration tests with the class, and in the BU version of the course, we will often add tests provided by students to the test suite we use in gradescope; the best way you can have tests you know you will pass is to contribute tests to us. In many cases, you can write integration tests in the same we described above, but calling the public interfaces of libraries. If the task is to develop a program, you can use, for example, shell scripts, with the output redirected to a file, to run tests, compare the result to an expect result, and raise an error if the results do not agree with the expected ones.