CS375: Software Engineering - Software Testing and Code Coverage
Activity Goals
The goals of this activity are:- To differentiate between white box and black box testing
- To indicate when one testing strategy is appropriate over another
- To explain that although testing all input possibilities and configurations is impossible, achieving good code coverage with heterogeneous inputs is a best practice
- To write unit tests with 100% code and control flow coverage
- To explain that 100% code coverage alone is insufficient because different inputs may exercise those branches differently
- To apply black box, white box, and acceptance testing at each layer (model, controller, and view) of an MVC application
Supplemental Reading
Feel free to visit these resources for supplemental background reading material.The Activity
Directions
Consider the activity models and answer the questions provided. First reflect on these questions on your own briefly, before discussing and comparing your thoughts with your group. Appoint one member of your group to discuss your findings with the class, and the rest of the group should help that member prepare their response. Answer each question individually from the activity, and compare with your group to prepare for our whole-class discussion. After class, think about the questions in the reflective prompt and respond to those individually in your notebook. Report out on areas of disagreement or items for which you and your group identified alternative approaches. Write down and report out questions you encountered along the way for group discussion.Model 1: Choosing Unit Tests
Questions
- What's wrong with this code? (there is more than one answer!)
- How many calls would you make to
triangleAreabefore you decide that it is "passing?" What parameter inputs would you supply to those calls? - Visit this guide and design a unit test for
triangleArea. You can just write the code in your notes: there is no need to compile or execute it now (we will do this in lab instead!). - Recall that floating point types cannot always be compared directly for equality, due to rounding and precision limitations. Change this program to use
doubledata types, and re-generate unit tests for it. Where do you think a floating point tolerance can be added with theassertEqualsfunction?
Model 2: Thinking Critically about Code
Questions
- What kinds of inputs would make this function fail (or return values that don't make sense)? What can you do about this?
- What tests, at a minimum, would you propose to thoroughly exercise this function?
Model 3: Facilitating Unit Tests
Questions
- What makes this a difficult function to test?
- What could we do to better facilitate testing a function like this? For example, how might running the program and evaluating the output be helpful?
- Can black-box testing be automated?
Model 4: Unit Testing and Code Coverage
Questions
- Run the unit tests above and generate a code coverage report.
- How can you improve code coverage to 100%, by testing all code branches?
- Should you continue to write and run additional unit tests beyond 100% code coverage? Give an example of why this might be necessary.
- Would you write your code first or your unit tests first? How might it help to write your unit tests before writing the code?
- How can unit testing and code coverage be automated via a github workflow? When would unit testing be executed?
Example: Python Testing and Coverage Example
Example: Github Automated Testing Workflow Example
Testing at Every Layer of an MVC Application
White box, black box, and acceptance testing are not three phases you do at the end – they are three lenses, and each lens applies to each layer of your MVC architecture. A team that only unit-tests its models will ship controllers that crash on bad input; a team that only clicks through the UI will not know which layer broke when a test fails. The matrix below shows one concrete worked test in every cell, using the note-taking JWT example from the MVC activity (POST /notes creates a note owned by the logged-in user).
| White Box (you can see the code; test branches and internals) | Black Box (inputs and outputs only; no knowledge of internals) | Acceptance (does it satisfy the requirement, in the stakeholder’s terms?) | |
|---|---|---|---|
| Model | Unit test each branch of createNote: note = createNote(1, "hi")assert note.ownerId == 1assert note.createdAt is not Noneand the error branch: creating with empty text raises/rejects. |
Treat the data layer as opaque: insert a row, restart the program, and query it back – same data? Insert two notes and verify the IDs are distinct and increasing. (You do not care how it stores them.) | “The system shall remember my notes.” With the product owner watching: add a note, close the app entirely, reopen, and confirm the note is still listed with a sensible creation date. |
| Controller | Unit test the controller with a mock model (a fake noteModel that records what it was called with): call createNote with no text and assert it returns 400 without calling the model; call it with valid input and assert it passes req.user.sub (not the body!) as the owner. |
Send real HTTP requests and check only status codes and JSON: POST /notes with a token → 201 and the note echoed back; without a token → 401; with {"text": ""} → 400. A shell script of curl commands works well here. |
“Only I can add notes to my account.” Script for a non-technical tester: “Log in as alice, add a note. Log out, log in as bob. Verify bob does not see alice’s note.” |
| View | Inspect the template/DOM logic directly: render the note-list view with a list of 3 notes and assert 3 <li> elements appear; render with 0 notes and assert the “No notes yet” branch appears. Render a note whose text contains <script> and assert it is escaped. |
Drive the real UI without reading its code (by hand, or automated with a tool like Selenium/Playwright): type a note, click Add, and verify it appears in the list, regardless of how the page is implemented. | “The app is usable for adding a note.” Hand a stakeholder the acceptance script from your test plan: “Type a note and save it. Did you see confirmation? Is the note displayed the way you expected?” |
Three observations to carry into your project test plan:
- Failures localize. If the black-box controller test fails but every model unit test passes, the bug is in the controller (or the wiring between them) – you just saved yourself an hour of debugging in the wrong file.
- Mocks are what make layer isolation possible. Because the controller receives its model as a dependency, you can substitute a fake and test the controller’s decisions (status codes, validation, which model function it calls) without a database. This is a direct payoff of MVC separation.
- Coverage means something different in each column. Code coverage (statements and branches exercised) is a white box metric: measure it for your model and controller unit tests, and aim for 100%. For black box tests, “coverage” means every input class and boundary (valid, empty, missing, malicious) appears in your test set. For acceptance tests, coverage means every functional requirement has at least one script – a requirements traceability matrix, not a code coverage report, is how you demonstrate it.
Questions
- Fill in the same 3x3 matrix for one requirement of your project, with one concrete test per cell.
- Which cells of the matrix can run in your GitHub Actions CI workflow on every push, and which require a human? What does that imply about how often each should run?
- Your model tests achieve 100% coverage but the controller passes
req.body.ownerIdto the model instead ofreq.user.sub. Which cell of the matrix catches this bug? What does that tell you about relying on any single row or column?