Rethinking Production Tools — Week 10

Testing

Authors
VenueNYU / ITP, Spring 2017

Testing is the practice of verifying that software behaves as expected. Without it, bugs accumulate silently and assumptions go unvalidated. There are two main categories to understand: user testing and software testing.

Why Testing Matters

Testing helps you find bugs early (when they're cheap to fix), validate assumptions about how users interact with your tool, and build confidence that changes don't break existing functionality.

User Testing

User testing means observing real people using your tool. The goal is to learn where the interface is confusing, where users get stuck, and whether the tool solves the problem you designed it to solve.

How to Conduct a User Test

  1. Define what you want to learn. Write specific research questions, not vague goals.
  2. Recruit participants who represent your target users.
  3. Create tasks — specific, realistic activities for participants to complete.
  4. Observe without intervening. Let participants struggle; your job is to watch and take notes, not to help.
  5. Debrief. Ask follow-up questions after each session.
A/B testing setup comparing two interface variations.
A/B testing: comparing two interface variations to determine which performs better on a specific metric.

Types of User Tests

  • Usability Testing: Evaluates ease of use by observing users completing tasks.
  • A/B Testing: Compares two versions of an interface to see which performs better.
  • Guerrilla Testing: Informal, quick tests with anyone available — hallway testing.
  • Remote Testing: Conducted over video call or with screen-sharing tools.

Software Testing

Test pyramid: unit tests at the base, integration tests in the middle, end-to-end tests at the top.
The test pyramid: unit tests form the wide base (fast, many), with integration and end-to-end tests above (slower, fewer).
  • Unit Testing: Tests individual functions or components in isolation. Fast, cheap to run, easy to debug.
  • Integration Testing: Tests how multiple components work together.
  • End-to-End Testing: Tests the full user workflow from start to finish.
  • Regression Testing: Ensures new code doesn't break existing functionality.

Writing Good Tests

  • Keep tests small and focused on one behavior.
  • Test behavior, not implementation details.
  • Use descriptive test names that explain what is being verified.
  • Prioritize testing critical paths over achieving 100% coverage.
Example test output from a Jest unit test suite.
Example output from a Jest test suite — each test case passes or fails with a descriptive label.

Testing Frameworks

A simple Jest test looks like this:

test('adds 1 + 2 to equal 3', () => { expect(sum(1, 2)).toBe(3); });
  • Jest — JavaScript testing framework with zero configuration.
  • Mocha — Flexible JavaScript test framework for Node.js.
  • pytest — Simple, powerful testing for Python.

Assignment

Plan and perform user tests for your prototype. Make a test plan, conduct tests on at least six users, and be prepared to present your findings (10–15 minutes) in the next class.