Model Sketching - Centering Concepts in Early-Stage Machine Learning Model Design

Title: Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design

Authors: Michelle S. Lam, Zixian Ma, Anne Li, Izequiel Freitas, Dakuo Wang, James A. Landay, and Michael S. Bernstein

Word Count: Approximately 24,000 words

Estimated Read Time: 80-90 minutes

Source Code/Repositories: ModelSketchBook API code - https://github.com/StanfordHCI/ModelSketchBook

Summary:

The paper introduces the concept of model sketching for early-stage machine learning model design. Model sketching allows ML practitioners to prototype model behavior through lightweight sketches that focus on high-level concepts relevant to a decision-making task.

The key ideas are:

• Concepts: Human-understandable factors that a model reasons over, like "profanity" or "sarcasm". Concepts serve as functional building blocks of model logic.

• Zero-shot concept instantiation: Models like GPT-3 and CLIP are leveraged to flexibly and rapidly instantiate concepts without diverting user attention.

• Sketch models: Composite models that aggregate concept scores, allowing ML practitioners to explore different combinations of concepts.

The authors implemented model sketching in ModelSketchBook, an open-source Python API. An evaluation with 17 ML practitioners found that model sketching shifted their focus from technical implementation to higher-level concepts. This cognitive shift helped participants explore a broader model design space and identify gaps in data, labels, and problem formulation.

The paper argues that model sketching can help ML practitioners move beyond reactive fixes and engage in more proactive model design exploration from the start.

Overall, model sketching allows ML practitioners to rapidly prototype different model design possibilities through lightweight sketches centered on human-understandable concepts. This represents a shift away from technical tunneling towards higher-level conceptual thinking during early model design stages.