Human or Machine? Out-of-Time Validation of Task-Level AI Exposure Ratings
Which task-level AI-exposure ratings continue to align with public measures of task use that appeared after the ratings were constructed?
- Status
- Work in progress
- Role
- Sole author
- Authors
- Ian Helfrich
- Method
- within-occupation models · paired cluster bootstrap · out-of-time validation
- What this record does not establish
- The paper is in progress and reports no results yet.
A public manuscript is not yet available.
Abstract
Summary
A sole-authored paper in progress that tests trained-human and GPT-4 task-level AI-exposure ratings against public measures of task use published after the ratings were constructed. It reports no results yet.
Methods and notes
AI-exposure scores are often evaluated against outcomes that were available when the scores were constructed, or against measures built from the same conceptual ingredients. That can make a rating look validated even when the test is partly circular.
This paper asks a stricter question: when trained-human and GPT-4 ratings are compared with evidence that became available later, which task-level ratings continue to line up with observed task use?