Skip to main content
Machine Learning Seminar: More For Less: Adaptive Labeling Payments in Online Labor Markets

Machine Learning Seminar: More For Less: Adaptive Labeling Payments in Online Labor Markets

In many predictive tasks where human intelligence is needed to label training instances, online crowdsourcing markets have emerged as promising platforms for large-scale, cost-effective labeling. However, these platforms also introduce challenges that must be addressed for these opportunities to materialize. In particular, it has been shown that different trade-offs arise between payment offered to labelers and the quality of labeling at different times, possibly as a result of different market conditions and even the nature of the tasks themselves. Because the underlying mechanism giving rise to different trade-offs is not well understood, for any given labeling task and at any given time, it is unknown which labeling payments to offer in the market so as to produce accurate models cost-effectively. Effective and robust methods for dealing with these challenges are essential to enable a growing reliance on these promising and increasingly popular labor markets for large-scale labeling. In this talk I will first present a new data science problem, Adaptive Labeling Payment (ALP): how to learn and sequentially adapt the payment offered to crowd labelers before they undertake a labeling task, so as to produce a given (machine learning) predictive model performance cost-effectively. I will then present our approach to address the problem and a rich set of results that explore our approach's performance under a variety of market settings.
Event Time: 
3:30 PM
Event Date: 
Wednesday, March 27, 2019
Duke SSRI

Durham, NC 27708 | 919.681.6019

SSRI

Gross Hall, 2nd Floor
140 Science Drive
Durham, NC 27708

SSRI

Erwin Mill Building
2024 W. Main St.
Durham, NC 27705

Social Media

Follow @dukessri on twitter