Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a utility rate_gold function #3

Open
mbernst opened this issue Aug 19, 2016 · 1 comment
Open

Create a utility rate_gold function #3

mbernst opened this issue Aug 19, 2016 · 1 comment

Comments

@mbernst
Copy link

mbernst commented Aug 19, 2016

I'm thinking about ways to enable traditional data capture workflows. Gold standards are a super common option. What if we created another utility rate_gold function? It takes in a dict input->correct answer as an argument, and then the utility function looks to see if any inputs from the tasks match the input, and if so check if the response matches a gold. It then takes sum(correct) and sends that to the rate() function.

@shirishgoyal
Copy link
Contributor

It depends a lot on the data structure requester wants to use. So we let requester pass the data into the system and allow them to retrieve it later during approval and thus define any function to compare data with gold truth and rate accordingly. Also added a sample script on using gold data in tasks (using gold data for rating)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants