BTS is a platform to test various peer prediction mechanisms on crowdsourcing platforms.
Researchers wishing to use the platform can supply a list of questions, create small HTML templates for the UI and specify the payment mechanism. BTS does the rest.
- Each agent sees a different random permutation of the questions
- Multiple agents may work on the same question. Payment mechanism may rely on all answers in that round
- GUI to set up the experiment (beta feature)
- Built using the Meteor framework with MongoDB
- Export data from the experiments in a CSV format.
To install a local copy of the platform, follow these instructions:
-
Clone the directory:
git clone https://github.com/CdavM/bts bts/ -
Start your app with the
docker runcommand. -
Navigate to
/setupto configure the experiment
It is possible to deploy the platform via Amazon AWS. Once you provision a Ubuntu instance (14.04 LTS), follow the instructions above.
Multiple experiments have been deployed with this platform:
- Iterative decision making under various utility models [Stanford University]: https://github.com/cdavm/harp
- Rewarding truthfulness with inversely proportional rewards [UC Berkeley]: https://github.com/CdavM/purple/
- Comparing output agreement mechanisms to state-of-the-art [UC Berkeley]: The master branch contains this experiment as the default setup.
- Comparing the confusion of agents: https://github.com/CdavM/bts/tree/comparison
Feel free to open pull requests or issues on GitHub should you notice any bugs.
This platform is licensed under the GPL 3.0 license.