Skip to content

Commit

Permalink
README improvements (#35)
Browse files Browse the repository at this point in the history
* README improvements

* README updates

* README updates
  • Loading branch information
gordonhart committed Sep 9, 2024
1 parent 1c6d2d3 commit cb0715b
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 3 deletions.
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,32 @@
# AutoArena

AutoArena helps you stack rank LLM outputs against one another using automated judge evaluation. Get started by:
AutoArena helps you stack rank LLM outputs against one another using automated judge evaluation.

Install from [PyPI](https://pypi.org/project/autoarena/) and run with:

```
pip install autoarena
python -m autoarena
```

Data is stored in an `autoarena.duckdb` file in the directory where you invoked AutoArena.
## Usage

Getting started with AutoArena is simple:

1. Run AutoArena via `python -m autoarena` and visit [localhost:8899](http://localhost:8899/) in your browser.
2. Create a project via the UI.
3. Add responses from a model by selecting a CSV file with `prompt` and `response` columns.
4. Configure an automated judge via the UI. Note that most judges require credentials, e.g. `X_API_KEY` in the
environment where you're running AutoArena.
5. Add responses from a second model to kick off an automated judging task using the judges you configured in the
previous step to decide which of the models you've uploaded provided a better `response` to a given `prompt`.

That's it! After these steps you're fully set up for automated evaluation on AutoArena.

### Data Storage

Data is stored in `./data/<project>.duckdb` files in the directory where you invoked AutoArena. See
[`data/README.md`](./data/README.md) for more details on data storage in AutoArena.

## Development

Expand Down
5 changes: 4 additions & 1 deletion ui/src/components/Page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ export function Page({ tab }: Props) {
const navigate = useNavigate();

useEffect(() => {
if (projectSlug != null && !isLoadingProject && project == null) {
// TODO: this is nice functionality to have but sometimes there is a race condition where it fires after creating
// a project but before it loads. Disable for now
const enableRedirect = false;
if (enableRedirect && projectSlug != null && !isLoadingProject && project == null) {
notifications.show({
title: `Project '${projectSlug}' not found`,
message: <>The project '{projectSlug}' does not seem to exist in the expected file. Redirecting home.</>,
Expand Down

0 comments on commit cb0715b

Please sign in to comment.