Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

README rework #4209

Merged
merged 12 commits into from
Nov 15, 2023
Merged

README rework #4209

Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9530d7d
Updates to README
jjbrosnan Jul 20, 2023
6748dfc
Big updates
jjbrosnan Jul 25, 2023
895e7f2
Merge branch 'main' into update-readme
jjbrosnan Aug 28, 2023
b3a1ead
Updates from Devin's review - should we add a BUILD.md file?
jjbrosnan Aug 29, 2023
47f14f4
Minor update
jjbrosnan Aug 29, 2023
b131e04
Updates from Chip's review
jjbrosnan Sep 28, 2023
0d6d171
Merge branch 'main' into update-readme
jjbrosnan Oct 18, 2023
12b424b
Minor update
jjbrosnan Oct 18, 2023
661e39b
Updates from Chip's latest review
jjbrosnan Oct 30, 2023
90b3508
Add note about supported browsers with link to web client UI README s…
jjbrosnan Oct 30, 2023
f8c7b7c
Update README.md
margaretkennedy Oct 30, 2023
660e7a9
Updates from Chip's review
jjbrosnan Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
304 changes: 107 additions & 197 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,9 @@ Deephaven includes an intuitive user experience and visualization tools.
It can ingest data from a variety of sources, apply computation and analysis algorithms
to that data, and build rich queries, dashboards, and representations with the results.

Deephaven Community Core is an open version of [Deephaven Enterprise](https://deephaven.io),
Deephaven Community Core is the open version of [Deephaven Enterprise](https://deephaven.io),
which functions as the data backbone for prominent hedge funds, banks, and financial exchanges.

[![Join the chat at https://gitter.im/deephaven/deephaven](https://badges.gitter.im/deephaven/deephaven.svg)](https://gitter.im/deephaven/deephaven?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
![Build CI](https://github.com/deephaven/deephaven-core/actions/workflows/build-ci.yml/badge.svg?branch=main)
![Quick CI](https://github.com/deephaven/deephaven-core/actions/workflows/quick-ci.yml/badge.svg?branch=main)
![Docs CI](https://github.com/deephaven/deephaven-core/actions/workflows/docs-ci.yml/badge.svg?branch=main)
![Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/check-ci.yml/badge.svg?branch=main)
![Nightly Check CI](https://github.com/deephaven/deephaven-core/actions/workflows/nightly-check-ci.yml/badge.svg?branch=main)
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

## Supported Languages

| Language | Server Application | Client Application |
Expand All @@ -27,255 +20,169 @@ which functions as the data backbone for prominent hedge funds, banks, and finan
| Java / Groovy | Yes | Yes |
| C++ | No | Yes |
| JavaScript | No | Yes |
| gRPC | - | Yes |
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

## Run Deephaven

This section is a quick start guide for running Deephaven from pre-built images. Almost all users will want to run Deephaven using pre-built images. It is the easiest way to deploy. For detailed instructions, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart).

Developers interested in tinkering with and modifying source code should build from the source code. For detailed instructions on how to do this, see [Build and launch Deephaven](https://deephaven.io/core/docs/how-to-guides/launch-build).

If you are not sure which of the two is right for you, use the pre-built images.

### Required Dependencies

Running Deephaven requires a few software packages.

| Package | Version | OS |
| -------------- | ----------------------------- | ------------ |
| docker | ^20.10.8 | All |
| docker-compose | ^1.29.0 | All |
| Windows | 10 (OS build 20262 or higher) | Only Windows |
| WSL | 2 | Only Windows |

You can check if these packages are installed and functioning by running:
```
docker version
docker-compose version
docker run hello-world
```

> :warning: **On Windows, all commands must be run inside a WSL 2 terminal.**
| Go | No | Yes |
| R | No | Yes |

If any dependencies are missing or unsupported versions are installed, see [Launch Deephaven from pre-built images](https://deephaven.io/core/docs/tutorials/quickstart#prerequisites) for installation instructions.
## Installation
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

Deephaven Community Core can be used from both server- and client-side APIs. Server-side APIs allow users to connect directly to the Deephaven server and execute commands, write queries, and much more. Client-side APIs allow users to connect to a Deephaven server through a client application.

### Create deployment
See the following documentation links for installation instructions and more:

A directory must be created to store files and mount points for your deployment. Here, we are using the `deephaven-deployment` directory.
- Python
- [Run from Docker](https://deephaven.io/core/docs/tutorials/quickstart/)
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
- [pip-installed](https://deephaven.io/core/docs/tutorials/quickstart-pip/)
- Groovy
- [Run from Docker](https://deephaven.io/core/groovy/docs/tutorials/quickstart/)
- [Python client](https://pypi.org/project/pydeephaven/)
- [Java client](https://deephaven.io/core/docs/how-to-guides/java-client/)
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
- [JS client](https://deephaven.io/core/docs/reference/js-api/documentation/)
- [Go client](https://pkg.go.dev/github.com/deephaven/deephaven-core/go)
- [R client](https://github.com/deephaven/deephaven-core/blob/main/R/rdeephaven/README.md)

You will need to `cd` into the deployment directory to launch or interact with the deployment.
Deephaven's client APIs use [gRPC](https://grpc.io/), [protobuf](https://github.com/deephaven/deephaven-core/tree/main/proto/proto-backplane-grpc/src/main/proto/deephaven/proto), [Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html), and [Barrage](https://github.com/deephaven/barrage) to handle ticking data. Users who wish to build their own client APIs can use these tools to do so.
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

```bash
mkdir deephaven-deployment
cd deephaven-deployment
```

> :warning: **Commands in the following sections for interacting with a deployment must be run from the deployment directory.**

### Launch: Python

Run the following commands to launch Deephaven for Python server applications.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/base/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with NLTK

Run the following commands to launch Deephaven for Python server applications with the [NLTK](https://nltk.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/NLTK/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with PyTorch
## Run the Deephaven server

Run the following commands to launch Deephaven for Python server applications with the [PyTorch](https://pytorch.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/PyTorch/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```
### From Docker

### Launch: Python with SciKit-Learn
This is the easiest way to get started with Deephaven. For full instructions, see our [quickstart for Docker](https://deephaven.io/core/docs/tutorials/quickstart/).

Run the following commands to launch Deephaven for Python server applications with the [SciKit-Learn](https://scikit-learn.org/stable/) module pre-installed.
TL;DR:
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/SciKit-Learn/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```sh
# Python
docker run --rm --name deephaven -p 10000:10000 ghcr.io/deephaven/server:latest
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
```

### Launch: Python with TensorFlow

Run the following commands to launch Deephaven for Python server applications with the [TensorFlow](https://www.tensorflow.org/) module pre-installed.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python/TensorFlow/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```sh
# Groovy
docker run --rm name deephaven -p 10000:10000 ghcr.io/deephaven/server-slim:latest
```

### Launch: Python with example data

Run the following commands to launch Deephaven for Python server applications, with example data.
### From Python

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/base/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```

### Launch: Python with example data and NLTK
Users who wish to avoid using Docker will likely want to use [pip-installed Deephaven](https://deephaven.io/core/docs/tutorials/quickstart-pip/).

Run the following commands to launch Deephaven for Python server applications, with example data and [NLTK](https://nltk.org/).
TL;DR

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/NLTK/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```sh
pip install deephaven-server deephaven-ipywidgets
```

### Launch: Python with example data and PyTorch

Run the following commands to launch Deephaven for Python server applications, with example data and [PyTorch](https://pytorch.org/).
Then, from Python:

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/PyTorch/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```python
from deephaven_server import Server
s = Server(port=10000, jvm_args=["-Xmx4g"]).start()
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
```

### Launch: Python with example data and SciKit-Learn

Run the following commands to launch Deephaven for Python server applications, with example data and [SciKit-Learn](https://scikit-learn.org/stable/).

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/SciKit-Learn/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```
### Built from source

### Launch: Python with example data and TensorFlow
Users who wish to modify source code and contribute to the project should build Deephaven from source. For full instructions, see [how to build Deephaven from source](https://deephaven.io/core/docs/how-to-guides/launch-build/).
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

Run the following commands to launch Deephaven for Python server applications, with example data and [TensorFlow](https://www.tensorflow.org/).
TL;DR for Python:
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/python-examples/TensorFlow/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```sh
git clone https://github.com/deephaven/deephaven-core.git
cd deephaven-core
python -m venv /tmp/my-dh-venv
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
source /tmp/my-dh-venv/bin/activate
./gradlew py-server:assemble
pip install "py/server/build/wheel/deephaven_core-<version>-py3-non-any.whl[autocomplete]
./gradlew server-jetty-app:run
```

### Launch: Groovy / Java

Run the following commands to launch Deephaven for Groovy / Java server applications.
TL;DR for Groovy:

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```sh
git clone https://github.com/deephaven/deephaven-core.git
cd deephaven-core
./gradlew server-jetty-app:run -Pgroovy
```

### Launch: Groovy / Java with example data
### Required dependencies
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

Run the following commands to launch Deephaven for Groovy / Java server applications, with example data.

```bash
curl https://raw.githubusercontent.com/deephaven/deephaven-core/main/containers/groovy-examples/docker-compose.yml -O
docker-compose pull
docker-compose up -d
```
Building and running Deephaven requires a few software packages.

### Monitor logs

The `-d` option to `docker-compose` causes the containers to run in the background, in detached mode. This option allows you to use your shell after Docker launches the containers.
| Package | Version | OS |
| -------------- | ----------------------------- | ------------ |
| git | ^2.25.0 | All |
| java | >=11, <20 | All |
| docker | ^20.10.8 | All |
| docker-compose | ^1.29.0 | All |
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
| Windows | 10 (OS build 20262 or higher) | Only Windows |
| WSL | 2 | Only Windows |

Since the container is running detached, you will not see any logs. However, you can follow the logs by running:
You can check if these packages are installed and functioning by running:

```bash
docker-compose logs -f
git version
java -version
docker version
docker-compose version
docker run hello-world
```

Use CTRL+C to stop monitoring the logs and return to a prompt.

### Shutdown

The deployment can be brought down by running:

```bash
docker-compose down
```
:::note

### Manage example data
Internally, the Java build process will use [Gradle Auto Provisioning](https://docs.gradle.org/current/userguide/toolchains.html#sec:provisioning)
to download and use the appropriate Java version for building and testing.

[Deephaven's examples repository](https://github.com/deephaven/examples) contains data sets that are useful when learning
to use Deephaven. These data sets are used extensively in Deephaven's documentation and are needed to run some examples. [Deephaven's examples repository](https://github.com/deephaven/examples) contains documentation on the available data sets and how to manage them.
On Windows, all commands must be run inside a WSL 2 terminal.

If you have chosen a deployment with example data, the example data sets will be downloaded. Production deployments containing your own data will not need the example data sets.
:::

### Authentication
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

To upgrade a deployment to the latest example data, run:
Deephaven, by default, uses pre-shared key authentication to authenticate against unauthorized access. For more information on supported authentication methods:

```bash
docker-compose run examples download
```
- [How to configure and use pre-shared key authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-psk/)
- [How to configure and use anonymous authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-anon/)
- [How to configure and use username/password authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-uname-pw/)
- [How to configure and use mTLS authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-mtls/)
- [How to configure and use Keycload/OpenID authentication](https://deephaven.io/core/docs/how-to-guides/authentication/auth-keycloak/)

To see what other example data management commands are available, run:
If using a randomly generated pre-shared key, you can find the key in the server logs:

```bash
docker-compose run examples
docker compose logs -f | grep "access through pre-shared key"
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
```

If your deployment does not have example data, these commands will fail with `ERROR: No such service`.


## Run Deephaven IDE

Once Deephaven is running, you can launch a Deephaven IDE in your web browser. Deephaven IDE allows you to interactively analyze data.

- If Deephaven is running locally, navigate to [http://localhost:10000/ide/](http://localhost:10000/ide/).
- If Deephaven is running remotely, navigate to `http://<hostname>:10000/ide/`, where `<hostname>` is the address of the machine Deephaven is running on.
### Connect to the server

![alt_text](docs/images/ide_startup.png "Deephaven IDE")
Deephaven is run from a web browser, and can be connected to via `http://localhost:10000/ide`. If using authentication, enter credentials to gain access to the IDE.
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved

# First query
## First query

From the Deephaven IDE, you can perform your first query.

This script creates two small tables: one for employees and one for departments.
This Python script creates two small tables: one for employees and one for departments.
It joins the two tables on the DeptID column to show the name of the department
where each employee works.

```python
jjbrosnan marked this conversation as resolved.
Show resolved Hide resolved
from deephaven import new_table
from deephaven.column import string_col, int_col
from deephaven.constants import NULL_INT

from deephaven.TableTools import newTable, stringCol, intCol
from deephaven.conversion_utils import NULL_INT
left = new_table([
string_col("LastName", ["Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"]),
int_col("DeptID", [31, 33, 33, 34, 34, NULL_INT]),
string_col("Telephone", ["(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None])
])

left = newTable(
stringCol("LastName", "Rafferty", "Jones", "Steiner", "Robins", "Smith", "Rogers"),
intCol("DeptID", 31, 33, 33, 34, 34, NULL_INT),
stringCol("Telephone", "(347) 555-0123", "(917) 555-0198", "(212) 555-0167", "(952) 555-0110", None, None)
)
right = new_table([
int_col("DeptID", [31, 33, 34, 35]),
string_col("DeptName", ["Sales", "Engineering", "Clerical", "Marketing"]),
string_col("Telephone", ["(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111"])
])

right = newTable(
intCol("DeptID", 31, 33, 34, 35),
stringCol("DeptName", "Sales", "Engineering", "Clerical", "Marketing"),
stringCol("Telephone", "(646) 555-0134", "(646) 555-0178", "(646) 555-0159", "(212) 555-0111")
)

t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone")
t = left.join(right, "DeptID", "DeptName, DeptTelephone=Telephone")
```

![alt_text](docs/images/ide_first_query.png "Deephaven IDE First Query")


## Resources

* [Help!](https://github.com/deephaven/deephaven-core/discussions/969)
Expand All @@ -285,15 +192,18 @@ t = left.join(right, "DeptID", "DeptName,DeptTelephone=Telephone")
* [Java API docs](https://deephaven.io/core/javadoc/)
* [Python API docs](https://deephaven.io/core/pydoc/)

## Code Of Conduct
## Contributing

See [CONTRIBUTING](./CONTRIBUTING.md) for full instructions on how to contribute to this project.

### Code Of Conduct

This project has adopted the [Contributor Covenant Code of Conduct](https://www.contributor-covenant.org/version/2/0/code_of_conduct/).
For more information see the [Code of Conduct](CODE_OF_CONDUCT.md) or contact [[email protected]](mailto:[email protected])
with any additional questions or comments.


## License
### License

Copyright © 2016-2023 Deephaven Data Labs and Patent Pending. All rights reserved.

Provided under the [Deephaven Community License](LICENSE.md).
Provided under the [Deephaven Community License](LICENSE.md).
Loading