Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Utilities to assist in retrieving data from tables #2857

Open
wants to merge 45 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
f971e58
Initial commit of table-to-record adapter/listener utility
rbasralian Sep 12, 2022
6fb85ac
add QueryUtil to build
rbasralian Sep 14, 2022
4b70eff
Update variable names
rbasralian Sep 14, 2022
45902c1
Fix retrieval of multiple records with single keys
rbasralian Sep 14, 2022
4f3ba84
Add Python version of KeyedRecordAdapter. (Currently only works with …
rbasralian Sep 14, 2022
afacf3d
spotless
rbasralian Sep 14, 2022
6d567aa
remove dependency
rbasralian Sep 14, 2022
591ceca
add dependency
rbasralian Sep 14, 2022
4c96038
work on support for primitive keys from Python
rbasralian Sep 21, 2022
0749ec0
fix bad 'isRefreshing' parameter in GetDataLockType. add not-yet-func…
rbasralian Oct 6, 2022
3952474
fix bug for missing single records
rbasralian Oct 7, 2022
a626e19
fix(?) some snapshot source stuff
rbasralian Oct 11, 2022
00851cd
Updates after merging `main`
rbasralian Jan 5, 2023
fe9cfc3
reduce visibility where possible
rbasralian Jan 5, 2023
b994f30
remove some outdated classes. simplify package structure.
rbasralian Jan 5, 2023
422a74e
javadocs update
rbasralian Oct 6, 2023
ceb0bfd
remove an extra layer of method dispatch from RecordUpdater
rbasralian Oct 6, 2023
68313eb
Fix DateTime -> Instant after rebase
rbasralian Jan 8, 2024
62a630e
Handle multiple update graphs after rebase
rbasralian Jan 8, 2024
b6643cb
Don't use terminal listeners. (And switch from InstrumentedTableUpdat…
rbasralian Jan 9, 2024
fd1efca
Misc. test cleanup
rbasralian Jan 9, 2024
6c640b0
Task to run select py tests
rbasralian Jan 9, 2024
05daad2
rename module
rbasralian Jan 9, 2024
14d8d1a
rename package
rbasralian Jan 9, 2024
4dfd3d7
spotless
rbasralian Jan 9, 2024
be5f0bc
Support building jpy from local sources when running python tests
rbasralian Jan 9, 2024
e672c1b
fix array conversion after rebase
rbasralian Jan 9, 2024
c7f8223
Use as_jobj to convert Python values to the expected Java datatypes. …
rbasralian Jan 9, 2024
c79949c
Merge remote-tracking branch 'origin/main' into raffi_record_adapter
rbasralian Feb 1, 2024
b7aaae2
Merge remote-tracking branch 'origin/main' into raffi_record_adapter
rbasralian Mar 21, 2024
4916a57
spotless (copyright)
rbasralian Mar 22, 2024
ec8309b
jpy.as_jobj renamed to jpy.convert
rbasralian Mar 22, 2024
2f204c7
Merge remote-tracking branch 'origin/main' into raffi_record_adapter
rbasralian Apr 2, 2024
e9b1d5d
javadoc fix
rbasralian Apr 2, 2024
95bcdf6
Merge remote-tracking branch 'origin/main' into raffi_record_adapter
rbasralian Apr 4, 2024
e59b117
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Jun 21, 2024
590b855
fix for QueryCompiler changes
rbasralian Jun 21, 2024
26ac7b0
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Jul 31, 2024
9294dc3
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Aug 2, 2024
6745f95
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Aug 5, 2024
d9a12bc
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Aug 23, 2024
81a4c71
update dependencies
rbasralian Aug 26, 2024
b58c7db
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Sep 3, 2024
2b09ce5
fix python test
rbasralian Sep 3, 2024
3c2ad4f
Merge remote-tracking branch 'refs/remotes/origin/main' into raffi_re…
rbasralian Oct 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
158 changes: 156 additions & 2 deletions Util/src/main/java/io/deephaven/util/type/TypeUtils.java
Original file line number Diff line number Diff line change
Expand Up @@ -339,8 +339,8 @@ public static double[] toDoubleArray(double[] array) {

public static boolean isConvertibleToPrimitive(Class<?> type) {
final Class<?> unboxedType = TypeUtils.getUnboxedType(type);
return unboxedType != null && unboxedType != boolean.class; // TODO: isConvertibleToPrimitive(Boolean.class) ==
// false ???
// (Note: Booleans must be boxed because primitive boolean does not support null.)
return unboxedType != null && unboxedType != boolean.class;
}

public static boolean isBoxedType(Class<?> exprType) {
Expand Down Expand Up @@ -628,4 +628,158 @@ public static long unbox(Long value) {
public static short unbox(Short value) {
return (value == null ? NULL_SHORT : value);
}

public static Byte[] box(byte[] values) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of these new methods need javadocs.

if (values == null) {
return null;
}
final Byte[] boxed = new Byte[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Character[] box(char[] values) {
if (values == null) {
return null;
}
final Character[] boxed = new Character[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Double[] box(double[] values) {
if (values == null) {
return null;
}
final Double[] boxed = new Double[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Float[] box(float[] values) {
if (values == null) {
return null;
}
final Float[] boxed = new Float[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Integer[] box(int[] values) {
if (values == null) {
return null;
}
final Integer[] boxed = new Integer[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Long[] box(long[] values) {
if (values == null) {
return null;
}
final Long[] boxed = new Long[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static Short[] box(short[] values) {
if (values == null) {
return null;
}
final Short[] boxed = new Short[values.length];
for (int ii = 0; ii < values.length; ii++) {
boxed[ii] = box(values[ii]);
}
return boxed;
}

public static byte[] unbox(Byte[] values) {
if (values == null) {
return null;
}
final byte[] unboxed = new byte[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static char[] unbox(Character[] values) {
if (values == null) {
return null;
}
final char[] unboxed = new char[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static double[] unbox(Double[] values) {
if (values == null) {
return null;
}
final double[] unboxed = new double[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static float[] unbox(Float[] values) {
if (values == null) {
return null;
}
final float[] unboxed = new float[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static int[] unbox(Integer[] values) {
if (values == null) {
return null;
}
final int[] unboxed = new int[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static long[] unbox(Long[] values) {
if (values == null) {
return null;
}
final long[] unboxed = new long[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}

public static short[] unbox(Short[] values) {
if (values == null) {
return null;
}
final short[] unboxed = new short[values.length];
for (int ii = 0; ii < values.length; ii++) {
unboxed[ii] = unbox(values[ii]);
}
return unboxed;
}
}
29 changes: 29 additions & 0 deletions data-adapter/build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
plugins {
id 'io.deephaven.project.register'
id 'java-library'
}

dependencies {
api project(':engine-api')
api project(':engine-table')
api project(':engine-function')

api 'com.fasterxml.jackson.core:jackson-databind:2.13.4'

implementation project(':log-factory')
implementation libs.commons.lang3

testRuntimeOnly project(path: ':configs')
testRuntimeOnly project(path: ':test-configs')

testImplementation TestTools.projectDependency(project, 'engine-table')
testImplementation project(path: ':Base', configuration: 'tests')

testRuntimeOnly project(':log-to-slf4j')
}

repositories {
mavenCentral()
}

TestTools.addEngineOutOfBandTest(project)
1 change: 1 addition & 0 deletions data-adapter/gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
io.deephaven.project.ProjectType=JAVA_PUBLIC
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
//
// Copyright (c) 2016-2024 Deephaven Data Labs and Patent Pending
//
package io.deephaven.dataadapter;

import io.deephaven.chunk.*;
import io.deephaven.chunk.attributes.Values;
import io.deephaven.engine.rowset.RowSequence;
import io.deephaven.engine.table.ChunkSource;
import io.deephaven.engine.table.ColumnSource;
import org.jetbrains.annotations.NotNull;

import java.util.Objects;

class ChunkRetrievalUtil {

@NotNull
static <T> ObjectChunk<T, ?> getObjChunkForKeys(
final ColumnSource<T> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asObjectChunk();
}

@NotNull
static CharChunk<? extends Values> getCharChunkForKeys(
final ColumnSource<Character> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asCharChunk();
}

@NotNull
static ByteChunk<? extends Values> getByteChunkForKeys(
final ColumnSource<Byte> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asByteChunk();
}

@NotNull
static ShortChunk<? extends Values> getShortChunkForKeys(
final ColumnSource<Short> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asShortChunk();
}

@NotNull
static IntChunk<? extends Values> getIntChunkForKeys(
final ColumnSource<Integer> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asIntChunk();
}

@NotNull
static FloatChunk<? extends Values> getFloatChunkForKeys(
final ColumnSource<Float> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asFloatChunk();
}

@NotNull
static LongChunk<? extends Values> getLongChunkForKeys(
final ColumnSource<Long> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asLongChunk();
}

@NotNull
static DoubleChunk<? extends Values> getDoubleChunkForKeys(
final ColumnSource<Double> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asDoubleChunk();
}

@NotNull
static BooleanChunk<? extends Values> getBooleanChunkForKeys(
final ColumnSource<Boolean> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context,
final boolean usePrev) {
return getChunkForKeys(
columnSource,
rowSequence,
context,
usePrev).asBooleanChunk();
}


@NotNull
static <T> Chunk<? extends Values> getChunkForKeys(
final ColumnSource<T> columnSource,
final RowSequence rowSequence,
final ChunkSource.GetContext context, boolean usePrev) {
return Objects.requireNonNull(usePrev ? columnSource.getPrevChunk(context, rowSequence)
: columnSource.getChunk(context, rowSequence));
}


private ChunkRetrievalUtil() {}

}
Loading