wayang-api

documentation/wayang-api/wayang-api-scala-java/DataQuanta/dataquanta_object.md

@@ -0,0 +1,27 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## DataQuanta Object
+- Represents the companion object for the `DataQuanta` class.
+
+### Key Methods:
+1. **create[T]**
+   - Creates a new `DataQuanta` instance.
+   - Parameters: `output` (output slot of type `T`), `planBuilder` (implicit plan builder).
+   - Returns: A new `DataQuanta` instance.

documentation/wayang-api/wayang-api-scala-java/DataQuanta/dataquanta_scala.md

@@ -0,0 +1,36 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+### Documentation Comments:
+- A TODO comment mentions adding unitary tests to the elements in this file.
+- Represents an intermediate result/data flow edge in a `WayangPlan`.
+
+### Class Definition: DataQuanta[Out: ClassTag]
+- Represents intermediate data or flow edges in a `WayangPlan`.
+- Takes an operator, an output index, and an implicit plan builder as parameters.
+
+#### Key Methods:
+1. `output`: Returns the corresponding `OutputSlot` of a wrapped `Operator`.
+2. `map`: Feeds the instance into a `MapOperator`.
+3. `mapJava`: Similar to `map`, but for Java 8 lambdas.
+4. `project`: Feeds the instance into a `MapOperator` with a `ProjectionDescriptor`.
+5. `connectTo`: Connects the `operator` to another `Operator`.
+6. `filter`: Feeds the instance into a `FilterOperator`.
+7. `filterJava`: Similar to `filter`, but for Java 8 lambdas.
+8. `flatMap`: Appears to feed the instance into a `FlatMapOperator`.

documentation/wayang-api/wayang-api-scala-java/DataQuanta/joined_dataquanta.md

@@ -0,0 +1,33 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## JoinedDataQuanta[Out0: ClassTag, Out1: ClassTag]
+- Represents a class that is the result of a join operation between two data quanta.
+- Contains an embedded `dataQuanta` representing the result of the join operation.
+
+### Key Methods:
+1. **assemble[NewOut: ClassTag]**
+   - Assembles a new element from a join product tuple.
+   - Parameters: `udf` (transformation function), `udfLoad` (load profile estimator).
+   - Returns: The join product `DataQuanta`.
+
+2. **assembleJava[NewOut: ClassTag]**
+   - Assembles a new element from a join product tuple using a Java function.
+   - Parameters: `assembler` (Java transformation function), `udfLoad` (load profile estimator).
+   - Returns: The join product `DataQuanta`.

documentation/wayang-api/wayang-api-scala-java/DataQuanta/keyed_dataquanta_scala.md

@@ -0,0 +1,36 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## KeyedDataQuanta[Out: ClassTag, Key: ClassTag]
+- Represents a class that provides operations on data with functionalities for keyed data.
+- Contains an embedded key extractor of type `SerializableFunction[Out, Key]`.
+
+### Key Methods:
+1. **join[ThatOut: ClassTag]**
+   - Performs a join using the keys of `KeyedDataQuanta`.
+   - Parameters: `that`, the other `KeyedDataQuanta` to join with.
+   - Returns: The join product `DataQuanta`.
+
+2. **coGroup[ThatOut: ClassTag]**
+   - Performs a co-group using the keys of `KeyedDataQuanta`.
+   - Parameters: `that`, the other `KeyedDataQuanta` to co-group with.
+   - Returns: The co-grouped `DataQuanta`.
+
+### Additional Class:
+- **JoinedDataQuanta[Out0: ClassTag, Out1: ClassTag]**: Likely represents the output of join operations.

documentation/wayang-api/wayang-api-scala-java/graph/edge.md

@@ -0,0 +1,31 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## Edge Object
+- This object offers helper functions for handling `Edge` instances in a graph.
+
+### Key Methods:
+1. **apply(src: Long, dest: Long)**
+   - Creates a new edge by specifying the source and target vertex identifiers.
+   - Parameters: `src` (source vertex), `dest` (target vertex).
+   - Return: A new instance of the `Edge` class.
+
+### Observations:
+- The `Edge` object provides a utility method for creating new edges.
+- The implementation is straightforward with a focus on edge creation.

documentation/wayang-api/wayang-api-scala-java/graph/edge_dataquanta.md

@@ -0,0 +1,36 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## EdgeDataQuanta Class
+- `EdgeDataQuanta` enhances the functionality of `DataQuanta` with `Record` objects, representing data flows as edges in a computational graph.
+
+### Comments:
+- The class enhances `DataQuanta` functionality with `Record` objects.
+
+### Key Properties:
+- **planBuilder**: Implicit property that derives a plan builder from `dataQuanta`.
+
+### Key Methods:
+1. **pageRank(numIterations: Int, graphDensity: ProbabilisticDoubleInterval)**
+   - Feeds this instance into a `PageRankOperator`.
+   - Parameters: `numIterations` (default 20), `graphDensity` (default is `PageRankOperator.DEFAULT_GRAPH_DENSITIY`).
+   - Return: A new `DataQuanta` instance representing the `PageRankOperator` output.
+
+### Observations:
+- The class focuses on graph edge functionalities, particularly related to the PageRank algorithm.

documentation/wayang-api/wayang-api-scala-java/graph/edge_dataquanta_builder.md

@@ -0,0 +1,49 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+## EdgeDataQuantaBuilder Trait
+- A trait that extends `DataQuantaBuilder` to offer graph-specific operations.
+
+### Key Methods:
+1. **pageRank(numIterations: Int)**
+   - Feeds built `DataQuanta` into a `PageRankOperator`.
+   - Parameters: `numIterations` for the PageRank algorithm.
+   - Return: An instance representing the `PageRankOperator`'s output.
+
+## EdgeDataQuantaBuilderDecorator Class
+- Acts as a decorator to enrich a regular `DataQuantaBuilder` with operations of a `RecordDataQuantaBuilder`.
+
+### Properties:
+- **baseBuilder**: The base `DataQuantaBuilder` that will be enriched.
+
+## PageRankDataQuantaBuilder Class
+- An implementation of `DataQuantaBuilder` for `MapOperator` objects, focused on handling the PageRank algorithm.
+
+### Properties:
+- **graphDensity**: Represents the presumed graph density.
+- **dampingFactor**: Damping factor for the PageRank algorithm.
+
+### Key Methods:
+1. **withDampingFactor(dampingFactor: Double)**
+   - Sets the damping factor for the PageRank algorithm.
+2. **withGraphDensity(graphDensity: ProbabilisticDoubleInterval)**
+   - Sets the graph density for the PageRank algorithm.
+
+### Observations:
+- The file appears to be centered around handling the PageRank algorithm within the context of data quanta in a graph.
+- The builder pattern is utilized to facilitate the creation and configuration of `EdgeDataQuanta` objects.

documentation/wayang-api/wayang-api-scala-java/util/dataquanta_builder_cache.md

@@ -0,0 +1,42 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## DataQuantaBuilderCache
+- This class caches products of `DataQuantaBuilder` that need to be executed simultaneously.
+
+### Key Properties:
+- **_cache**: An `IndexedSeq` of `DataQuanta[_]` representing the cache storage.
+
+### Key Methods:
+1. **hasCached**
+   - Determines if there's content in the cache.
+   - Returns: Boolean indicating cache status.
+
+2. **apply[T](index: Int)**
+   - Retrieves cached `DataQuanta` by index.
+   - Parameters: `index`.
+   - Return: Cached `DataQuanta` of type `T`.
+
+3. **cache(dataQuanta: Iterable[DataQuanta[_]])**
+   - Caches `DataQuanta`. Should be called once.
+   - Parameters: `dataQuanta`.
+
+### Observations:
+- The class provides caching utilities to optimize certain operations.
+- It offers methods to check, retrieve, and populate the cache.

documentation/wayang-api/wayang-api-scala-java/util/dataquanta_builder_decorator.md

@@ -0,0 +1,49 @@
+<!--
+
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+-->
+
+## DataQuantaBuilderDecorator
+- This utility extends a `DataQuantaBuilder`'s functionality through decoration.
+
+### Comments:
+- There's a note indicating the need to add documentation to the methods of `org.apache.wayang.api.util.DataQuantaBuilderDecorator`.
+
+### Key Methods:
+1. **outputTypeTrap**
+   - Represents the type of the `DataQuanta` to be built.
+   - Return: `TypeTrap` of the base builder's output type.
+
+2. **javaPlanBuilder**
+   - Provides a `JavaPlanBuilder` to which this instance is associated.
+   - Return: `JavaPlanBuilder` of the base builder.
+
+3. **withName**
+   - Sets a name for the `DataQuanta`.
+   - Parameters: `name`.
+   - Return: Current instance with updated name.
+
+4. **withExperiment**
+   - Associates an experiment with the `DataQuanta`.
+   - Parameters: `experiment`.
+   - Return: Current instance with associated experiment.
+
+... (And several other methods for setting or updating properties of the `DataQuantaBuilderDecorator`).
+
+### Observations:
+- Most methods in this utility are fluent setters, allowing for method chaining.
+- The utility acts as a wrapper around another `DataQuantaBuilder`, referred to as `baseBuilder`.