Concatenate outputs layers to generate other layer input

[Isdriai]

Hi,

I'm new on DJL and I want implement my idea, unfortunately, I don't find how to do.

My model is like that: I have data composed by different "types", a specific data can share some types with others data but not all types. My idea is for all combinations of type: to do a little Neural Network (NN) by type and a last NN to aggregate outputs from little NN.

(When I say type, it's a set of features)

An illustration with a simple example

I tried to read documentation and the DJL's code on github, I have two ideas to implement my idea but I have problems with each one

First idea: ParalleleBlock

I saw the ParalleleBlock, I thought I was able to do a function which does what I want but when I read the code of this class I see the function is applied on each children with all inputs. So my question is: Is it possible, for each child, to take only a part of the input and at the end concatenate the parts ?

To better explanation, I found an image illustrating the mechanism of ParalelleBlock and I added an illustration of what I want

And an other question with this idea: I saw I can use a Trainer to control the training process here. But will there be some conflicts if different trainers (so with different models) share parameters ? (the idea implies that what you call Model is 2 little NN with a NN for the combination in my first illustration)

Second idea: from scratch

With Pytorch, I saw I could do my idea here
So I searched how to implement a NN from scratch with JDL and I found this but when I want to do a forward, I have to give a ParameterStore, and I don't understand the purpose of this element. I mean I saw usually the parameterStore is from the trainer but with this approach I don't have a trainer so I have to do:

val yHat = typeBlocks.forward(ParameterStrore(), .... ) (kotlin code)

And I don't know if it's the good practice

With the from scratch approach (without trainer so), is it possible to use SequentialBlock and to do the forward/bachpropagation manually ?

If you have question on my idea, don't hesitate :-)

EDIT:

I'm wondering if I have to create my own block class inherited from AbstractBlock, I read the code of subclasses already existing, if I understand well I have to override three methods:

public Shape[] getOutputShapes(Shape[] inputShapes)

public void initializeChildBlocks(NDManager manager, DataType dataType, Shape... inputShapes)

protected NDList forwardInternal(ParameterStore ps, NDList inputs, boolean training, PairList<String, Object> params)

[zachgk]

What you want is to create a custom AbstractBlock. The ParallelBlock and SequentialBlock are there because they represent common patterns used in many models. But, they can't implement all of the models that people want to implement. With a custom block, you can write pretty much any model as it is just imperative Java code.

For the Trainer, I want to understand what your use case is a bit more. Do you have three different models because they work on different datasets, but you have shared parameters between them? Or, is it that the models are all working together on a single dataset? Or, is it something else?

[Isdriai]

Yes I have the same loss for all packets. Ok I'll do what you propose.

Thanks for your answers :)

I let the topic as unanswered if I have other questions while the implementation

[Isdriai]

Hi,

Finally I have many trainers (one by protocols combination) and many dataset (same) grouped into one class which return the good one when I pass it good parameters.

I have one issue right now, I don't know how to transform my data into NDArray, I use Table from Tablesaw if you know but if you don't, don't worry I can easily transform a table into Map or a List.

I saw I can't instantiate a NDArray because it's an interface and I have to instantiate for example a PtNDArray.

A PtNDArray takes a PtNDManager and a (long) handle, if I understand well, I can have a PtNDManager thanks to

public PtNDManager newSubManager(Device device)

But what is the handle and where I can get it ?

[zachgk]

You can create an NDArray by using the methods attached to the NDManager class. Due to limitations with memory management, all NDArrays have to be attached to a manager so that closing the manager also cleans up the attached arrays.

For your table data, what you want to do is first transform the data into a single 1D feature vector of type float[]. Then, you can do:

float[] myFloatData = ...
NDArray dataArray = NDManager.create(myFloatData);

You can also look at https://d2l.djl.ai/chapter_preliminaries/ndarray.html for more examples of creating and manipulating NDArrays.

[Isdriai]

Sorry for the delay, I fixed a lot of bugs :)

According to your link about data management I use try(NDManager manager = NDManager.newBaseManager()){ my training function .... }

( NDManager.newBaseManager().use { manager -> training fuction ... } in my case because kotlin )

But I have a bug with my manager:

Exception in thread "main" java.lang.IllegalStateException: NDManager has been closed already.
	at ai.djl.ndarray.BaseNDManager.attachInternal(BaseNDManager.java:274)
	at ai.djl.mxnet.engine.MxNDArray.<init>(MxNDArray.java:89)
	at ai.djl.mxnet.engine.MxNDArray.<init>(MxNDArray.java:68)
	at ai.djl.mxnet.engine.MxNDManager.create(MxNDManager.java:101)
	at ai.djl.mxnet.engine.MxNDManager.create(MxNDManager.java:35)
	at ai.djl.ndarray.NDManager.create(NDManager.java:505)
	at ai.djl.ndarray.NDManager.create(NDManager.java:482)
	at ai.djl.ndarray.NDManager.create(NDManager.java:519)
	at neuralPacket.utils.MultiPathDataset$PathDataset$PathIterable.tableToNDArray(MultiPathDataset.kt:189)
	at neuralPacket.utils.MultiPathDataset$PathDataset$PathIterable.constructNDListFromListTable(MultiPathDataset.kt:196)
	at neuralPacket.utils.MultiPathDataset$PathDataset$PathIterable.next(MultiPathDataset.kt:211)
	at neuralPacket.utils.MultiPathDataset$PathDataset$PathIterable.next(MultiPathDataset.kt:127)
	at neuralPacket.NeuralPacket.trainAndTest(neuralPacket.kt:362)
	at MainKt.main(main.kt:13)

It happens in the training phase after the first batch

The code where it happens:

pathTrainer.iterateDataset(multiDataset[path.getName(), true]).forEachIndexed { index, batch ->

                    println("training batch $index in progress")

                    EasyTrain.trainBatch(pathTrainer, batch)
                    pathTrainer.step()
                    batch.close()

                    println("training batch $index ended")
                }

( multiDataset[path.getName(), true] ===> training data for the protocols combination, true means training)

I checked when the manager is closed and it appears it at the line pathTrainer.step()

I pass the manager to my custom dataset, it uses it to calculate the next batch

override fun next(): Batch {
                val dataBatchNext = getDataBatchNext()
                val inputs = constructNDListFromListTable(dataBatchNext.map(Pair<Table, Any>::first))
                val labels = constructNDListFromListListInt(dataBatchNext.map(Pair<Any, List<Int>>::second))
                val batch = Batch(
                    manager,
                    inputs,
                    labels,
                    this.pathDataset.dataset.batchSize,
                    Batchifier.STACK,
                    Batchifier.STACK,
                    indexBatch.toLong() * this.pathDataset.dataset.batchSize,
                    getDataPath(this.pathDataset.training).values.first().rowCount().toLong()
                )
                indexBatch++
                return batch
            }

(the manager attribute is from the training function I show you previously and it is used in constructNDListFromListTable and constructNDListFromListListInt)

Furthermore, I use the manager in my custom block, I fetch it from the data

The code of custom block where I use the manager:

	override fun forwardInternal(
	        parameterStore: ParameterStore?,
	        data: NDList?,
	        labels: NDList?,
	        params: PairList<String, Any>?
	    ): NDList {


	    	val res = this.children.values().zip(data).map { (child, childData) ->

	            child.forward(parameterStore, NDList(childData), labels, params)
	                .singletonOrThrow()
	                .toFloatArray()

	        }.reduce(FloatArray::plus)

	        val nbrRows = data.map{it.shape[0]}.distinct()[0]
	        val nbrCol = res.size / nbrRows 
	        return NDList(data.manager.create(res, Shape(nbrRows, nbrCol)))

(I didn't find a method to concatenate NDArrays so I create a temporary FloatArray)

Don't hesitate if you need to know more details about my code to help me (I tried to be concise)

Also I have a question about Batchs, I saw in documentation/examples I have to use batch.close() in the training phase like in my code sample. Is it possible to save the batch in my dataset class to avoid same computations in next epochs even if I close it ?

EDIT: my bad I finally found a method to concat NDarrays, but I think it's not important.

And I forgot to say, I have a custom dataset, but it has 3 parts: a global one (it's what you see with multiDataset) which has a collection of dataset (herited class from dataset) and finally, getData returns a custom `Iterable' (it's where I compute batches)

EDIT2: I tried to remove the try(NDManager manager = NDManager.newBaseManager()){ my training function .... } because I saw I didn't use the MDManager argument in getData so I tested to use this parameter, I tried to use it to create subManager for each Iterable<Batch> I still have the same error

My code in dataset child class:

    override fun getData(manager: NDManager?): Iterable<Batch> {
        return PathIterable(this, manager)
    }

The constructor of my Iterable<Batch>

    private class PathIterable(
        private val pathDataset: PathDataset,
        managerGlobal: NDManager?
    ) : Iterable<Batch>, Iterator<Batch> {

        private val manager: NDManager
        var indexBatch = 0

        init {
            require(managerGlobal != null)

            this.manager = managerGlobal.newSubManager()
        }

The most important thing to see here is the attribute manager of the object is the subManager from the "global manager" from getData

EDIT3: If you can't read kotlin code I can easily transform it in java with Intellij :-)

[Isdriai]

I found the solution, I had to remove batch.close()

Thanks for the help =)