Compiler integration of the Split and Concat operations.

[mdanilow]

The backends of these operations are now implemented as a part of the finn-hlslib repo, the PR is created (Xilinx/finn-hlslib#143).

• The Concat op code is not hardcoded in the compiler anymore and it now accepts different datatypes of inputs. The output datatype is inferred.
• The new Split custom op is added.
• HLS implementations are defined for a one clock cycle, without defining NUM_REPS, so the new timeout cppsim template is also added and utilized by these operations.
• InferDuplicateStreamsLayer() transform in src/finn/transformation/fpgadataflow/convert_to_hw_layers.py is fixed and every output of a fork node is checked (the transform used to be applying the transform just for the first output of a node)
• InsertFIFO() transform in src/finn/transformation/fpgadataflow/insert_fifo.py is fixed for the case of the last node in the graph being a fork node (the transform was breaking the graph in such a case)
• The tests/fpgadataflow/test_fpgadataflow_concat.py test now tests the case of inputs being different datatypes
• tests/fpgadataflow/test_fpgadataflow_split.py test added

[iksnagreb]

Hey @mdanilow, this looks promising and I am currently trying to integrate the Split operation to streamline and deploy the GLU activation function here, which seems to be successful, at least for a small isolated GLU function in simulation. This comes just at the right time, I was about to build it myself, but this might safe me a lot of time, thank you!

However, I still have a few questions or concerns regarding more generalized split/concat support, in particular aiming at full ONNX compliance:

• How difficult do you think would it be to extend this to splitting and concatenating along arbitrary axes, not just the last one?
• The InferSplitLayer transformation seems to target opset version 13 or above, right? That is where the split sizes are given as a second input tensor. QONNX currently still seems to prefer opset version 11 (where the split sizes are given as an attribute). This is not necessarily an issue, you can just manually set the opset version, but maybe we want to have some cleanup transformation converting earlier opsets to the format accepted by your transformation?
• Wouldn't it make sense to decouple the parallelism at the input and output side of the operators? For example, it seems like a completely unrolled input for the split, i.e., read all elements and distribute them to all output streams within one cycle, is currently not possible? I.e., splitting N elements to N outputs each of size 1 should be possible in a single cycle with SIMD=N, but the current implementation would do this in N cycles with SIMD=1 as SIMD cannot exceed the smallest output size, or do I miss something?