Memory Allocation

[AKMourato]

Greetings. I'm using a standard UNet on 4K images with a supposed resize to 800px and I'm still getting cuda out of memory with an 8GB GPU.
Are the transformations presented in the config file just for augmentation purposes or also for a normal network entry point?

Is it possible to perform Model Parallel Training? Where the model is split into different GPUs, therefore allowing more capacity?
i.e: https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html

python train.py --config unet.yml --do_eval --use_vdl --save_interval 10000 --save_dir output
2021-08-22 19:03:17 [INFO]
------------Environment Information-------------
platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.17
Python: 3.8.11 (default, Aug  3 2021, 15:09:35) [GCC 7.5.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.4.r11.4/compiler.30188945_0
cudnn: 7.6
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: NVIDIA GeForce', 'GPU 1: NVIDIA GeForce', 'GPU 2: NVIDIA GeForce', 'GPU 3: NVIDIA GeForce']
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PaddlePaddle: 2.1.2
OpenCV: 4.5.3
------------------------------------------------
2021-08-22 19:03:17 [INFO]
---------------Config Information---------------
batch_size: 1
iters: 60000
loss:
  coef:
  - 1
  types:
  - ignore_index: 255
    type: CrossEntropyLoss
lr_scheduler:
  end_lr: 0
  learning_rate: 0.0003
  power: 0.9
  type: PolynomialDecay
model:
  num_classes: 2
  pretrained: null
  type: UNet
  use_deconv: false
optimizer:
  momentum: 0.9
  type: sgd
  weight_decay: 4.0e-05
train_dataset:
  dataset_root: /home/amourato/VM/New_COCO_DPD/train/
  mode: train
  num_classes: 2
  train_path: /home/amourato/VM/New_COCO_DPD/train/train.txt
  transforms:
  - target_size:
    - 800
    - 800
    type: Resize
  - max_scale_factor: 2.0
    min_scale_factor: 0.5
    scale_step_size: 0.25
    type: ResizeStepScaling
  - type: RandomHorizontalFlip
  - brightness_range: 0.1
    contrast_range: 0.1
    saturation_range: 0.1
    type: RandomDistort
  - type: Normalize
  type: Dataset
val_dataset:
  dataset_root: /home/amourato/VM/New_COCO_DPD/test/
  mode: val
  num_classes: 2
  transforms:
  - target_size:
    - 800
    - 800
    type: Resize
  - type: Normalize
  type: Dataset
  val_path: /home/amourato/VM/New_COCO_DPD/test/test.txt
------------------------------------------------
W0822 19:03:17.662544 587010 device_context.cc:404] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 11.4, Runtime API Version: 10.2
W0822 19:03:17.662566 587010 device_context.cc:422] device: 0, cuDNN Version: 7.6.
/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/nn/layer/norm.py:640: UserWarning: When training, we now always track global mean and variance.
  warnings.warn(
Traceback (most recent call last):
  File "train.py", line 190, in <module>
    main(args)
  File "train.py", line 169, in main
    train(
  File "/home/amourato/VM/paddle/PaddleSeg/paddleseg/core/train.py", line 188, in train
    logits_list = model(images)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/VM/paddle/PaddleSeg/paddleseg/models/unet.py", line 64, in forward
    x = self.decode(x, short_cuts)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/VM/paddle/PaddleSeg/paddleseg/models/unet.py", line 114, in forward
    x = self.up_sample_list[i](x, short_cuts[-(i + 1)])
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/VM/paddle/PaddleSeg/paddleseg/models/unet.py", line 154, in forward
    x = self.double_conv(x)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/container.py", line 98, in forward
    input = layer(input)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/VM/paddle/PaddleSeg/paddleseg/models/layers/layer_libs.py", line 51, in forward
    x = self._conv(x)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 902, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/nn/layer/conv.py", line 655, in forward
    out = F.conv._conv_nd(
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/nn/functional/conv.py", line 116, in _conv_nd
    out = nn.elementwise_add(pre_bias, bias, axis=channel_dim)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/layers/nn.py", line 11533, in elementwise_add
    return _elementwise_op_in_dygraph(
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/framework.py", line 227, in __impl__
    return func(*args, **kwargs)
  File "/home/amourato/anaconda3/envs/segm/lib/python3.8/site-packages/paddle/fluid/layers/nn.py", line 205, in _elementwise_op_in_dygraph
    out = op(x, y, 'axis', axis, 'use_mkldnn', use_mkldnn)
SystemError: (Fatal) Operator elementwise_add raises an paddle::memory::allocation::BadAlloc exception.
The exception content is
:ResourceExhaustedError: 

Out of memory error on GPU 0. Cannot allocate 156.250244MB memory on GPU 0, 7.844421GB memory has been allocated and available memory is only 83.750000MB.

Please check whether there is any other process using GPU 0.
1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model. 

 (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79)
. (at /paddle/paddle/fluid/imperative/tracer.cc:192)