Extracting Arabic embeddings

[juman98]

Hello, I'm new to transformers, I'm trying to extract the embeddings for a custom dataset, I'm not sure if the implementation is right and honestly I'm not sure if this is normal to happen, I was trying to retrieve the last hidden state so I could use them later on, so could the problem be the implementation of my whole code, or could it actually be as it said?

Here's the snippet of the code:

and here's the error:

[WissamAntoun]

you are probably passing all the data at the same time. I suggest sending the sentences in batches or 1-by-1

[juman98]

I actually am passing them all, could you suggest me away for using batches?

[WissamAntoun]

import more_itertools
from tqdm import tqdm_notebook as tqdm

last_hidden_states = []
for batch in tqdm(more_itertools.chunked(sentences, 32)):
  #process the sentences here then pass through the model
  processed_batch = tokenizer(batch, return_tensors 'pt)
  with torch.no_grad():
    last_hidden_states.extend(model(**processed_batch))

this should work