Skip to content

Commit

Permalink
set a cap on parallel reading
Browse files Browse the repository at this point in the history
  • Loading branch information
Baunsgaard committed Oct 21, 2024
1 parent 96bcf72 commit a2fca3d
Showing 1 changed file with 0 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ protected void readCSVFrameFromHDFS( Path path, JobConf job, FileSystem fs,
throws IOException
{
Timing time = new Timing(true);
// final int numThreads = 4 ; // InfrastructureAnalyzer.getLocalParallelism() / 2;
final int numThreads = Math.min(32, OptimizerUtils.getParallelTextReadParallelism());

TextInputFormat informat = new TextInputFormat();
Expand Down

0 comments on commit a2fca3d

Please sign in to comment.