-
Notifications
You must be signed in to change notification settings - Fork 5
/
Changes
2110 lines (1409 loc) · 74.7 KB
/
Changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Revision history for Perl module MCE.
1.900 Mon Sep 09 20:30:00 EST 2024
* Improve MCE::Child exiting when signaled.
1.899 Fri Sep 06 00:15:00 EST 2024
* Fix for MCE::Child and MCE::Channel signal anomaly #24.
Thank you, @exodist.
1.898 Wed Aug 21 15:30:00 EST 2024
* Fix for MCE::Child #22, Can't call method "len" on an undefined value
during global destruction. Thanks, @exodist.
1.897 Wed Jun 19 22:00:00 EST 2024
* In scalar context, the init function in MCE Child and models Flow, Grep,
Loop, Map, Step, and Stream returns a guard to call finish automatically
upon leaving the { scope i.e. omitting finish }.
* Add out_iter_callback to MCE::Candy.
1.896 Tue Jun 11 16:00:00 EST 2024
* Weaken internal core MCE reference to reap workers automatically
upon leaving the { scope i.e. omitting shutdown }. Note: No change
to MCE models Flow, Grep, Loop, Map, Step, and Stream. Call finish
explicitly to reap workers.
This resolves the case when omitting calling either one of $mce->run(),
$mce->run(1), or $mce->shutdown() inside a scope, causing workers to
linger around until completion of the script.
1.895 Sun Jun 09 01:00:00 EST 2024
* Revert back to calling CORE::rand() to set the internal seed.
MCE and MCE::Child cannot assume the srand or setter function used
by the application for predictability.
1. https://perlmonks.org/?node_id=11159834
2. https://perlmonks.org/?node_id=11159827
* Add class methods MCE->seed and MCE::Child->seed to retrieve the seed.
1.894 Sun Jun 09 15:30:00 EST 2024
* Improve support for PDL.
1.893 Sat Jun 08 08:30:00 EST 2024
* Preserve functionality for older Perl, non-threads.
1.892 Sat Jun 08 08:00:00 EST 2024
* Remove check if spinning threads i.e. use_threads.
Predictable output matches non-threads for CORE,
Math::Prime::Util, and Math::Random::MT::Auto.
https://perlmonks.org/?node_id=11159834
1.891 Fri Jun 06 04:00:00 EST 2024
* Apply workaround for PDL::srand in MCE and MCE::Child. Thank you, PerlMonks.
https://www.perlmonks.org/?node_id=11159773
* Add PDL::srand (v2.062 ~ v2.089) and PDL::srandom (v2.089_01+).
* Call CORE::srand inside child processes, only.
1.890 Fri May 24 19:00:00 EST 2024
* Improve reaping completed workers in MCE::Child.
* Fix the _sprintf function, failing multiple arguments.
1.889 Wed Sep 13 18:00:00 EST 2023
* Add Android support. Thank you, Dimitrios Kechagias.
* Revert defer signal-handling in MCE::Channel (send2 method).
* Improve mutex synchronize (aka enter) with guard capability.
Thank you, José Joaquín Atria.
* Fix mutex re-entrant lock on the Windows platform.
* Add mutex guard_lock method.
1.888 Wed Jun 21 17:00:00 EST 2023
* Fix typos caught by lintian. Thank you, Étienne Mollier.
1.887 Fri Jun 09 08:00:00 EST 2023
* Fix typo in Queue dequeue_timed documentation.
Thank you, Łukasz Strzelecki.
1.886 Tue Jun 06 12:00:00 EST 2023
* Added dequeue_timed method to MCE::Queue.
* Fixed taint mode in MCE->printf and _sprintf.
* Improved reliability on the Windows platform.
1.885 Tue May 30 20:00:00 EST 2023
* Improved reliability on the Windows platform.
1.884 Thu Jan 05 10:00:00 EST 2023
* Disabled non-blocking dequeue_nb and recv_nb tests on the Windows platform.
Reason: Author cannot reproduce failing tests reported by CPAN Tester aero.
Copied nb tests to xt folder: nonblocking_channel.t and nonblocking_queue.t
1.883 Tue Jan 03 20:00:00 EST 2023
* Fix typo in MCE::Channel::SimpleFast documentation.
* Improve 05_mce_child.t test.
1.882 Fri Dec 02 21:00:00 EST 2022
* Added ABRT to the list of signals to trap in MCE::Signal.
* Added a guard to MCE::Core::Worker for checking if exited prematurely.
* Added init_relay and use_threads import options to MCE and MCE Models.
* Separated input mutexes from the rest of IPC for lesser latency.
* Auto-detect if init_relay is defined and set chunk_size to 1 in
MCE::Grep, MCE::Map, and MCE::Stream.
* Update the import function in MCE models, detecting if the caller
is another MCE module, to not export model functions.
* Update the error status if MCE::Child died due to receiving a signal.
* Improved reaping in MCE::Child, before creating a new child.
* Improved the timeout handler in MCE::Child and MCE::Mutex::Channel.
* Fixed private functions _quit and _trap not setting the return value.
1.881 Thu Oct 13 23:45:00 EST 2022
* Improved the private _parse_chunk_size function. For better
utilization of CPU cores in MCE::Grep, MCE::Map, and MCE::Stream,
processing small input sizes.
Previously, chunk_size => 'auto' equals 2 minimally.
Starting with MCE v1.881, 'auto' equals 1 minimally.
1.880 Mon Oct 10 04:00:00 EST 2022
* Improved reliability on the Windows platform.
* Improved MCE::Mutex::Channel::timedwait on the Windows platform.
* Improved MCE::Mutex::Channel performance on UNIX platforms.
* Resolved edge case in MCE::Child reaching deadlock.
1.879 Tue May 24 05:00:00 EST 2022
* Replace http with https in documentation and meta files.
* Call PDL::set_autopthread_targ(1); disables PDL auto-threading.
1.878 Sun Feb 20 06:45:00 EST 2022
* Fix for the fast channel implementations.
Thank you, twata for the test report.
1.877 Sun Feb 20 02:30:00 EST 2022
* Improved suppressing the PDL CLONE warning. Piddles should not be
naively copied into new threads.
* Added fast channel implementations optimized for non-Unicode strings.
The main difference is that these lack freeze-thaw serialization.
MCE::Channel::MutexFast, MCE::Channel::SimpleFast, and
MCE::Channel::ThreadsFast.
1.876 Thu Dec 02 18:00:00 EST 2021
* Allow percentage above 100% for max_workers in MCE.
* MCE::Child update.
Improved _ordhash.
Renamed JOINED to REAPED in code for better clarity.
Specify a percentage for max_workers.
Added t/05_mce_child_max_workers.t
1.875 Tue Nov 16 04:00:00 EST 2021
* Specify a percentage for max_workers.
Thank you, kcott@PerlMonks (Ken) for the idea.
https://www.perlmonks.org/?node_id=11134439
* Added t/03_max_workers.t
1.874 Tue Aug 18 16:00:00 EST 2020
* Improved MCE->yield when used together with MCE::Relay.
1.873 Sat Aug 01 16:00:00 EST 2020
* Removed unused variable in MCE::Mutex::Channel.
* Fixed typo in prior change log.
Replaced "completed" with "feature complete".
1.872 Sun Jun 14 22:30:00 EST 2020
* Added open to required dependencies.
* Set default encodings on standard filehandles in tests using UTF-8.
* Bumped minimal Perl version to 5.8.1.
* Bumped MCE version to 1.872 to align with MCE::Shared.
* The MCE project is feature complete.
1.868 Sun May 10 22:00:00 EST 2020
* Completed threads-like detach capability in MCE::Child.
* Resolved MCE::Channel failing when calling dequeue multiple times
on an ended channel.
* MCE->say, MCE->print, and MCE->printf now return 1.
1.867 Sun May 03 18:00:00 EST 2020
* Bug fix for UTF-8 issues during inter-process communication.
This update required undoing optimizations specific to scalar args.
Essentially, IPC involves serialization for everything going forward.
Install Sereal::Encoder and Sereal::Decoder for better performance
in Perl 5.8.8+.
* MCE options flush_stdout, flush_stderr, and flush_file now default to
enabled for the MCE->print, MCE->printf, and MCE->say output routines.
* Improved MCE::Child with threads-like detach capability. See POD.
* Improved IPC in MCE::Queue with permanent fast-like dequeue including
dequeue_nb. Going forward, the fast and barrier options are silently
ignored if specified (i.e. no-op).
* Improved IPC performance on Linux.
1.866 Sat Feb 08 21:00:00 EST 2020
* Bug fix for restart_worker, race condition introduced in 1.863.
Thank you, Oliver Gorwits for reporting the issue.
1.865 Wed Dec 25 18:00:00 EST 2019
* Bug fix for two-way IPC stalling on Windows in MCE::Channel::Threads.
See https://www.perlmonks.org/?node_id=11110612
* Remove the check for MSWin32 in MCE::Channel::Mutex. MCE::Channel since
the 1st release silently defaults to MCE::Channel::Threads on Windows.
* Small tweak to MCE::Signal.
1.864 Wed Dec 04 13:00:00 EST 2019
* Bug fix in MCE::Signal - Shared manager not exiting, introduced in 1.863.
* Use monotonic clock if available in MCE->yield and MCE::Child->yield.
See https://www.perlmonks.org/?node_id=11109673
1.863 Sun Nov 26 20:00:00 EST 2019
* On Cygwin, silently use Mutex in MCE::Channel when Threads is specified
for better performance.
* New defer capability in MCE::Signal. This applies to MCE::Shared 1.863.
See POD section labled "DEFER SIGNAL" in MCE::Signal.
* Reverted $child->exit back to sending the SIGQUIT signal in MCE::Child
now that MCE::Shared::Server 1.863 defers signal during IPC.
* Improved reliability for spawning MCE and MCE::Child inside threads
including nested parallelization, made possible using a global lock
$MCE::_GMUTEX.
* Updated signal handling in mce-examples/framebuffer on GitHub.
1.862 Wed Sep 18 22:00:00 EST 2019
* Hotfix for 1.861.
1.861 Wed Sep 18 08:30:00 EST 2019
* Hotfix for 1.849-1.860. The edge cases regarding signal handling have
been finally resolved for MCE::Child. Thank you, Richard Kelsch for the
use case involving ctrl-c. See mce-examples/framebuffer on GitHub.
1.860 Sun Sep 15 20:00:00 EST 2019
* Signal-handling update release.
* Localized input and output record separators in MCE::Channel.
* IPC safety in MCE::Child during SIGINT and SIGTERM.
* Method $child->exit in MCE::Child now sends the SIGINT signal
for extra reliability with MCE::Shared (previously SIGQUIT).
1.850 Mon Sep 09 12:30:00 EST 2019
* Bumped version to coincide with the stable MCE::Shared 1.850 release.
1.849 Sun Sep 08 23:30:00 EST 2019
* Fixed edge case in MCE::Child when reaping inside a signal handler.
* Added list_pids class method to MCE::Child.
1.848 Tue Sep 03 23:30:00 EST 2019
* Improved IO::All::{ File, Pipe, STDIO } output via MCE->print($io, ...),
printf, and say. This resolves a bug introduced in 1.845 when using
App::Cmd::Tester to capture output.
1.847 Mon Sep 02 23:30:00 EST 2019
* Obsolete RedHat MCE-1.840-Sereal-deps.patch file. This patch file
is no longer needed and finally resolved with this release.
* PDL random numbers now unique between threads. Thank you, PerlMonks (vr).
See https://www.perlmonks.org/?node_id=1214439.
* Replaced "PF_UNIX" with "AF_UNIX" in MCE::Util.
1.846 Mon Aug 26 21:30:00 EST 2019
* Fixed code tags in documentation. Thank you, Mohammad S Anwar.
1.845 Sun Aug 25 22:00:00 EST 2019
* Croak if is_joinable, is_running, or join is called by a non-managed
process in MCE::Child. Added LIMITATION section to the documentation.
* Improved is_joinable, is_running, list_joinable, and list_running
in MCE::Child. Thank you, Philippe Baumgart.
* Added example (Consumer requests item) to MCE::Channel documentation.
* Support the task_end option regardless if user_tasks is specified.
* Support IO::All::{ File, Pipe, STDIO } for input data including
output via MCE->print($io, ...), printf, and say.
* Support gather => MCE::Candy::out_iter_fh($io) using MCE::Candy.
1.844 Wed Aug 14 21:30:00 EST 2019
* Resolved MCE stalling when specifying max_retries with init_relay.
Ditto for loop_timeout with init_relay on UNIX platforms.
Thank you, Chris Denley.
* Enhanced loop_timeout to handle workers dieing uncontrollably from
any user_tasks (i.e. task_id >= 0). Previously, only task_id == 0.
* Improved IPC on the Windows platform for edge case when a worker is
awaiting input while the manager process is restarting a worker.
* MCE, MCE::Child workers exit immediately upon receiving a SIGSEGV signal.
This safeguards IPC from stalling inside the manager process.
* Enhanced the _wait_one private function in MCE::Child.
* Removed Prima from the list for auto-enabling the posix_exit option.
Prima (since 1.52) is parallel safe during global cleanup.
* Reached 100% Pod coverage.
1.843 Tue Jul 23 22:30:00 EST 2019
* Updated results in MCE::Child (Parallel::ForkManager-like demonstration).
* Completed missing interrupt signal-safety for the non-blocking methods
in MCE::Channel::Mutex and MCE::Channel::Threads.
1.842 Sun Jul 21 19:00:00 EST 2019
* Fixed race condition abnormalities in MCE::Child.
* Added Parallel::ForkManager-like demonstration to MCE::Child.
1.841 Sun Jul 07 23:30:00 EST 2019
* Disabled t/04_channel_threads testing on Unix platforms for Perl less than
5.10.1. Basically, the MCE::Channel::Threads implementation is not supported
on older Perls unless the OS vendor applied upstream patches (i.e. works on
RedHat/CentOS 5.x running Perl 5.8.x).
* Added LIMITATIONS section to MCE::Channel::Threads.
1.840 Sun Jul 07 05:00:00 EST 2019
* Update MCE::Channel POD documentation.
1.839 Sun Jul 07 04:30:00 EST 2019
[NEW FEATURES]
* Added MCE::Mutex::Channel2 providing two locks using a single channel.
The secondary lock is accessible by calling methods with the '2' suffix.
E.g. primary mutex ->lock, ->unlock; secondary mutex ->lock2, ->unlock2
* Added MCE::Channel providing queue-like and two-way communication
supporting threads and processes.
* Added MCE::Child and compatibility with Perl 5.8. MCE::Child is based on
MCE::Hobo, but using MCE::Channel for data retrieval without involving
a shared-manager process.
* Added MCE::Channel examples { channel1.pl and channel2.pl } using
threads and MCE::Child respectively.
https://github.com/marioroy/mce-examples/tree/master/chameneos
[ENHANCEMENTS]
* IPC update; removed unnecessary overhead including private methods
_sysseek and _syswrite from MCE::Util (no longer needed).
* Improved MCE->do, now callable by workers and the manager process.
* Updated MCE::{ Flow, Grep, Loop, Map, Step, and Stream } documentation
on passing an array reference versus a list for deeply input data.
* Updated and re-organized the top-level MCE documentation, particularly
improved clarity for the 'MCE Models' section.
* Removed MANIFEST.SKIP.
1.838 Wed Jan 23 08:30:00 EST 2019
* IPC update, raising reliability across multiple platforms.
* Improved hack for the Windows platform for nested MCE sessions.
* Added _sysread, _sysseek, _syswrite, and _nonblocking to MCE::Util.
* Added barrier option to MCE::Queue allowing one to disable.
1.837 Sat Aug 25 13:00:00 EST 2018
* Seeds the Math::Random::MT::Auto generator automatically when present
for non-threads, similarly to Math::Random and Math::Prime::Util, to
avoid child processes sharing the same seed value as the parent and
each other. The new seed is computed using the current seed.
1.836 Mon Jun 25 08:30:00 EST 2018
* Moved validation code from MCE::Util to MCE::Core::Validation.
* Applied small optimizations.
1.835 Tue Mar 13 15:00:00 EST 2018
* Added gather and relay demonstrations to MCE::Relay.
* Load IO::Handle for extra stability, preventing workers loading uniquely.
* Load Net::HTTP and Net::HTTPS before spawning if present LWP::UserAgent.
See https://www.perlmonks.org/?node_id=1199760
and https://www.perlmonks.org/?node_id=1199891.
1.834 Tue Jan 23 08:00:00 EST 2018
* Improved Queue await and dequeue performance on the Windows platform.
* Added chameneos-redux parallel demonstrations on GitHub:
https://github.com/marioroy/mce-examples/tree/master/chameneos
1.833 Thu Dec 28 16:00:00 EST 2017
* Fixed bug with sequence (#10), broken in 1.832. Thank you, @djerius.
1.832 Tue Nov 21 15:30:00 EST 2017
* Added LWP::UserAgent to list for enabling posix_exit.
* Improved number-sequence generation for big integers.
* Improved wantarray support in MCE::Mutex synchronize.
* Removed limit check on chunk_size option.
1.831 Sun Oct 08 20:30:00 EST 2017
* Added STFL (Terminal UI) to list for enabling posix_exit.
See https://www.perlmonks.org/?node_id=1200923.
* Math::Prime::Util random numbers now unique between MCE workers.
See https://www.perlmonks.org/?node_id=1200960.
1.830 Tue Sep 12 17:00:00 EST 2017
[BUG FIXES]
* Fixed MCE and MCE::Relay stalling when setting the input record separator.
See https://www.perlmonks.org/?node_id=1196701. Thank you, JediWombat.
* Fixed bug with dequeue_nb in MCE::Queue (#8). Thank you, @bokutin.
* Fixed signal handler (#9). Thank you, @chrisdenley.
[ENHANCEMENTS]
* Added Coro and Win32::GUI to list for enabling posix_exit.
* Added support for Haiku to get_ncpu in MCE::Util.
* Allow gathering to a shared array in MCE::Candy.
* Improved CPU count on the AIX platform in MCE::Util.
* Improved signal handling, including nested parallel-sessions.
* Improved running MCE::Hobo inside MCE workers.
* Improved running MCE with PDL.
* Refactored logic for MCE->do, bi-directional callback feature.
* Preserve lexical type for numbers during IPC: MCE->do and MCE::Queue.
* No longer loads threads on the Windows platform in MCE::Signal.
This enables MCE::Hobo 1.827 to spin faster, including lesser
memory consumption. Threads isn't required to run MCE::Hobo.
* Removed extra white-space from POD documentation.
* Validated MCE on SmartOS.
1.829 Wed May 03 03:00:00 EST 2017
* Reduced memory consumption.
1.828 Fri Apr 28 16:00:00 EST 2017
[BUG FIXES]
* Do not enable barrier mode for Queue on the Windows platform.
* Fixed MCE::Mutex::Flock, tmp_file missing script name in path.
[ENHANCEMENTS]
* Added Curses and Prima to list for enabling the posix_exit option.
* Allow a hash as input_data: Core API, MCE::{ Flow, Loop, Step }.
* Improved API documentation on MCE models with more synopsis.
* Enhanced IPC and signal handling. Reduced memory consumption.
* Make tmp_dir on demand in MCE::Signal. Ditto for sess_dir in MCE.
* Load Fcntl, File::Path, Symbol on demand.
1.827 Wed Apr 05 01:30:00 EST 2017
* Do not enable barrier mode for Queue if constructed inside a thread
or by MCE models (e.g. Step, Stream). Ditto for fast => 1 option.
* Updated MCE to not croak when running Perl in taint mode via perl -T.
Failing -T was MCE::Core::Input::{ Generator, Sequence }, MCE::Signal,
and MCE::Util.
* Added Denis Fateyev, Felipe Gasper, and Paul Howarth to Credits.
1.826 Sun Apr 02 23:00:00 EST 2017
* Is now safe running MCE with the Wx GUI toolkit (wxWidgets).
1.825 Sun Apr 02 07:00:00 EST 2017
* Updated MCE::Queue. The following provides a comparison for the
enhancements made regarding IPC during 1.822 through 1.825, in order
to run on machines having "many" cores. I ran with 12, 96, and 192
workers on an 8 core box.
A. MCE queue, dequeue 100k items.
my $Q = MCE::Queue->new();
$Q->enqueue( 1 .. 100000 );
$Q->end(); MCE 1.608 $Q->enqueue((undef) x 12, 96, or 192);
MCE->new(
max_workers => 12, 96, or 192,
user_func => sub {
while ( defined ( my $item = $Q->dequeue ) ) {
;
}
}
)->run();
MCE 1.608: 12 ~ 1.799 secs 96 ~ 8.702 secs 192 ~ 18.083 secs
MCE 1.821: ~ 1.450 secs ~ 5.231 secs ~ 8.102 secs
MCE 1.825: ~ 0.976 secs ~ 1.067 secs ~ 1.509 secs
B. Input file containing 250k lines ( 300 MiB ).
MCE->new(
input_data => "/dev/shm/file_250k.txt",
max_workers => 12, 96, or 192,
chunk_size => 1,
use_slurpio => 1,
user_func => sub { }
)->run();
MCE 1.608: 12 ~ 3.605 secs 96 ~ 8.074 secs 192 ~ 8.465 secs
MCE 1.821: ~ 3.613 secs ~ 8.058 secs ~ 8.607 secs
MCE 1.825: ~ 3.567 secs ~ 2.601 secs ~ 3.199 secs
C. Sequence of numbers from 1 to 200k.
MCE->new(
sequence => [ 1, 200000 ],
max_workers => 12, 96, or 192,
chunk_size => 1,
user_func => sub { }
)->run();
MCE 1.608: 12 ~ 1.236 secs 96 ~ 2.922 secs 192 ~ 3.113 secs
MCE 1.821: ~ 1.227 secs ~ 2.915 secs ~ 3.106 secs
MCE 1.825: ~ 1.250 secs ~ 1.203 secs ~ 1.581 secs
* Results were captured on a fast 8 core system running CentOS Linux 7.
The thing to take from this is that running many workers "no longer"
results in up to 5.6x penalty regarding IPC.
1.824 Sat Apr 01 01:00:00 EST 2017
* Completed validation for running MCE on a box having 100+ cores.
* Tuned the number of data-channels for IPC. Set upper limit in
MCE::Core::Input::{ Handle and Sequence } to not impact the
OS kernel. The result is better performance, yet graceful.
1.823 Fri Mar 31 19:30:00 EST 2017
* Calibrated the number of data-channels for IPC.
1.822 Fri Mar 31 11:00:00 EST 2017
* Check for EINTR during sysread and syswrite.
* Improved reliability when running nested MCE sessions.
* Updated MCE::Mutex with Channel and Fcntl implementations.
* Completed validation for using MCE with 200+ cores.
1.821 Sun Mar 19 04:00:00 EST 2017
* Improved reliability when running MCE with threads.
* Added parallel Net::Pcap and Ping demonstrations on GitHub:
https://github.com/marioroy/mce-examples/tree/master/network
* Optimized 'dequeue' method in MCE::Queue.
* Optimized 'synchronize' method in MCE::Mutex.
1.820 Thu Mar 09 02:00:00 EST 2017
* Improved reliability when running MCE inside an eval block.
1.819 Fri Mar 03 23:00:00 EST 2017
* Fixed issue with localizing AUTOFLUSH variable before autoflush handles.
Thank you, Charles Hendry for raising the issue.
1.818 Wed Mar 01 22:00:00 EST 2017
* Updated bin/mce_grep for determining chunk level. Ditto for chunk size.
Fixed an issue for not seeing STDERR output with '--chunk-level=file'.
Added support for zgrep, zegrep, and zfgrep. Thank you, Jeff Rouse.
* Changed Sereal to Sereal::Decoder and Sereal::Encoder in recommends
section inside Makefile and META files.
* Refactored MCE::Queue. Merged local and manager code base into one.
Removed t/04_norm_que_local.t and t/04_prio_que_local.t.
* Added 'end' method to MCE::Queue.
Updated documentation on dequeue and pending.
1.817 Sat Feb 25 02:00:00 EST 2017
* Improved bin/mce_grep. When -r is specified and zero paths are given,
start recursively in the current directory versus await data from STDIN.
Set chunk-level accordingly to list mode.
1.816 Fri Feb 24 19:00:00 EST 2017
* Revised the description on max_retries in MCE::Core.pod.
1.815 Fri Feb 24 01:00:00 EST 2017
* Fixed divide-by-zero error in MCE->yield.
* Refactored code for the interval option by moving the code to the manager
process. This allows the manager process to accomodate the next available
worker and ready to run. Previoulsy, a worker taking a long time resulted
in empty time slots. Thank you, Philippe Baumgart for your patience.
* Revised the description on posix_exit in MCE::Core.pod.
1.814 Mon Feb 20 05:30:00 EST 2017
* Enhanced the progress option for use with MCE->process.
Updated demonstrations in MCE::Core.pod.
1.813 Thu Feb 16 02:30:00 EST 2017
* Last minute request by Philippe Baumgart (reminder and long overdue).
Added progress option, a code block for receiving info on progress made.
See MCE::Core.pod for demonstrations accommodating all input data types.
1.812 Tue Feb 14 17:00:00 EST 2017
* Bumped minimum requirement for Sereal to 3.015 when available.
Added check ensuring matching version for Encoder and Decoder.
1.811 Mon Feb 13 23:30:00 EST 2017
* Fixed bug in MCE::Queue (dequeue_nb) when queue has zero items.
* Applied small optimization in MCE::Core::Input::Sequence and Generator.
* Added cross-platform template to MCE::Examples for making an executable.
* Removed signal handling for XCPU and XFSZ from MCE::Signal.
* Imply posix_exit => 1 if Gearman::XS or Gearman::Util is present during
MCE construction.
* Added MCE + Gearman demonstrations (xs and non-xs) on GitHub:
https://github.com/marioroy/mce-examples/tree/master/gearman_xs
https://github.com/marioroy/mce-examples/tree/master/gearman
* Changed kilobytes and megabytes to kibiBytes (KiB) and mebiBytes (MiB)
respectively inside the documentation.
1.810 Fri Dec 09 23:30:00 EST 2016
* Updated check for IO handle allowed. This allows $gather_fh = *STDOUT{IO},
construction in Perl <= 5.10.1. Thank you, Qiang Wang.
1.809 Wed Nov 23 16:00:00 EST 2016
* Bug fixes for running MCE inside threads.
* Random numbers are unique between workers.
1.808 Sat Nov 05 02:00:00 EST 2016
* Workers persist unless shutdown explicity while running alongside
the Mojolicious framework.
1.807 Tue Nov 01 16:00:00 EST 2016
* Enhanced relay capabilities. Added Mandelbrot example to MCE::Example.
Added extra demonstrations to MCE::Relay. Also, added test script.
* Tweaked manager-loop delay for special cases -- applies to MSWin32 only.
1.806 Tue Oct 11 21:30:00 EST 2016
* Fixed two typos. Thank you, Florian Schlichting.
* Support input_data with nested arrays in MCE models.
1.805 Thu Sep 01 16:00:00 EST 2016
* Fixed bug in MCE::Queue (#4). Thank you, Mary Ehlers.
* Improved support for running MCE with Tk. Added Tk demonstrations to
MCE::Examples. Thank you, Götz Meyer.
1.804 Thu Jul 28 23:00:00 EST 2016
* Removed the sleep statement in MCE->restart_worker.
* Added FCGI::ProcManager demonstrations to MCE::Examples.
* Automatically set posix_exit to 1 whenever (F)CGI.pm is present.
* Thank you Kai Wasserbäch (TheRealCuran) for the cool test case.
https://github.com/marioroy/mce-perl/issues/1
1.803 Sun Jul 10 23:30:00 EST 2016
* Re-enabled Sereal 3.008+ for Perl < v5.12.0, if available.
* Optimized dequeue methods in MCE::Queue.
1.802 Mon Jul 04 03:30:00 EST 2016
* Default to Storable for serialization in Perl less than v5.12.0.
Sereal 3.008+, if available, is loaded automatically in Perl v5.12+.
1.801 Sun Jul 03 00:30:00 EST 2016
[BUG FIXES]
* Fixed race condition in Queue->await.
[ENHANCEMENTS]
* MCE 1.801 is stable on all supported platforms.
* Completed work supporting cyclical include of MCE Core / models.
* Updated MCE to support Perl included with Git Bash.
* Renamed temp dir from 'mce' to 'Perl-MCE' under user's %TEMP%
location on Windows. E.g. Native Perl, Cygwin, Git Bash.
1.800 Sat Jun 18 16:30:00 EST 2016
[BUG FIXES]
* Fixed dequeue (count) in MCE::Queue for standalone mode.
[ENHANCEMENTS]
* On Windows, improved stablity and feature parity with UNIX.
* Use Sereal 3.008+ automatically if available on the box.
[NEW FEATURES]
* Added support for cyclical include of MCE Core, MCE models, and
MCE Queue by scoping the configuration to the local package.
This resolves (RT#107384), bug reported by Kai Wasserbäch.
1.708 Sat May 28 14:00:00 EST 2016
[BUG FIXES]
* Improved import routine in MCE models and MCE::Subs. This resolves an
issue where functions are not exported; e.g. mce_flow, mce_flow_s.
[ENHANCEMENTS]
* Added support for IO::TieCombine handles. This enables MCE->print and
MCE->sendto to work reliably with App::Cmd and App::Cmd::Tester.
See Testing and Capturing Output in MCE::Examples.
1.707 Wed May 25 16:00:00 EST 2016
[BUG FIXES]
* Fixed logic when workers exit. Improved reliability on Windows.
[ENHANCEMENTS]
* Applied MCE-1.700-provides.patch from RedHat. Thank you, Paul Howarth.
* Added META.json to the distribution.
1.706 Fri Apr 22 21:30:00 EST 2016
[ENHANCEMENTS]
* Time::HiRes sleep resolution is 15 milliseconds on Windows and Cygwin.
Adjusted timeout values accordingly. Thank you, Daniel Dragan.
* Reinstated the hack for faster IO when use_slurpio => 1 is specified.
Tuned chunk_size => 'auto'.
1.705 Thu Apr 14 10:00:00 EST 2016
* Bumped version for Test::More to 0.88. Thank you, Paul Howarth.
1.704 Thu Apr 14 05:00:00 EST 2016
[BUG FIXES]
* Fixed restart on the Windows platform, bug introduced in 1.700.
* Reached *stable* on all major platforms for MCE 1.7x.
[ENHANCEMENTS]
* Enabled auto-destroy for MCE objects.
* Enabled freeze callbacks for Sereal.
* Switched bug tracking to GitHub.
* Tweaked test scripts.
1.703 Sat Mar 19 23:00:00 EST 2016
* Completed IPC optimizations for 1.7.
1.702 Tue Mar 15 17:00:00 EST 2016
* Bumped version.
1.701 Tue Mar 15 12:00:00 EST 2016
[ENHANCEMENTS]
* Some folks have expressed a wish for running MCE 1.7 with Perl v5.8.
To restore support for Perl v5.8, removed MCE::Shared and MCE::Hobo
from the MCE 1.701 distribution.
* MCE::Shared will be released after MCE 1.700 is deleted from CPAN.
Thank you for your patience during this transition.
1.700 Tue Mar 08 15:30:00 EST 2016
[BUG FIXES]
* Fixed race condition on Windows for non-threaded workers.
* Updated MCE models to not fail when running inside an eval statement.
This addresses (RT#105557) and (RT#105559). Thank you, Benjamin McKeown.
* Added new MCE option loop_timeout to prevent the MCE Manager process
from hanging perpetually. The manager process wrongly assumes a worker
is still running when the worker died in an uncontrollable manner.
This resolves (RT#111780). Thank you, Benjamin McKeown.
[ENHANCEMENTS]
* Perl 5.10.1 or later is required to run MCE 1.7. Perl < 5.10.1 lacks
'overloading.pm'.
* Added code in MCE::Grep's documentation for parsing huge files.
* Added support for running MCE with Perl under MobaXterm on Windows.
* MCE/examples and MCE/images are no longer included with the distribution.
These are maintained separately at https://github.com/marioroy/mce-examples
and https://github.com/marioroy/mce-assets respectively.
* MCE performs channel locking via a pipe or socket depending on platform.
Previously, locking was through file locking using flock. This resolves
the slow locking performance on Cygwin.
* Optimized signal handling including improved support on Windows.
* Reduced overhead during spawning and job submissions on Windows and Cygwin.
This enables IPC to complete up to 20x faster, thus benefiting Monte-Carlo
simulations; e.g. calling ->run(0) or ->process(...) repeatedly.
* The MCE::Flow and MCE::Step models can take an anonymous array for
specifying use_threads uniquely for sub-tasks.
[NEW FEATURES]
* Added MCE::Hobo for running code asynchronously. This provides async/join
functionality for processes similarly to async/join in threads. It includes
->is_joinable, ->is_running, ->join, ->kill, ->list, ->waitall, ->waitone,
and other methods not metioned here.
* Added MCE::Shared for sharing objects/data between threads/processes.
* Added MCE::Shared::{ Array, Handle, Hash, Ordhash, and Scalar }.
* Added MCE::Shared::{ Condvar, Minidb, Queue, and Sequence }.
* Added MCE::Shared::{ Server }.
* Added methods ->await, ->enq, and ->enqp to MCE::Step.
* Added method ->await to MCE::Queue.
* Added option max_retries => N for retrying a failed chunk from a worker
dying while processing input data or sequence of numbers.
* Added option posix_exit => 1 to avoid END and destructor processing.
This is necessary for running with Tk and child processes or with
use_threads => 0.
* Seeds the Math::Random generator automatically when present for non-threads
to avoid child processes sharing the same seed value as the parent and
each other. The new seed is computed using the current seed. Thus, okay
to set the seed at the application level for predictable results.
1.608 Fri Apr 10 03:00:00 EST 2015
* Correction for $prog_name (-e) to (perl) in MCE::Signal.
1.607 Fri Apr 10 01:00:00 EST 2015
[BUG FIXES]
* Updated t/01_load_signal_arg.t to address false-positive from bug fix
in 1.606. This was missed in the 1.606 release. Thank you Nigel Horne.
[ENHANCEMENTS]
* Replaced (-e) with (perl) for the $prog_name value in MCE::Signal;
e.g. from running perl -e 'command'.
1.606 Wed Apr 08 18:00:00 EST 2015
[BUG FIXES]
* Added -d and -w tests to ensure $ENV{TEMP} exists and writeable in
MCE::Signal. Otherwise /tmp is used as usual.
[ENHANCEMENTS]
* Determine running state in MCE->exit. Call stop_and_exit if not
already running to not hang the script.
1.605 Mon Apr 06 00:30:00 EST 2015
[BUG FIXES]
* Improved fix for the die handler in MCE::Signal and MCE::Core::Worker.
* Improved support for threads in MCE::Signal's stop_and_exit function.
1.604 Sat Mar 21 21:00:00 EST 2015
[BUG FIXES]
* All bugs found during testing of the upcoming 1.7 release have been
backported to the 1.6 branch.
[NEW FEATURES]
* Added out_iter_array and out_iter_fh to MCE::Candy. These preserve output
order and cover the two general use cases.
1.603 Tue Mar 17 21:00:00 EST 2015
[ENHANCEMENTS]
* A safer solution by Dmitry Karasik for the die handler in MCE::Signal
and MCE::Core::Worker.
* Moved ->forchunk, ->foreach, and ->forseq sugar methods to MCE::Candy.
Stubs exist in MCE. Thus, no breakage to existing apps.
* Removed the link to MCE::Shared on the main page. I decided to backport
all the fixes into 1.6. The MCE::Shared link was missed and requires
the upcoming 1.7 release.
1.602 Mon Mar 16 20:00:00 EST 2015
[BUG FIXES]
* Updated die handler in MCE::Core::Worker and MCE::Signal (RT#102802)
Bug reported by Dmitry Karasik.
[ENHANCEMENTS]
* MCE child processes call CORE::exit during exiting; previously CORE::kill.