Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add on-device test results? #2

Open
cbowns opened this issue Aug 7, 2019 · 4 comments
Open

Add on-device test results? #2

cbowns opened this issue Aug 7, 2019 · 4 comments

Comments

@cbowns
Copy link

cbowns commented Aug 7, 2019

I just built this project and its tests in Release mode, and ran it on an iPhone Xs Max running iOS 13 beta 5 (build 17A5547d). Here's my results, if you'd like to add them to the project/post.

2019-08-07 11:13:01.663002-0700 SpeedTest[686:332338] Launching with XCTest injected. Preparing to run tests.
2019-08-07 11:13:01.678508-0700 SpeedTest[686:332338] Waiting to run tests until the app finishes launching.
Test Suite 'All tests' started at 2019-08-07 11:13:01.831
Test Suite 'SpeedTestTests.xctest' started at 2019-08-07 11:13:01.831
Test Suite 'CombineTests' started at 2019-08-07 11:13:01.831
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' measured [Time, seconds] average: 0.317, relative standard deviation: 6.035%, values: [0.373770, 0.315254, 0.311101, 0.313436, 0.308780, 0.308293, 0.310991, 0.309422, 0.308666, 0.308311], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' passed (3.484 seconds).
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' measured [Time, seconds] average: 0.254, relative standard deviation: 6.278%, values: [0.301132, 0.246959, 0.246933, 0.246930, 0.247169, 0.256698, 0.248910, 0.247402, 0.247251, 0.251685], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' passed (2.793 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.174, relative standard deviation: 9.541%, values: [0.223468, 0.168516, 0.168904, 0.168670, 0.168296, 0.168805, 0.168205, 0.167899, 0.166943, 0.167873], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' passed (1.990 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.666, relative standard deviation: 2.373%, values: [0.713103, 0.658805, 0.659432, 0.661265, 0.660234, 0.660054, 0.661817, 0.661364, 0.660532, 0.661145], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' passed (6.909 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' measured [Time, seconds] average: 0.119, relative standard deviation: 13.664%, values: [0.167399, 0.113666, 0.113355, 0.113330, 0.113366, 0.113154, 0.113349, 0.113165, 0.113210, 0.113298], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' passed (1.439 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' measured [Time, seconds] average: 0.368, relative standard deviation: 4.310%, values: [0.415549, 0.362704, 0.363176, 0.361723, 0.362332, 0.363070, 0.363443, 0.363041, 0.362221, 0.362598], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' passed (3.932 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' measured [Time, seconds] average: 0.114, relative standard deviation: 15.577%, values: [0.166588, 0.107760, 0.107645, 0.107777, 0.107944, 0.107716, 0.107834, 0.107345, 0.107410, 0.107330], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' passed (1.387 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' measured [Time, seconds] average: 0.146, relative standard deviation: 11.349%, values: [0.195804, 0.140595, 0.140445, 0.140486, 0.140395, 0.140572, 0.140685, 0.140616, 0.140502, 0.140615], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' passed (1.713 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.217, relative standard deviation: 8.163%, values: [0.269519, 0.210303, 0.210658, 0.210114, 0.210629, 0.210665, 0.210536, 0.210768, 0.210886, 0.210952], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' passed (2.417 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.126, relative standard deviation: 13.366%, values: [0.176685, 0.120584, 0.120393, 0.120398, 0.120567, 0.120582, 0.120625, 0.120456, 0.120588, 0.120272], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' passed (1.513 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.213, relative standard deviation: 8.093%, values: [0.264186, 0.206561, 0.206730, 0.206495, 0.206359, 0.206669, 0.206638, 0.206718, 0.206959, 0.208698], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' passed (2.378 seconds).
Test Suite 'CombineTests' passed at 2019-08-07 11:13:31.788.
	 Executed 11 tests, with 0 failures (0 unexpected) in 29.955 (29.956) seconds
Test Suite 'RxSwiftTests' started at 2019-08-07 11:13:31.788
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' measured [Time, seconds] average: 0.424, relative standard deviation: 4.463%, values: [0.480355, 0.417623, 0.417502, 0.417144, 0.417412, 0.417046, 0.417352, 0.417095, 0.417495, 0.417291], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' passed (4.489 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' measured [Time, seconds] average: 0.244, relative standard deviation: 7.176%, values: [0.297019, 0.238397, 0.238588, 0.238113, 0.239896, 0.240161, 0.238805, 0.236376, 0.238680, 0.238684], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' passed (2.697 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.166, relative standard deviation: 11.201%, values: [0.221616, 0.159861, 0.159359, 0.159367, 0.159202, 0.160056, 0.159847, 0.159774, 0.159760, 0.159958], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' passed (1.911 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.569, relative standard deviation: 2.964%, values: [0.619199, 0.562502, 0.563592, 0.561449, 0.563635, 0.560609, 0.563452, 0.563670, 0.563945, 0.565810], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' passed (5.940 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' measured [Time, seconds] average: 0.158, relative standard deviation: 10.053%, values: [0.205639, 0.153024, 0.152946, 0.152780, 0.152942, 0.152866, 0.152381, 0.152329, 0.152604, 0.152438], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' passed (1.832 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' measured [Time, seconds] average: 0.522, relative standard deviation: 3.159%, values: [0.570750, 0.517117, 0.514583, 0.515876, 0.517330, 0.514596, 0.515429, 0.516117, 0.519115, 0.514222], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' passed (5.467 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' measured [Time, seconds] average: 0.130, relative standard deviation: 12.899%, values: [0.179818, 0.124204, 0.124040, 0.124229, 0.124294, 0.123647, 0.124333, 0.124075, 0.123292, 0.124630], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' passed (1.549 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' measured [Time, seconds] average: 0.129, relative standard deviation: 13.754%, values: [0.182531, 0.123416, 0.123589, 0.123505, 0.123656, 0.123497, 0.123388, 0.121803, 0.123525, 0.123406], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' passed (1.544 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.219, relative standard deviation: 8.262%, values: [0.273149, 0.212711, 0.212802, 0.212842, 0.212670, 0.212838, 0.212886, 0.213015, 0.213080, 0.212935], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' passed (2.441 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.203, relative standard deviation: 8.249%, values: [0.252791, 0.197036, 0.196841, 0.197075, 0.196869, 0.197041, 0.196938, 0.197119, 0.197845, 0.196916], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' passed (2.278 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.361, relative standard deviation: 4.768%, values: [0.412536, 0.357086, 0.356588, 0.355424, 0.354889, 0.354509, 0.355141, 0.353818, 0.354854, 0.354913], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' passed (3.862 seconds).
Test Suite 'RxSwiftTests' passed at 2019-08-07 11:14:05.797.
	 Executed 11 tests, with 0 failures (0 unexpected) in 34.008 (34.009) seconds
Test Suite 'SpeedTestTests.xctest' passed at 2019-08-07 11:14:05.797.
	 Executed 22 tests, with 0 failures (0 unexpected) in 63.963 (63.965) seconds
Test Suite 'All tests' passed at 2019-08-07 11:14:05.797.
	 Executed 22 tests, with 0 failures (0 unexpected) in 63.963 (63.966) seconds
@cbowns
Copy link
Author

cbowns commented Aug 7, 2019

And here are results from beta 6 (17A5556d):

2019-08-07 13:26:07.210946-0700 SpeedTest[341:24328] Launching with XCTest injected. Preparing to run tests.
2019-08-07 13:26:07.297163-0700 SpeedTest[341:24328] Waiting to run tests until the app finishes launching.
Test Suite 'All tests' started at 2019-08-07 13:26:07.489
Test Suite 'SpeedTestTests.xctest' started at 2019-08-07 13:26:07.490
Test Suite 'CombineTests' started at 2019-08-07 13:26:07.490
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' measured [Time, seconds] average: 0.318, relative standard deviation: 6.016%, values: [0.375417, 0.311859, 0.312010, 0.311715, 0.311345, 0.311301, 0.311863, 0.311725, 0.311432, 0.311565], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestCreating]' passed (3.792 seconds).
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' measured [Time, seconds] average: 0.253, relative standard deviation: 6.738%, values: [0.304167, 0.247866, 0.247569, 0.247646, 0.246472, 0.246450, 0.247136, 0.247631, 0.247710, 0.247747], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testCombineLatestPumping]' passed (2.783 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.178, relative standard deviation: 9.184%, values: [0.226429, 0.171536, 0.173712, 0.172446, 0.172430, 0.171946, 0.171854, 0.171848, 0.171527, 0.171743], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestCreating]' passed (2.028 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.674, relative standard deviation: 2.398%, values: [0.721777, 0.669685, 0.669539, 0.668771, 0.671338, 0.664318, 0.671677, 0.666001, 0.664972, 0.670764], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapLatestPumping]' passed (6.991 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' measured [Time, seconds] average: 0.119, relative standard deviation: 14.836%, values: [0.172441, 0.113581, 0.113365, 0.113296, 0.113340, 0.113505, 0.113642, 0.113303, 0.113457, 0.113377], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsCreating]' passed (1.445 seconds).
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' measured [Time, seconds] average: 0.367, relative standard deviation: 4.489%, values: [0.416726, 0.362522, 0.361731, 0.361342, 0.360232, 0.362128, 0.361719, 0.362184, 0.361633, 0.362829], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testFlatMapsPumping]' passed (3.925 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' measured [Time, seconds] average: 0.118, relative standard deviation: 14.921%, values: [0.171123, 0.112612, 0.112610, 0.112359, 0.112215, 0.112296, 0.112182, 0.112111, 0.112174, 0.112422], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterCreating]' passed (1.434 seconds).
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' measured [Time, seconds] average: 0.150, relative standard deviation: 11.140%, values: [0.199921, 0.144274, 0.144080, 0.145355, 0.144366, 0.143941, 0.144544, 0.144062, 0.144000, 0.144013], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testMapFilterPumping]' passed (1.750 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.218, relative standard deviation: 8.530%, values: [0.274080, 0.212130, 0.211926, 0.211638, 0.212343, 0.211968, 0.211992, 0.212033, 0.212016, 0.212248], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectCreating]' passed (2.434 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.142, relative standard deviation: 11.701%, values: [0.191209, 0.136250, 0.134096, 0.136065, 0.136219, 0.136662, 0.136309, 0.136293, 0.136283, 0.136210], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumping]' passed (1.667 seconds).
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.240, relative standard deviation: 5.999%, values: [0.282560, 0.236603, 0.234190, 0.234281, 0.234306, 0.234278, 0.234263, 0.234078, 0.236806, 0.234136], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.CombineTests testPublishSubjectPumpingTwoSubscriptions]' passed (2.647 seconds).
Test Suite 'CombineTests' passed at 2019-08-07 13:26:38.387.
	 Executed 11 tests, with 0 failures (0 unexpected) in 30.896 (30.898) seconds
Test Suite 'RxSwiftTests' started at 2019-08-07 13:26:38.388
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' measured [Time, seconds] average: 0.425, relative standard deviation: 3.861%, values: [0.474261, 0.419360, 0.419179, 0.419647, 0.419880, 0.419783, 0.419791, 0.419441, 0.419762, 0.419206], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestCreating]' passed (4.502 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' measured [Time, seconds] average: 0.243, relative standard deviation: 6.657%, values: [0.291370, 0.237798, 0.237569, 0.237795, 0.237357, 0.237314, 0.237417, 0.237283, 0.237399, 0.237377], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testCombineLatestPumping]' passed (2.681 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.167, relative standard deviation: 10.386%, values: [0.218782, 0.161544, 0.160732, 0.160912, 0.160940, 0.160696, 0.160516, 0.161527, 0.161166, 0.161331], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' passed (1.920 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' measured [Time, seconds] average: 0.570, relative standard deviation: 2.700%, values: [0.615635, 0.566253, 0.568533, 0.564175, 0.563708, 0.563772, 0.560887, 0.567682, 0.563589, 0.564728], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapLatestPumping]' passed (5.950 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' measured [Time, seconds] average: 0.160, relative standard deviation: 10.657%, values: [0.211394, 0.154491, 0.154568, 0.154149, 0.154565, 0.154470, 0.154470, 0.154504, 0.154380, 0.154831], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsCreating]' passed (1.854 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' measured [Time, seconds] average: 0.521, relative standard deviation: 3.294%, values: [0.572535, 0.514815, 0.514877, 0.514572, 0.515807, 0.515892, 0.515797, 0.515386, 0.515575, 0.515356], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testFlatMapsPumping]' passed (5.462 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' measured [Time, seconds] average: 0.133, relative standard deviation: 14.420%, values: [0.190349, 0.126352, 0.126598, 0.126862, 0.126845, 0.126385, 0.126264, 0.126307, 0.126540, 0.126210], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterCreating]' passed (1.581 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' measured [Time, seconds] average: 0.129, relative standard deviation: 12.877%, values: [0.179192, 0.123458, 0.123519, 0.125303, 0.123745, 0.123529, 0.123485, 0.123447, 0.123565, 0.123534], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testMapFilterPumping]' passed (1.545 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' measured [Time, seconds] average: 0.221, relative standard deviation: 7.788%, values: [0.272106, 0.214946, 0.214991, 0.214795, 0.214900, 0.214847, 0.214936, 0.214999, 0.214808, 0.214403], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectCreating]' passed (2.458 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' measured [Time, seconds] average: 0.203, relative standard deviation: 8.138%, values: [0.252146, 0.197085, 0.197622, 0.197080, 0.197083, 0.197175, 0.197090, 0.197246, 0.197188, 0.196996], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumping]' passed (2.278 seconds).
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' started.
<unknown>:0: Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' measured [Time, seconds] average: 0.360, relative standard deviation: 4.809%, values: [0.412296, 0.355104, 0.353584, 0.354840, 0.352790, 0.354416, 0.356227, 0.354917, 0.354394, 0.355166], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[SpeedTestTests.RxSwiftTests testPublishSubjectPumpingTwoSubscriptions]' passed (3.855 seconds).
Test Suite 'RxSwiftTests' passed at 2019-08-07 13:27:12.474.
	 Executed 11 tests, with 0 failures (0 unexpected) in 34.085 (34.086) seconds
Test Suite 'SpeedTestTests.xctest' passed at 2019-08-07 13:27:12.474.
	 Executed 22 tests, with 0 failures (0 unexpected) in 64.981 (64.984) seconds
Test Suite 'All tests' passed at 2019-08-07 13:27:12.474.
	 Executed 22 tests, with 0 failures (0 unexpected) in 64.981 (64.985) seconds

@cbowns
Copy link
Author

cbowns commented Aug 7, 2019

Also, the first-run time deviation from subsequent runs is super interesting, non?

SpeedTestTests.RxSwiftTests testFlatMapLatestCreating]' measured [Time, seconds] average: 0.167, relative standard deviation: 10.386%, values:

[0.218782, 0.161544, 0.160732, 0.160912, 0.160940, 0.160696, 0.160516, 0.161527, 0.161166, 0.161331],

The std-dev for all of those tests above is 0.01826284758, but for the 2..<n tests, it's just 0.0003718030766.

@grafele
Copy link
Collaborator

grafele commented Aug 8, 2019

Hey. Thanks for your input. Do you mind opening a merge request?

@cbowns
Copy link
Author

cbowns commented Aug 13, 2019

I wasn't going to; I figured I'd let you (or someone else) regenerate the graphs, I don't have the cycles to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants