Collected data seems suspicious; Canary mode doesn't report metrics

[jimfb]

I checked out the demo and ran it with the following results:

    forcedGcAmount |       forcedGcTime |           gcAmount |             gcTime |        majorGcTime |     pureScriptTime |         renderTime |         scriptTime
------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------
           1299.77 |              13.06 |             491.90 |              10.73 |               0.00 |              51.52 |             134.63 |              62.25
           2367.62 |              29.73 |               0.00 |               0.00 |               0.00 |              73.11 |             138.69 |              73.11
           2146.84 |              28.58 |            1400.20 |               6.02 |               0.00 |              64.64 |             138.59 |              70.66
           3729.60 |              33.67 |               0.00 |               0.00 |               0.00 |              64.24 |             141.94 |              64.24
           3765.45 |              34.57 |               0.00 |               0.00 |               0.00 |              65.50 |             137.77 |              65.50
           3775.76 |              39.67 |               0.00 |               0.00 |               0.00 |              67.22 |             135.41 |              67.22
           3769.90 |              37.77 |               0.00 |               0.00 |               0.00 |              66.30 |             142.54 |              66.30
           3776.86 |              35.73 |               0.00 |               0.00 |               0.00 |              70.08 |             146.30 |              70.08
           3761.22 |              37.83 |               0.00 |               0.00 |               0.00 |              67.39 |             139.36 |              67.39
           3762.50 |              37.86 |               0.00 |               0.00 |               0.00 |              64.04 |             148.04 |              64.04
           3794.86 |              38.85 |               0.00 |               0.00 |               0.00 |              63.96 |             135.96 |              63.96
           3781.95 |              38.89 |               0.00 |               0.00 |               0.00 |              65.23 |             143.53 |              65.23
           3764.52 |              38.47 |               0.00 |               0.00 |               0.00 |              67.48 |             144.74 |              67.48
           3759.41 |              36.89 |               0.00 |               0.00 |               0.00 |              66.44 |             137.67 |              66.44
           3762.62 |              36.82 |               0.00 |               0.00 |               0.00 |              67.38 |             142.58 |              67.38
           3774.42 |              41.91 |               0.00 |               0.00 |               0.00 |              65.47 |             138.46 |              65.47
           3764.98 |              40.21 |               0.00 |               0.00 |               0.00 |              66.89 |             140.60 |              66.89
           3759.39 |              41.70 |               0.00 |               0.00 |               0.00 |              66.40 |             144.26 |              66.40
           3755.74 |              36.05 |               0.00 |               0.00 |               0.00 |              71.11 |             137.41 |              71.11
           3764.54 |              37.54 |               0.00 |               0.00 |               0.00 |              69.03 |             143.31 |              69.03
================== | ================== | ================== | ================== | ================== | ================== | ================== | ==================
      3491.90+-19% |         35.79+-17% |        94.60+-336% |         0.84+-312% |               0.00 |          66.17+-6% |         140.59+-2% |          67.01+-3%
.

The GC times looked awfully suspicious to me (but I wasn't surprised, since the runner appeared to be running the default version of chrome). I modified chromeOptions to include 'binary': '/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary',, which did successfully cause Canary to be launched, but then the results were empty:

    forcedGcAmount |       forcedGcTime |           gcAmount |             gcTime |        majorGcTime |     pureScriptTime |         renderTime |         scriptTime
------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.26 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.27 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.61 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.46 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.22 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.44 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.57 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.36 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.31 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.28 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.52 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.37 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.61 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.43 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.53 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.37 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.28 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.62 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.41 |               0.00
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.41 |               0.00
================== | ================== | ================== | ================== | ================== | ================== | ================== | ==================
              0.00 |               0.00 |               0.00 |               0.00 |               0.00 |               0.00 |          0.42+-29% |               0.00

I'm new to angular / protractor / benchpress, so maybe I'm missing something obvious. Any ideas?

[jeffbcross]

@jimfb did you modify your tests at all between samples? Seems as if test isn't running as expected. You may also want to start up a server and manually interact with the application, then inspect the console to see if any errors are reported.

[jimfb]

@jeffbcross Nope, didn't modify the tests. Clean checkout of this repository, only change is that on the second run I set the chromeOptions binary path (and saw that chrome canary did run, so the path is correct). I also saw the webdriver clicking the tests, so it appeared to be working, and it doesn't appear to be a complete failure because the renderTime column has data.

I just tried starting the server manually and running the test myself in chrome and chrome-canary, both looked perfectly reasonable to me.

[jeffbcross]

Is it possible you could share the complete runnable app, benchmark, setup, etc?

[jimfb]

It's literally just a clean checkout for the first run, and a single line ('binary': '/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary',) added to protractor.conf.js for the second run. I just ran the three commands in the wiki and pasted the results here.

Other than that, the only relevant setup is my chrome&canary installations (both standard, afaik) and my OS (Yosemite 10.10.4 (14E46)) - I'm not really sure how to "share" those. What did you have in mind?

[jeffbcross]

Sounds like the same issue as reported here: angular/angular#3411

[jeffbcross]

@jimfb do you want to take a stab at a PR to fix benchpress?