install_clcache_msbuild.py : helper for msbuild integration #330

pthom · 2018-11-17T16:48:46Z

Since the integration of clcache with msbuild is rather cumbersome,
this script provide an helper in order to simplify the process.

Notes:

Incremental builds: There still is an issue with msbuild incremental build : we always get a full rebuild. A merge of the PR https://github.com/frerich/clcache/pull/319/commits would make this script more relevant : as of now, it cannot reliably be used from inside Visual Studio, since any file modification always triggers a full rebuild.
This PR is a cousin of the pull request Feature prepare environment script #172, which I discovered long after I started to work on this

It is compatible with MSVC 2010, 2015 and 2017 (tested with MSVC 2015 and 2017).

What this script does:

Check that python3 and pip3 are installed and are in the PATH
Check that the pip installed scripts are in the PATH (PYTHONHOME\Scripts)
Call pip install . from the repo and check that clcache is then in the PATH.
clcache will subsequently be used from the PYTHONHOME\Scripts directory.
Modify the user msbuild preference files inside %AppData%\..\Local\Microsoft\MSBuild\v4.0
so that clcache becomes the default compiler. (These prefs are shared between MSVC 2010 to 2017).
Find all cl.exes version on your computer (for MSVC 2010 to MSVC 2017), and allows you
to select the correct one, by showing a detailed list of their version and target architecture.
Set the env variable CLCACHE_CL with the correct path to cl.exe

As additional options, this script can also

change the cache location
change the cache size
change the timeout CLCACHE_OBJECT_CACHE_TIMEOUT_MS

Caveat

Since the msbuild preference files inside %AppData%\..\Local\Microsoft\MSBuild\v4.0 are shared
between different MSVC installations, clcache will be activated for all instances of MSVC.

Note

vswhere.exe is a tool provided by Microsoft in order to locate installations of MSVC >= 2017.

Usage

usage: install_clcache_msbuild.py [-h] [--cachedir CACHEDIR]
                                  [--cache_size CACHE_SIZE]
                                  [--clcache_timeout CLCACHE_TIMEOUT]
                                  {status,install,disable,enable_logs,disable_logs,show_cl_list,select_cl}

Since the integration of `clcache` with `msbuild` is rather cumbersome, this script provide an helper in order to simplify the process. What this script does: ********************* * Check that python3 and pip3 are installed and are in the PATH * Check that the pip installed scripts are in the PATH (PYTHONHOME\Scripts) * Call `pip install .` from the repo and check that clcache is then in the PATH. `clcache` will subsequently be used from the PYTHONHOME\\Scripts directory. * Modify the user msbuild preference files inside `%AppData%\..\Local\Microsoft\MSBuild\v4.0` so that clcache becomes the default compiler * Set the env variable `CLCACHE_CL` with the correct path to cl.exe As additional options, this script can also * change the cache location * change the cache size * change the timeout CLCACHE_OBJECT_CACHE_TIMEOUT_MS Usage : ******* ```` c: cd C:\clcache\install_for_msbuild python install_clcache_msbuild.py -h usage: install_clcache_msbuild.py [-h] [--cachedir CACHEDIR] [--cache_size CACHE_SIZE] [--clcache_timeout CLCACHE_TIMEOUT] {status,install,disable,enable_logs,disable_logs} Configure clcache for use with msbuild positional arguments: {status,install,disable,enable_logs,disable_logs} action optional arguments: -h, --help show this help message and exit --cachedir CACHEDIR clcache directory --cache_size CACHE_SIZE clcache size in Go --clcache_timeout CLCACHE_TIMEOUT clcache object cache timeout in seconds (increase if you have failures during your build) Actions summary: status : Show the install status and tells if clcache is enabled install: : Install and enable clcache for msbuild integration" disable: : Disable clcache enable_logs : Activate clcache logs during builds disable_logs : Disable clcache logs during builds ````

codecov-io · 2018-11-17T16:58:53Z

Codecov Report

Merging #330 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #330   +/-   ##
=======================================
  Coverage   88.23%   88.23%           
=======================================
  Files           4        4           
  Lines        1275     1275           
  Branches      190      190           
=======================================
  Hits         1125     1125           
  Misses        111      111           
  Partials       39       39

Flag	Coverage Δ
#integrationtests_memcached	`66.85% <ø> (ø)`	⬆️
#unittests	`85.25% <ø> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d62cc1...0ba3f16. Read the comment docs.

pthom · 2018-11-18T13:40:55Z

Hum, this script only works for Visual Studio 2015.
The installation and settings have varied a lot for Visual studio 2017.
I will try to study this.

This script should now be compatible with MSVC installs, even if not installed in the default folder. The install will now ask to select the correct compiler Notes: * `vswhere.exe` is a tool provided by Microsoft in order to locate installations of MSVC >= 2017. * locate_cl_exe.py : will create a list of all the available cl.exe compiler such as below: Sample usage session: ****************** > python install_clcache_msbuild.py install Looking for python in PATH C:\Python36-32\python.exe Looking for pip in PATH C:\Python36-32\Scripts\pip.exe Installing clcache (installClcache) ====> pip install .(in folder F:\dvp\OpenSource\clcache) Processing f:\dvp\opensource\clcache Requirement already satisfied: pymemcache in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (2.0.0) Requirement already satisfied: pyuv in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (1.4.0) Requirement already satisfied: six in c:\python36-32\lib\site-packages (from pymemcache->clcache==4.1.1.dev65+g105e486.d20181119) (1.11.0) Installing collected packages: clcache Found existing installation: clcache 4.1.1.dev65+g105e486.d20181119 Uninstalling clcache-4.1.1.dev65+g105e486.d20181119: Successfully uninstalled clcache-4.1.1.dev65+g105e486.d20181119 Running setup.py install for clcache ... done Successfully installed clcache-4.1.1.dev65+g105e486.d20181119 You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. Looking for clcache in PATH C:\Python36-32\Scripts\clcache.exe Force clcache via Msbbuild user settings (copyMsvcPrefClcache) Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.ARM.user.props Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.Win32.user.props Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.x64.user.props Select cl compiler: (selectCl) # version targetArch hostArch folder (shortened) 1 14.0 amd64 amd64 C:\ProgX86\MSVC 14.0\vc\bin\amd64 2 14.0 arm amd64 C:\ProgX86\MSVC 14.0\vc\bin\amd64_arm 3 14.0 x86 amd64 C:\ProgX86\MSVC 14.0\vc\bin\amd64_x86 4 14.0 amd64 x86 C:\ProgX86\MSVC 14.0\vc\bin\x86_amd64 5 14.0 arm x86 C:\ProgX86\MSVC 14.0\vc\bin\x86_arm 6 14.0 x86 x86 C:\ProgX86\MSVC 14.0\vc\bin 7 15.7.27703.2047 x64 x64 C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x64 8 15.7.27703.2047 x86 x64 C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x86 9 15.7.27703.2047 x64 x86 C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x64 10 15.7.27703.2047 x86 x86 C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x86 Enter the number corresponding to the desired compiler: 6 Selected : C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe ====> SETX CLCACHE_CL "C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe" SUCCESS: Specified value was saved. Note about clcache usage: (showClCacheUsage) ====> clcache --help clcache.py v4.1.0-dev --help : show this help -s : print cache statistics -c : clean cache -C : clear cache -z : reset cache statistics -M <size> : set maximum cache size (in bytes)

pthom · 2018-11-19T09:55:55Z

Ok, this script is now compatible with MSVC 2015 and 2017 (and should also work with MSVC 2012).
It enables to select the correct cl.exe version by presenting a list of the available cl.exe compilers, sorted by version, host Arch and target Arch.

Sample usage session:

> python install_clcache_msbuild.py install
Looking for python in PATH
C:\Python36-32\python.exe
Looking for pip in PATH
C:\Python36-32\Scripts\pip.exe

######################################################################
Installing clcache (installClcache)
######################################################################
====> pip install .(in folder F:\dvp\OpenSource\clcache)
Processing f:\dvp\opensource\clcache
Requirement already satisfied: pymemcache in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (2.0.0)
Requirement already satisfied: pyuv in c:\python36-32\lib\site-packages (from clcache==4.1.1.dev65+g105e486.d20181119) (1.4.0)
Requirement already satisfied: six in c:\python36-32\lib\site-packages (from pymemcache->clcache==4.1.1.dev65+g105e486.d20181119) (1.11.0)
Installing collected packages: clcache
  Found existing installation: clcache 4.1.1.dev65+g105e486.d20181119
    Uninstalling clcache-4.1.1.dev65+g105e486.d20181119:
      Successfully uninstalled clcache-4.1.1.dev65+g105e486.d20181119
  Running setup.py install for clcache ... done
Successfully installed clcache-4.1.1.dev65+g105e486.d20181119
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Looking for clcache in PATH
C:\Python36-32\Scripts\clcache.exe

######################################################################
Force clcache via Msbbuild user settings (copyMsvcPrefClcache)
######################################################################
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.ARM.user.props
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.Win32.user.props
Wrote pref in C:\Users\pascal\AppData\Local\Microsoft\MSBuild\v4.0\Microsoft.Cpp.x64.user.props

######################################################################
Select cl compiler: (selectCl)
######################################################################
   #           version targetArch   hostArch                                                              folder (shortened)
   1              14.0      amd64      amd64                                               C:\ProgX86\MSVC 14.0\vc\bin\amd64
   2              14.0        arm      amd64                                           C:\ProgX86\MSVC 14.0\vc\bin\amd64_arm
   3              14.0        x86      amd64                                           C:\ProgX86\MSVC 14.0\vc\bin\amd64_x86
   4              14.0      amd64        x86                                           C:\ProgX86\MSVC 14.0\vc\bin\x86_amd64
   5              14.0        arm        x86                                             C:\ProgX86\MSVC 14.0\vc\bin\x86_arm
   6              14.0        x86        x86                                                     C:\ProgX86\MSVC 14.0\vc\bin
   7   15.7.27703.2047        x64        x64       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x64
   8   15.7.27703.2047        x86        x64       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx64\x86
   9   15.7.27703.2047        x64        x86       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x64
  10   15.7.27703.2047        x86        x86       C:\ProgX86\MSVC\2017\Enterprise\VC\Tools\MSVC\14.14.26428\bin\Hostx86\x86
Enter the number corresponding to the desired compiler: 6
Selected : C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe
====> SETX CLCACHE_CL "C:\Program Files (x86)\Microsoft Visual Studio 14.0\vc\bin\cl.exe"

SUCCESS: Specified value was saved.

######################################################################
Note about clcache usage: (showClCacheUsage)
######################################################################
====> clcache --help
clcache.py v4.1.0-dev
  --help    : show this help
  -s        : print cache statistics
  -c        : clean cache
  -C        : clear cache
  -z        : reset cache statistics
  -M <size> : set maximum cache size (in bytes)

This add two requirements: winshell and pywin32

(+ some linting)

frerich · 2018-11-20T08:39:51Z

@pthom This looks really useful! Alas, I must admit that I never used msbuild myself in the past, so I don't feel like I would be the best person to maintain the script. As such, I'm hesitant to merge the PR.

Maybe it could be hosted on your GitHub account and then I add a link to it in the Wiki?

pthom · 2018-11-20T15:31:54Z

Hi, thanks for you answer.
I understand your concern and your proposition sounds ok.
I propose to setup a different repo, where ccache will be a submodule (and initialy it will contain the branch with the fix for incremental builds). I think I will name it "clcache-msbuild".

What do you think ?

pthom · 2018-11-21T11:07:41Z

Hello,

I created a repo for this utility at https://github.com/pthom/clcache-msbuild-install

When running, it will clone clcache and the run the installation script (clcache is not a submodule, but it is cloned during the installation).

Can you tell me your opinion?

Note: Since I wanted to use the version of clcache that includes the correction for incremental build, I used a a fork of clcache (https://github.com/pthom/clcache/tree/clcache-msbuild-install). This fork is essentialy the same as the PR #319.

Can you tell me if you intend to merge the PR in the future ? If so, I will be able to use the original clcache repo in this tool :-)

pthom mentioned this pull request Nov 17, 2018

Add msbuild tracker support #319

Open

install_clcache_msbuild.py : add enable option

377c2c8

pthom mentioned this pull request Nov 19, 2018

Incremental builds with MSBuild/Visual Studio are broken #227

Open

Pascal Thomet added 2 commits November 19, 2018 17:04

install_clcache_msbuild.py : add enable_server / disable_server options

9001bb0

This add two requirements: winshell and pywin32

install_clcache_msbuild.py : more robust cl.exe detection

5d48d44

(+ some linting)

pthom force-pushed the feature/install_for_msbuild_master branch from 78a4630 to 5d48d44 Compare November 19, 2018 16:05

install_clcache_msbuild.py : correct option disable_server

0ba3f16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

install_clcache_msbuild.py : helper for msbuild integration #330

install_clcache_msbuild.py : helper for msbuild integration #330

pthom commented Nov 17, 2018 •

edited

Loading

codecov-io commented Nov 17, 2018 •

edited

Loading

pthom commented Nov 18, 2018

pthom commented Nov 19, 2018

frerich commented Nov 20, 2018

pthom commented Nov 20, 2018

pthom commented Nov 21, 2018

install_clcache_msbuild.py : helper for msbuild integration #330

Are you sure you want to change the base?

install_clcache_msbuild.py : helper for msbuild integration #330

Conversation

pthom commented Nov 17, 2018 • edited Loading

What this script does:

Caveat

Note

Usage

codecov-io commented Nov 17, 2018 • edited Loading

Codecov Report

pthom commented Nov 18, 2018

pthom commented Nov 19, 2018

frerich commented Nov 20, 2018

pthom commented Nov 20, 2018

pthom commented Nov 21, 2018

pthom commented Nov 17, 2018 •

edited

Loading

codecov-io commented Nov 17, 2018 •

edited

Loading