April 29, 2016 March 16, 2021 By Roberto Prieto In Deep learning, News

How to install Caffe on Mac OS X 10.11

Deep learning is a hot topic these days and it is greatly increased by the fact that AMD/nVidia video cards can be used for accelerating the training of very complex neural networks. In the meantime, some powerful frameworks have been developed by the community. One of the most known is Caffe which is specialized in image recognition, just one of the areas where I am more interested in.

However, it is not an easy task to get a working Caffe environment for a standard user. Of course, there are plenty of online documentation and some very nice guides but I didn’t find any for a Mac OS X 10.11.4 version working out of the box, so after struggling several hours trying to have everything ready, I decided to post here all the steps just in case it could help someone else.

In my case, I wanted to run the cool new automatic colorization methods using deep learning algorithms and the results are absolutely fantastic.

In addition, I also wanted to use CPU and GPU code paths in order to compare the processing speed, unfortunately, the video card I have on my workstation is an old AMD card and I need to have a nVidia one for using CUDA libraries so my only machine with that specs is a MacBook Pro with Mac OS X 10.11.4, 16GB RAM, Intel Core i7 4960HQ and a nVidia GeForce GT 750M (2GB).

Let’s start with the optional but highly recommended prerequisites:

• Check your Mac OS X and update everything is pending, you also need to install Xcode, 7.3 is the latest one but not supported by CUDA yet so later on, I will show you a work around for this issue
• Install Homebrew package manager and if possible, start up with a clean /usr/local directory
• Install Anaconda Python 2.7 on /usr/local directory. Once it is done, add to your ~/.bash_profile file:

  export PATH=/usr/local/ananconda/bin:$PATH
  export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/anaconda:/usr/local/lib:/usr/lib
  

Now, we are installing the needed libraries:

• Install nVidia CUDA 7.5, in my case the exact version is 7.5.27. I also installed it on /usr/local/cuda. Once it is done, add to your ~/.bash_profile file:

  export PATH=/usr/local/cuda/bin:$PATH
  export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH
  export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_FALLBACK_LIBRARY_PATH
  

• Install nVidia cuDNN, you have to register and get approved by nVidia. I got 7.5 v5 but unfortunately, I was not able to use it, apparently some kind of incompatibility with my CUDA version or the Xcode used. In any case, once you have the tgz file, just uncompress it (tar -xfvz) and copy the contents of the /lib directory in to /usr/local/cuda/lib and cudnn.h to /usr/local/cuda/include directory.

• Install Intel MKL libraries, which is a part of Intel Parallel Studio XE Composer, you can download a trial version or a student/academic research one if you fulfill the requirements. MKL libraries are installed on /opt/intel/mkl. Once it is done, add to your ~/.bash_profile file:

  export DYLD_FALLBACK_LIBRARY_PATH=/opt/intel/mkl/lib:$DYLD_FALLBACK_LIBRARY_PATH
  

• Using Homebrew, we are installing some needed libraries. At this point, it is very important to have the right environment variables, if you followed the guide step-by-step, you have them on your ~/.bash_profile but if you were working on the same terminal session, they are not loaded so, you can close this terminal and open a new one or just execute this command:

  source ~/.bash_profile
  

  Before starting to download, compile and install the pending libraries, execute this command:

  python --version
  

  and you should see something like “Python 2.7.xx :: Anaconda 4.x.x (x86_64)”, if that is the case, the go on, otherwise, revise your environment vars as something is not well configured.

  At this point, we are using the python provided by Anaconda so let’s execute these commands (note that in some cases, the GNU C/C++ 5.3.0 will be compiled and it takes a while, ~30min on my rig):

  brew install --fresh -vd snappy leveldb gflags glog szip lmdb homebrew/science/opencv
  brew install --build-from-source --with-python --fresh -vd protobuf
  brew install --build-from-source --fresh -vd boost boost-python
  

• Finally, we can download Caffe, I am donwloading it to /usr/local/caffe:

  git clone https://github.com/BVLC/caffe.git
  cd caffe
  cp Makefile.config.example Makefile.config
  

  Next step is to configure it, open Makefile.config file with your preferred text editor (vim?) and let’s configure the following vars:

   ## Refer to http://caffe.berkeleyvision.org/installation.html
  # Contributions simplifying and improving our build system are welcome!
  
  # cuDNN acceleration switch (uncomment to build with cuDNN).
  # USE_CUDNN := 1
  
  # CPU-only switch (uncomment to build without GPU support).
  # CPU_ONLY := 1
  
  # uncomment to disable IO dependencies and corresponding data layers
  # USE_OPENCV := 0
  # USE_LEVELDB := 0
  # USE_LMDB := 0
  
  # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
  #    You should not set this flag if you will be reading LMDBs with any
  #    possibility of simultaneous read and write
  # ALLOW_LMDB_NOLOCK := 1
  
  # Uncomment if you're using OpenCV 3
  # OPENCV_VERSION := 3
  
  # To customize your choice of compiler, uncomment and set the following.
  # N.B. the default for Linux is g++ and the default for OSX is clang++
  # CUSTOM_CXX := g++
  
  # CUDA directory contains bin/ and lib/ directories that we need.
  CUDA_DIR := /usr/local/cuda
  # On Ubuntu 14.04, if cuda tools are installed via
  # "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
  # CUDA_DIR := /usr
  
  # CUDA architecture setting: going with all of them.
  # For CUDA < 6.0, comment the *_50 lines for compatibility.
  CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
  -gencode arch=compute_20,code=sm_21 \
  -gencode arch=compute_30,code=sm_30 \
  -gencode arch=compute_35,code=sm_35 \
  -gencode arch=compute_50,code=sm_50 \
  -gencode arch=compute_50,code=compute_50
  
  # BLAS choice:
  # atlas for ATLAS (default)
  # mkl for MKL
  # open for OpenBlas
  BLAS := mkl
  # Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
  # Leave commented to accept the defaults for your choice of BLAS
  # (which should work)!
  BLAS_INCLUDE := /opt/intel/mkl/include
  BLAS_LIB := /opt/intel/mkl/lib
  
  # Homebrew puts openblas in a directory that is not on the standard search path
  # BLAS_INCLUDE := $(shell brew --prefix openblas)/include
  # BLAS_LIB := $(shell brew --prefix openblas)/lib
  
  # This is required only if you will compile the matlab interface.
  # MATLAB directory should contain the mex binary in /bin.
  # MATLAB_DIR := /usr/local
  # MATLAB_DIR := /Applications/MATLAB_R2012b.app
  
  # NOTE: this is required only if you will compile the python interface.
  # We need to be able to find Python.h and numpy/arrayobject.h.
  #PYTHON_INCLUDE := /usr/include/python2.7 \
  #        /usr/lib/python2.7/dist-packages/numpy/core/include
  # Anaconda Python distribution is quite popular. Include path:
  # Verify anaconda location, sometimes it's in root.
  ANACONDA_HOME := /usr/local/anaconda
  PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
  $(ANACONDA_HOME)/include/python2.7 \
  $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
  
  # Uncomment to use Python 3 (default is Python 2)
  # PYTHON_LIBRARIES := boost_python3 python3.5m
  # PYTHON_INCLUDE := /usr/include/python3.5m \
  #                 /usr/lib/python3.5/dist-packages/numpy/core/include
  
  # We need to be able to find libpythonX.X.so or .dylib.
  #PYTHON_LIB := /usr/lib
  PYTHON_LIB := $(ANACONDA_HOME)/lib
  # PYTHON_LIB := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/
  
  # Homebrew installs numpy in a non standard path (keg only)
  PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
  PYTHON_LIB += $(shell brew --prefix numpy)/lib
  
  # Uncomment to support layers written in Python (will link against Python libs)
  # WITH_PYTHON_LAYER := 1
  
  # Whatever else you find you need goes here.
  INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
  LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
  
  # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
  INCLUDE_DIRS += $(shell brew --prefix)/include
  LIBRARY_DIRS += $(shell brew --prefix)/lib
  
  # Uncomment to use `pkg-config` to specify OpenCV library paths.
  # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
  # USE_PKG_CONFIG := 1
  
  BUILD_DIR := build
  DISTRIBUTE_DIR := distribute
  
  # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
  # DEBUG := 1
  
  # The ID of the GPU that 'make runtest' will use to run unit tests.
  TEST_GPUID := 0
  
  # enable pretty build (comment to see full commands)
  Q ?= @
  

  Update 23/August/2016: Instead of providing a partial Makefile.config file, a full copy of this file is attached above.

  Give it a try…:

  make clean
  make all -j8
  

  If it compiles fine, you can skip this paragraph, you probably have Xcode 7.0.2 or a lower version which is OK…
  but if all you get is this error:
  “nvcc fatal : The version (‘70300’) of the host compiler (‘Apple clang’) is not supported”
  it means you have the Xcode 7.3 and CUDA still does not support it.
  Of course, it has a solution, to install a supported Xcode version and use it for compiling Caffe:
  – go to https://developer.apple.com/downloads/ and log in
  – download Command Line Tools OS X 10.11 Xcode 7.2 and install it
  – now, switch to use this version:

   sudo xcode-select --switch /Library/Developer/CommandLineTools 

  Remember that using:

   sudo xcode-select -r 

  will configure the default compiler, which, in my case is 7.3.
  Let’s try to compile it again:

  make clean
  make all -j8
  

  This time, it should be OK. Now, let’s compile the tests:

  make test -j8
  

  Everything went fine? Perfect! At this point, you probably want to run the tests but… they will fail(!). This time the guilty one is Apple and the latest OS X 10.11 “El Capitan” version, apparently it is undocumented but spending some time ‘googling’, I found that other application are failing too as it seems that Apple unset all DYLD_ prefix environment variables when running scripts. I only know one solution:
  – to disable System Integrity Protection, which I totally do not recommend to do, or if you do that, revert to SIP on as soon as you check that all the tests were OK.
  So, we can discard to run the tests and trust that everything is fine and continue with only a few more steps.

• Compiling PyCaffe, which is the python interface to Caffe:

  for req in $(cat python/requirements.txt); do pip install $req; done
  make pycaffe
  make distribute
  

  If everything went fine, you are almost there! Add this environment variable to your ~/.bash_profile:

  export PYTHONPATH=/usr/local/caffe/python
  

  And now the final test:

  cd /usr/local/caffe/python
  python
  import caffe
  

  You should get the python prompt and no errors at all which means that your python interface to Caffe is fully working and ready for some cool applications:

  Python 2.7.11 |Anaconda 4.0.0 (x86_64)| (default, Dec  6 2015, 18:57:58)
  [GCC 4.2.1 (Apple Inc. build 5577)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  Anaconda is brought to you by Continuum Analytics.
  Please check out: http://continuum.io/thanks and https://anaconda.org
  >>> import caffe
  >>>
  

  Update: if you receive some errors regarding to locale, you just need to add these new environment vars to your famous  ~/.bash_profile file:

  export LC_ALL=en_US.UTF-8
  export LANG=en_US.UTF-8
  

You are now ready for playing with some examples:

• LeNet on MNIST
• ImageNet
• CaffeNet C++ Classification example

As you can see, it is not an easy task to configure Caffe environment but finally I got it and can test the Automatic Colorization algorithms for automatically converting grey-pictures into fully colored ones, in a few days I will write down a small post with my result.

Have fun with this great but a bit difficult to configure deep learning framework!!