Removing unused Jupyter notebook kernels on Mac

Since I am experimenting with various Python versions 2 vs 3.5 vs 3.6 and packages, e.g. TensorFlow I often end up with conda environments and corresponding Jupyter kernels that I do not use.

Removing Kernels

$ cd ~/Library/Jupyter/kernels 
$ rm -r tensorflow_gpu/

See my post on:

Installing Jupyter kernels to match Conda environments

http://ukitech.blogspot.com/2017/02/kernel.html

Developing Alexa skills using flask-ask Python framework

Install Flask-Ask for Alexa

$ pip install flask-ask

Start your favorite Python dev editor

$ jupyter notebook

Write code

To avoid repeating myself in 2 places, here is my code on GitHub:

https://raw.githubusercontent.com/UkiDLucas/uki.guru/master/Alexa/skill_001.py

https://github.com/UkiDLucas/uki.guru/blob/master/Alexa/templates.yaml

Test code running locally via HTTPS using ngrok

$ python skill_001.py  * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) * Restarting with stat * Debugger is active! * Debugger pin code: 275-052-815

https://ngrok.com/

- unzip
- put in Applications folder
- click to open
- you should get a message "ngrok was blocked from opening because it is not from an identified developer"
- got to System Preferences -> Security & Privacy
- click "Open Anyway"
- run from Terminal:

$ /Applications/ngrok http 5000

You will need the HTTPS URL form ngrok

ngrok by @inconshreveable                                                                                                                                                (Ctrl+C to quit)                                                                                                                                                                                         Session Status                online                                                                                                                                                     Version                       2.1.18                                                                                                                                                     Region                        United States (us)                                                                                                                                         Web Interface                 http://127.0.0.1:4040                                                                                                                                      Forwarding                    http://908cd95a.ngrok.io -> localhost:5000                                                                                                                 Forwarding                    https://908cd95a.ngrok.io -> localhost:5000                                                                                                                                                                                                                                                                                                         Connections                   ttl     opn     rt1     rt5     p50     p90                                                                                                                                              0       0       0.00    0.00    0.00    0.00 

Log in to Amazon Apps and Services

Make sure you use the same email and you have set up your Alexa Echo device with.

https://developer.amazon.com/edw/home.html#/

Alexa Skills Kit

Easily add new skills to Alexa

Installing TensorFlow 1.0 as conda environment with yml definition file on MacBook Pro with NVidia GPU support

My conda environment yml file on GitHub

https://github.com/UkiDLucas/uki.guru/blob/master/conda_enviroment_GPU.yml

Execute yml file, set environment

$ conda env create --file .../conda_environment.yml
$ conda info --envs  

$ source activate py352_tf_gpu

Update jupyter notebook (iPython) kernel with new environment we created

$ python -m ipykernel install --user --name py352_tf_gpu --display-name "conda env py352_tf_gpu"$ jupyter notebook

Test run in jupyter notebook

You can see that jupyter notebook recognize GPU

You can see execution on MacBook Pro (late 2013 with NVidia). Both CPU and GPU are measured.

PLEASE READ THIS if you get DEAD KERNELS in GPU mode:
https://github.com/aymericdamien/TensorFlow-Examples/issues/38#issuecomment-280468579

Google TensorFlow 1.0 - Mac with Nvidia GPU installation and testing in jupyter notebook

TensorFlow 1.0 is promising more stable GPU build for Mac, let's test it.

Make sure you have CUDA installed

$ echo $DYLD_LIBRARY_PATH/usr/local/cuda/lib:

http://docs.nvidia.com/cuda/cuda-installation-guide-mac-os-x/#axzz4Ynrb42hi

$ echo $CUDA_HOME/Developer/NVIDIA/CUDA-8.0/

https://developer.nvidia.com/cudnn

$ sudo easy_install pip

Set variable with TensorFlow 1.0 binary

export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.0.0-py3-none-any.whl
$ echo $TF_BINARY_URL https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.0.0-py3-none-any.whl

Use pip to install TensorFlow

$ sudo pip3 install --upgrade  $TF_BINARY_URL Password:The directory '/Users/ukilucas/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.The directory '/Users/ukilucas/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.Collecting tensorflow-gpu==1.0.0 from https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.0.0-py3-none-any.whl  Downloading https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-1.0.0-py3-none-any.whl (89.0MB)    100% |████████████████████████████████| 89.0MB 8.8kB/s Requirement already up-to-date: numpy>=1.11.0 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from tensorflow-gpu==1.0.0)Requirement already up-to-date: wheel>=0.26 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from tensorflow-gpu==1.0.0)Requirement already up-to-date: six>=1.10.0 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from tensorflow-gpu==1.0.0)Requirement already up-to-date: protobuf>=3.1.0 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from tensorflow-gpu==1.0.0)Collecting setuptools (from protobuf>=3.1.0->tensorflow-gpu==1.0.0)  Downloading setuptools-34.2.0-py2.py3-none-any.whl (389kB)    100% |████████████████████████████████| 399kB 2.0MB/s Requirement already up-to-date: appdirs>=1.4.0 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from setuptools->protobuf>=3.1.0->tensorflow-gpu==1.0.0)Requirement already up-to-date: packaging>=16.8 in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from setuptools->protobuf>=3.1.0->tensorflow-gpu==1.0.0)Requirement already up-to-date: pyparsing in /Users/ukilucas/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages (from packaging>=16.8->setuptools->protobuf>=3.1.0->tensorflow-gpu==1.0.0)Installing collected packages: tensorflow-gpu, setuptools  Found existing installation: tensorflow-gpu 0.12.1    Uninstalling tensorflow-gpu-0.12.1:      Successfully uninstalled tensorflow-gpu-0.12.1  Found existing installation: setuptools 34.1.1    Uninstalling setuptools-34.1.1:      Successfully uninstalled setuptools-34.1.1Successfully installed setuptools-34.2.0 tensorflow-gpu-1.0.0(tensorflow_gpu) uki@UkiPEsMcBookPro 192.168.1.24 19:44 dev $ 

Start jupyter notebook in the right (same) environment

(tensorflow_gpu) $ jupyter notebook

Try in jupyter notebook

import tensorflow as tf
import time
from tensorflow.python.client import device_lib
​
def get_available_CPU_GPU():
   devices = device_lib.list_local_devices()
   #return [x.name for x in devices if x.device_type == 'CPU']
   return [x.name for x in devices ]
​
print(get_available_CPU_GPU())
['/cpu:0', '/gpu:0']

start = timeit.timeit()
print ("starting")with tf.device('/gpu:0'):
    # [1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print (sess.run(c))
end = timeit.timeit()
print ("elapsed", end - start)starting
[[ 22. 28.]
[ 49. 64.]]
elapsed -0.003607149003073573

It is a win so far, time will show if it is usable.

Reference:

https://www.tensorflow.org/install/install_mac

Nvidia just released an update to their CUDA drivers for Mac

Nvidia CUDA just released an update to their Mac OS X driver.

Jupyter python user input forms

REFERENCE:

Building Interactive Dashboards with Jupyter

Corrupt NVidia driver causes user to be stuck in the login screen

I was playing with dual monitors on my NVidia GTX 1050Ti card and at some point, the UI stopped to allow me to log in. It was going in the loop after each (correct) login.

Steps to fix it:

Press Ctrl+Alt+F1

or F2, F3

Log into in Command Line

Remove NVidia drivers

$ apt-get purge nvidia-*

Add repository

add-apt-repository ppa:graphics-drivers/ppa 

Refresh packages

sudo apt-get update

Install your NVidia driver  GTX 1050Ti driver 275.10

apt-get install nvidia-375

Reboot

sudo reboot

This has fixed it for me!

Python: converting PIL.Image to nympy array and back, resizing image

The example, I have used today:

import PIL
import numpy
from PIL import Image

def resize_image(numpy_array_image, new_height):
    # convert nympy array image to PIL.Image
    image = Image.fromarray(numpy.uint8(numpy_array_image))
    old_width = float(image.size[0])
    old_height = float(image.size[1])
    ratio = float( new_height / old_height)
    new_width = int(old_width * ratio)
    image = image.resize((new_width, new_height), PIL.Image.ANTIALIAS)
    # convert PIL.Image into nympy array back again
    return array(image)

stackoverflow.com

Installing Keras (with TensorFlow and Thano) using Python conda

(tensorflow_cpu) $ anaconda search -t conda keras | grep forge

     conda-forge/keras         |    1.0.7 | conda           | linux-64, win-32, win-64, osx-64

(tensorflow_cpu) $ conda install -c conda-forge keras=1.0.7

conda_clone

You can clone (duplicate, copy) environments, this is useful when you are experimenting with packages whereas everything else is working fine:

- Existing tensorflow_gpu
- Newly created tensorflow_cpu

$ conda create --name tensorflow_cpu --clone tensorflow_gpu

List your envirmonets :

$ conda info --envs
# conda environments:
#
tensorflow * /Users/ukilucas/anaconda3/envs/tensorflow
tensorflow_cpu /Users/ukilucas/anaconda3/envs/tensorflow_cpu
tensorflow_gpu /Users/ukilucas/anaconda3/envs/tensorflow_gpu
root /Users/ukilucas/anaconda3

Change to new one:

$ source activate tensorflow_cpu

Register this enviroment with Jupyter Notebook

$ python -m ipykernel install --user --name tensorflow_cpu --display-name "conda env tensorflow_cpu"

Setting Jupyter kernel to match Python conda environment

For Jupyter kernel to reflect changes in conda env (tensorflow), I had to set up the following:

$ source activate tensorflow
$ sudo pip install ipykernel
$ python -m ipykernel install --user --name tensorflow --display-name "conda env tensorflow"

Installing TensorFlow GPU on MacBook Pro (late 2013)

Note, installation works, but GPU TensorFlow does not.




CPU VERSION (tensorflow 0.12.1):

import tensorflow as tf
print("OUT: tensorflow imported")

OUT: tensorflow imported

from tensorflow.python.client import device_lib
def get_available_CPU_GPU():
    devices = device_lib.list_local_devices()
    #return [x.name for x in devices if x.device_type == 'CPU']
    return [x.name for x in devices ]

print(get_available_CPU_GPU())

['/cpu:0']


SAME CODE GPU VERSION (tensorflow-gpu 0.12.1 ):


OUT: tensorflow imported

print(get_available_CPU_GPU())

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
 in ()
----> 1 from tensorflow.python.client import device_lib
      2 
      3 def get_available_CPU_GPU():
      4     devices = device_lib.list_local_devices()
      5     #return [x.name for x in devices if x.device_type == 'CPU']

ImportError: No module named 'tensorflow.python'

I have logged the issue:
https://github.com/tensorflow/tensorflow/issues/7193


=====

Check for newest version:
TensorFlow get started page

I previously did export TF_BINARY_URL=..

$ echo $TF_BINARY_URL

https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.1-py3-none-any.whl

$ sudo  pip3 install --ignore-installed --upgrade $TF_BINARY_URL

The directory '/Users/ukilucas/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/ukilucas/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting tensorflow-gpu==0.12.1 from https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.1-py3-none-any.whl

  Downloading https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.1-py3-none-any.whl (83.6MB)

    100% |████████████████████████████████| 83.6MB 8.5kB/s 

Collecting numpy>=1.11.0 (from tensorflow-gpu==0.12.1)

  Downloading numpy-1.12.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.4MB)

    100% |████████████████████████████████| 4.4MB 165kB/s 

Collecting protobuf>=3.1.0 (from tensorflow-gpu==0.12.1)

  Downloading protobuf-3.2.0-py2.py3-none-any.whl (360kB)

    100% |████████████████████████████████| 368kB 1.2MB/s 

Collecting six>=1.10.0 (from tensorflow-gpu==0.12.1)

  Downloading six-1.10.0-py2.py3-none-any.whl

Collecting wheel>=0.26 (from tensorflow-gpu==0.12.1)

  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)

    100% |████████████████████████████████| 71kB 2.9MB/s 

Collecting setuptools (from protobuf>=3.1.0->tensorflow-gpu==0.12.1)

  Downloading setuptools-34.1.0-py2.py3-none-any.whl (389kB)

    100% |████████████████████████████████| 399kB 1.1MB/s 

Collecting appdirs>=1.4.0 (from setuptools->protobuf>=3.1.0->tensorflow-gpu==0.12.1)

  Downloading appdirs-1.4.0-py2.py3-none-any.whl

Collecting packaging>=16.8 (from setuptools->protobuf>=3.1.0->tensorflow-gpu==0.12.1)

  Downloading packaging-16.8-py2.py3-none-any.whl

Collecting pyparsing (from packaging>=16.8->setuptools->protobuf>=3.1.0->tensorflow-gpu==0.12.1)

  Downloading pyparsing-2.1.10-py2.py3-none-any.whl (56kB)

    100% |██████████████████    100% |██████████████████    100% |██████████████████    100% |████████████████████████████████| 61kB 2.0MB/s 

Installing collected packages: numpy, six, appdirs, pyparsing, packaging, setuptools, protobuf, wheel, tensorflow-gpu

Successfully installed appdirs-1.4.0 numpy-1.12.0 packaging-16.8 protobuf-3.2.0 pyparsing-2.1.10 setuptools-34.1.0 six-1.10.0 tensorflow-gpu-0.12.1 wheel-0.29.0

(tensorflow) $ conda list

# packages in environment at /Users/ukilucas/anaconda3/envs/tensorflow:

#

alabaster                 0.7.9                    py35_0  

appdirs                   1.4.0                    

appnope                   0.1.0                    py35_0  

appscript                 1.0.1                    py35_0  

numpy                     1.12.0                   

openssl                   1.0.2k                        0  

packaging                 16.8                     

pip                       9.0.1                    py35_1  

protobuf                  3.2.0                    

pyparsing                 2.1.10                   

python                    3.5.2                         0  

readline                  6.2                           2  

setuptools                27.2.0                   py35_0  

setuptools                34.1.0                   

six                       1.10.0                   

sqlite                    3.13.0                        0  

tensorflow-gpu            0.12.1                   

tk                        8.5.18                        0  

wheel                     0.29.0                   py35_0  

wheel                     0.29.0                   

xz                        5.2.2                         1  

zlib                      1.2.8                         3  

Python import error for sklearn.model_selection 0.17.1

ImportError: No module named 'sklearn.model_selection'

Before doing the embarrassing things I did below, read this:
Setting Jupyter kernel to match Python conda environment


Change to conda environment you need to update (tensorflow_gpu in may case).

List packages that have "scikit" in them.

(tensorflow_gpu) $ conda list | grep scikit

Nothing, nada.

$ conda install scikit-learn
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /Users/ukilucas/anaconda3/envs/tensorflow_gpu:
The following NEW packages will be INSTALLED:
    mkl:          2017.0.1-0            numpy:        1.11.3-py35_0         scikit-learn: 0.18.1-np111py35_1    scipy:        0.18.1-np111py35_1

(tensorflow_gpu)  $ conda list | grep scikit

scikit-learn              0.18.1              np111py35_1 

import sklearn
print (sklearn.__version__)
from sklearn.model_selection import train_test_split

0.18.1

You do not have to re-start Jupyter Notebook

that is all!

And here is my very long and embarrassing story:

Check the version of the package

import sklearn
print (sklearn.__version__)
0.17.1

Install the latest package

$ conda install scikit-learn

installs:
scikit-learn-0.18.1

However, already running jupyter notebook does not see the change.

Making sure I am in the right  (tensorflow) environment:

$ source activate tensorflow
$ conda install scikit-learn

installs:

    mkl:          2017.0.1-0        

    numpy:        1.11.3-py35_0     

    scikit-learn: 0.18.1-np111py35_1

    scipy:        0.18.1-np111py35_1

Still, Jupyter Notebook does not see scikit-learn 0.18.1

Restarting jupyter notebook from the same tensorflow environment

(tensorflow) uki@Uki ~ $ jupyter notebook ~/dev/

Still, Jupyter Notebook does NOT see scikit-learn 0.18.1

(tensorflow) uki@Uki ~ $ conda update --all

Fetching package metadata .......

Solving package specifications: ..........

# All requested packages already installed.

# packages in environment at /Users/ukilucas/anaconda3/envs/tensorflow:

#

mkl                       2017.0.1                      0  

numpy                     1.11.3                   py35_0  

openssl                   1.0.2k                        0  

pip                       9.0.1                    py35_1  

python                    3.5.2                         0  

readline                  6.2                           2  

scikit-learn              0.18.1              np111py35_1  

scipy                     0.18.1              np111py35_1  

setuptools                27.2.0                   py35_0  

sqlite                    3.13.0                        0  

tk                        8.5.18                        0  

wheel                     0.29.0                   py35_0  

xz                        5.2.2                         1  

zlib                      1.2.8                         3  

(tensorflow) uki@UkiPEsMcBookPro 192.168.1.77 09:03 ~ $ 

Jupyter Kernels

Ok, so there is a clear distinction of the conda env (in my case tensorflow) from which I am starting Jupyter Notebook and what the kernels see.

I tried to switch kernels Python [conda root] and Python [default], NO difference: sklearn.__version__ = 0.17.1

(py35) ~ $ conda list

# packages in environment at /Users/ukilucas/anaconda3/envs/py35:

#

cycler                    0.10.0                   py35_0  

freetype                  2.5.5                         2  

icu                       54.1                          0  

libpng                    1.6.27                        0  

matplotlib                2.0.0               np111py35_0  

mkl                       2017.0.1                      0  

mock                      2.0.0                    py35_0  

numpy                     1.11.3                   py35_0  

olefile                   0.44                     

openssl                   1.0.2k                        0  

pandas                    0.19.2              np111py35_1  

pbr                       1.10.0                   py35_0  

Pillow                    4.0.0                    

pip                       9.0.1                    py35_1  

protobuf                  3.1.0                    py35_0    conda-forge

pyparsing                 2.1.4                    py35_0  

pyqt                      5.6.0                    py35_2  

python                    3.5.2                         0  

python-dateutil           2.6.0                    py35_0  

pytz                      2016.10                  py35_0  

qt                        5.6.2                         0  

readline                  6.2                           2  

scikit-learn              0.18.1                   

scikit-learn              0.18.1              np111py35_1  

scipy                     0.18.1              np111py35_1  

seaborn                   0.7.1                    py35_0  

setuptools                27.2.0                   py35_0  

sip                       4.18                     py35_0  

six                       1.10.0                   py35_0  

sklearn                   0.0                      

sqlite                    3.13.0                        0  

tensorflow                0.12.1                   py35_1    conda-forge

tk                        8.5.18                        0  

wheel                     0.29.0                   py35_0  

xz                        5.2.2                         1  

zlib                      1.2.8                         3  

(py35) ~ $ jupyter notebook ~/dev/

NO difference: sklearn.__version__ = 0.17.1
Jupyter Notebook does NOT see scikit-learn 0.18.1

Removing Old Packages

$ conda listscikit-learn 0.18.1
sklearn 0.0
(py35) ~ $ conda remove scikit-learn
(py35) ~ $ pip uninstall scikit-learn
(py35) ~ $ pip uninstall sklearn
(py35) ~ $ conda install scikit-learn
(py35) ~ $ jupyter notebook ~/dev/

Did not help!

Check which Python Executable is being used

import sys

sys.executable

'/Users/ukilucas/anaconda3/bin/python'

$ conda list

# packages in environment at /Users/ukilucas/anaconda3/envs/py35:

Deleting Old and Reinstalling Jupyter Notebook

from jupyter_core.paths import jupyter_data_dir 

print(jupyter_data_dir())

/Users/ukilucas/Library/Jupyter

Deleting :
- /Users/ukilucas/Library/Jupyter
- ~/.ipython
- ~/.jupyter

(py35)  ~ $ jupyter --version

4.2.0

(py35)  ~ $ jupyter notebook --generate-config

Writing default config to: /Users/ukilucas/.jupyter/jupyter_notebook_config.py

(py35)  dev $ jupyter notebook

[I 09:52:59.160 NotebookApp] [nb_conda_kernels] enabled, 2 kernels found

Did not HELP!!

Removing Anaconda3 and Jupyter - fresh installs

$ rm -r ~/Library/Jupyter

$ rm -r ~/.ipython

$ rm -r ~/.jupyter

$ rm -r ~/anaconda3/

Installation (Mac OSX)

Downloads $ wget https://repo.continuum.io/archive/Anaconda3-4.3.0-MacOSX-x86_64.sh

Note the new Python is 3.6, but for TensorFlow I will need 3.5

bash Anaconda3-4.3.0-MacOSX-x86_64.sh

Anaconda3 will now be installed into this location:

/Users/ukilucas/anaconda3

Refresh Bash Terminal

$ source ~/.bash_profile
conda info --envs

root                  *  /Users/ukilucas/anaconda3

conda list

...

python                    3.6.0 

...

Create a new enviroment for Python 3.5

TensorFlow Installation instructions

# Python 3.5
$ conda create -n tensorflow python=3.5

At this time GPU Mac version is available only via pip

# Mac OS X, GPU enabled, Python 3.4 or 3.5:
(tensorflow)$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow_gpu-0.12.1-py3-none-any.whl

$ sudo  pip3 install --ignore-installed --upgrade $TF_BINARY_URL

Summary

Full delete of Anaconda3 and Jupyter and fresh re-installation fixed the problem.