IPython Notebook workshop report: still plagued by installation issues

Update (Nov. 2014): IPython Notebook installation has advanced considerably since I originally published this post. Check out the Anaconda Python Distribution for an easy one-click installer for all your Python library needs.


Today I ran a small (~20 person) 1-hour workshop at Michigan State University focusing on installing IPython Notebook and using it as a research notebook. Since I knew it’s quite a pain to install IPython Notebook for the first time, I put together a set of installation instructions for both Windows and Mac/Linux and asked the attendees to attempt to install IPython Notebook ahead of time. On top of that, I held a 30-minute session before the workshop dedicated solely to installing IPython Notebook (which admittedly, helped a little bit).

By the end of it, I was able to get through the IPython Notebook demo, and many people seemed excited about using it for their research. Before everyone left, I had the attendees fill out a post-workshop survey to get their opinion on the workshop. In the post-workshop survey, I found some interesting trends in the responses:

  • All but one attendee indicated that they understood how to use IPython Notebook and found its interface intuitive
  • Half of the workshop attendees indicated that they planned to start using IPython Notebook for their research
  • Those that indicated that they don’t plan to use IPython Notebook for their research cited installation issues as the primary or secondary reason for not doing so

I think the third trend is the most telling: one of the biggest barriers to the adoption of IPython Notebook is installation issues. This isn’t an uncommon observation, either. In my case, despite all of my precautions, the workshop was still plagued by installation issues. At least 1/4 of the workshop attendees had these issues during the workshop, including:

  • Missing libraries being required for the newer version of IPython Notebook that don’t come with the installation package (not even EPD) by default
  • IPython Notebook not running in Chrome, only Firefox
  • pandas compatibility issues with the newer version IPython Notebook
  • Newer ipynb files not being backwards-compatible with older IPython Notebook versions

This highlights two key some key issues the IPython community needs to work on before IPython can reach a broader audience: (1) we cannot rely on EPD to maintain a free, update-to-date Python package installer and (2) we need to sort out the IPython installation process and distill it to a single, double-click install file.

UPDATE: Fernando Perez offered some easy installation instructions that seem to have solved the installation issues. I will report on how these work out in an upcoming workshop soon!

Randy is a PhD candidate in Michigan State University's Computer Science program. As a member of Dr. Chris Adami's research lab, he studies biologically-inspired artificial intelligence and evolutionary processes.

Posted in ipython, outreach Tagged with: , , ,
  • http://twitter.com/ccordoba12 Carlos Córdoba (@ccordoba12)

    Randy, I think people should really promote more Python(x,y) for Windows. It’s free as in speech (GPLv3), comes with a lot of scientific packages and it’s very up to date. i don’t see EPD working on the long run because of its restrictions.

    Another pretty good alternative is WinPython, which runs from an USB stick and already has a package manager which installs Gökhan packages.

    On Mac, I have created an Spyder installer (see our website) which comes with the most important libraries embedded: numpy, scipy, matplotlib, IPython, for now, and pandas, PIL, sympy, sklearn and skimage on the next release. Once IPython guys add multi-directory support to the notebook I could easily add it too. Another option is using Chris Fonnesbeck superpack.

    I think a viable alternative for the future is to have a really great graphical package manager (something similar to RStudio’s one) connected to a community maintained repos, similar to the ones used on Linux distros. Continuum.io guys already have one, but it’s only console based.

    • http://gravatar.com/monkmartinez monkmartinez

      I wrote up a little post on how to install iPython-Notebook for Mountain Lion… I didn’t cover windows. But hey, it may provide some help for you in the future: http://michaelmartinez.in/installing-ipython-notebook-on-mountain-lion.html

    • Wes Turner

      +1 for Python(x,y) or WinPython .

      I started to write a shell script for installing IPython notebook with brew on OSX (gist #3196564)

      A maintained graphical installer would be great as well.

  • http://brianegranger.com ellisonbg

    Randy, hi this is Brian Granger, the lead developer of the IPython Notebook. I know that installation is still a bit painful, but I think some of the things your students were running into have solutions. I am actually excited about your results – to think that a full 1/2 of the students said there were going to use the notebook is pretty great given the installation issues they ran into. But there is more work to be done on this front.

    First about your installation instructions. There is a much simpler way of getting the latest ipython for EPD free that doesn’t require using git (ouch that is painful for new folks). The following should work:

    easy_install http://github.com/ipython/ipython/tarball/master

    That should be all that is required as EPD free comes with tornado, jinja2, pyzmq, etc. What libraries were missing from EPD free that the notebook needs? I regularly use this approach with University students and haven’t had too many problems.

    Also, I see that you installation instructions include manual installation of pandas. But EPD free includes pandas. Seems like there is something funny going on…

    You mention issues with the compatibility of the ipython notebook documents. If everyone is using the latest ipython, they should be able to open both v2 and v3 notebooks. Did you also have some people running an older version of the notebook? That would definitely cause frustration as they wouldn’t be able to open older v2 notebooks.

    I should mention that it is strongly recommended to not run older versions of the notebook – it is moving very quickly and github master is by far the most stable version of the notebook out there. At this point *all* notebooks should be v3 notebooks. We expect the v3 notebook format to be pretty stable for the coming future.

    Finally, you mention “pandas compatibility issues with the newer version of the notebook.” Can you explain more? We are unaware of any such issues and I know that Wes M. (the pandas author) uses the IPython notebook extensively. If there is a problem though, we need to know about it so we can fix it…

    • http://www.randalolson.com Randy Olson

      Hi Brian, thanks for popping in! :-)

      First about your installation instructions. There is a much simpler way of getting the latest ipython for EPD free that doesn’t require using git (ouch that is painful for new folks). The following should work:

      easy_install http://github.com/ipython/ipython/tarball/master

      That should be all that is required as EPD free comes with tornado, jinja2, pyzmq, etc. What libraries were missing from EPD free that the notebook needs? I regularly use this approach with University students and haven’t had too many problems.

      Also, I see that you installation instructions include manual installation of pandas. But EPD free includes pandas. Seems like there is something funny going on…

      I’ll be sure to incorporate these into my instructions for the next workshop! Admittedly my installation instructions were a bit hacky; I was retracing the steps I had taken to get IPython Notebook working months ago.

      You mention issues with the compatibility of the ipython notebook documents. If everyone is using the latest ipython, they should be able to open both v2 and v3 notebooks. Did you also have some people running an older version of the notebook? That would definitely cause frustration as they wouldn’t be able to open older v2 notebooks.

      Yes, some students hadn’t updated to the latest version of IPython Notebook, and I had prepared an example notebook in v3 format. Thus when the attendees with the older version of IPython Notebook tried to open the notebook in v3 format, they got an error stating it couldn’t be opened. My plan moving forward for this is to not rely on a pre-made example notebook, and instead have the students create a new notebook and code along with me.

      I should mention that it is strongly recommended to not run older versions of the notebook – it is moving very quickly and github master is by far the most stable version of the notebook out there. At this point *all* notebooks should be v3 notebooks. We expect the v3 notebook format to be pretty stable for the coming future.

      I tried to get everyone to install the latest version of IPython Notebook, but many people seem to have stopped at the “install EPD” step, and the latest version of EPD has a much older version of IPython Notebook. This is why I reached the conclusion that we need a single-step installation process to reach a broader audience: people who are new to programming are unlikely to invest much time into installing software.

      Finally, you mention “pandas compatibility issues with the newer version of the notebook.” Can you explain more? We are unaware of any such issues and I know that Wes M. (the pandas author) uses the IPython notebook extensively. If there is a problem though, we need to know about it so we can fix it…

      This was the most perplexing issue for me, and only one attendee had it. The attendee followed my installation instructions for Mac verbatim. At first, IPython Notebook wouldn’t work on Chrome, so we switched to Firefox and it worked fine. Then we tried reading the data file in my workshop github repo with pandas:

      from pandas import *
      exampleData = read_csv("parasite_data.csv")

      and we’d get some strange error about “incorrect number of columns.” Another attendee sitting right next to us with the same setup/installation didn’t have this issue. Very strange…

      • http://brianegranger.com ellisonbg

        I just looked through our code and back at EPD free again. IPython doesn’t require distribute unless it is on Python 3. We do however require “setuptools” (setuptools or distribute) on Windows Python 2. BUT, EPD free comes with distribute so there shouldn’t be any need to install it again.

        As far as Firefox versus Chrome that one is very odd. Chrome tends to be better supported than even Firefox – I develop 90% of the time in Chrome and only run Firefox to check the notebook. Here is my guess as to what was happening. I think the person tried the older version of the notebook on Chrome and then updated IPython. When they retried Chrome, it used the cached version of some of the files, causing it to break. But a simple page reload would fix that issue. If the user then went and tried Firefox (which hadn’t loaded the older version) it would have worked fine.

        For pandas, that is really odd. I would honestly have to sit there with the student and try debugging things. They may have had a previous Python install that was using an older version of pandas that wasn’t able to handle that csv file. Most of the install problems I do run into are related to people not starting from a clean slate. That can screw things up in a hurry.

        One thing I like to do with students is have them check their install by doing things like:

        which python
        which ipython

        These should show versions in an EPD labeled directory on Mac/Linux.

        Then I test the IPython install location by doing:

        import IPython
        IPython.__file__

        And make sure that looks good.

        Cheers,

        Brian

    • Sayantan Hore

      Hi Brian, thanks for this information. I am new to iPython and found it very useful in my daily work. I ran into some issues with ipython notebook. I don’t have much knowledge about package management so I followed official installation instruction on iPython website. From Ubuntu terminal I issued commands :

      sudo apt-get install python-matplotlib python-scipy python-pandas python-sympy python-nose

      and

      sudo apt-get install ipython-notebook

      it installed V2 notebook.

      On windows I have installed Enthought Canopy Academic version which came with V3 notebook. Now I am having compatibility issues.

      I have projects setup on Ubuntu so I need to upgrade my V2 notebook to V3 on Ubuntu. Kindly help me to do that.

  • http://fperez.org Fernando Perez

    One quick comment (I’m in a lecture)…

    For general purpose use, I suggest not even trying to pull a master taball: it makes it hard to know precisely what each student ends up with as it depends on when they pull. And though we’re pretty careful about keeping master very usable, you never know, bugs do happen…

    But EPD provides a proper package updater, so you can type

    enpkg enstaller
    enpkg ipython

    (prepended with sudo on a mac). That will install an officially managed 0.13.1 straight from the EPD servers, and it works even with EPD Free. Here are some more details:

    http://fperez.org/py4science/workshop_geilo_2013.html

    I’ve used this approach in two recent workshops and pretty much everyone has been able to get 0.13.1 working in a few minutes, except for people who had really broken machines to begin with (woefully insufficient ram, massive permissions mess on their disk, etc).

    It’s very true that we should make the install process easier, but I also think that EPD+these two commands for now does work very well, and in a few minutes you can have a whole room up and running.

    • Wes Turner

      It is also very easy to install tagged versions from GitHub:

      easy_install https://github.com/ipython/ipython/archive/rel-0.13.1.zip

      To upgrade, update ‘rel-0.13.1.zip’ to the newer version.

      • http://fperez.org Fernando Perez

        Very true, Wes! Thanks for that reminder, I keep forgetting about the auto-generated tar/zip archives github makes for tags.

  • Rutger

    Installation and packaging issues are a Python-wide problem i think, not specific to iPython. As for Windows, if you want the easy way you should use the installers from Christoph Gohlke at: http://www.lfd.uci.edu/~gohlke/pythonlibs/

    He has nice, up-to-date, installers for all iPython dependencies and many more, so its just a matter of ‘next, next finish’. I have used them many times to install iPython on different machines and it always works out of the box.

    A site like his’ is really an example of how easy it can be for end users. Imagine something like that combining all platforms and many more packages. It would make getting started with Python as easy as the language itself.

  • Wes Turner

    Helpful, thanks!

    AFAIK, this is the documentation for installing IPython and for IPython notebook:
    * https://github.com/ipython/ipython/blob/master/docs/source/install/install.txt
    * https://github.com/ipython/ipython/blob/master/docs/source/interactive/htmlnotebook.txt

    I’m not sure what the intended scopes of these documentation documents are (e.g. would it be appropriate/helpful to link to all of the various techniques for installing IPython notebook?)

    • Wes Turner

      … Here are a few [more] ways to install and maintain an IPython notebook installation: http://www.reddit.com/r/Python/comments/1785dd/ipython_notebook_workshop_report_one_of_the/c83lg8u

      • http://www.randalolson.com Randy Olson

        Thank you for compiling all of these! This is great! Right now, I’m testing Fernando’s suggestion using EPD and the enpkg commands to install and update IPython to the latest stable version. Looks like it’s working solidly on Mac — now need to test Linux and Windows.

    • http://www.randalolson.com Randy Olson

      “easy_install ipython[zmq,qtconsole,notebook,test]” from that installation documentation didn’t work for many people. IIRC that command requires some sort of Distribute library or something.

  • http://twitter.com/cournape David Cournapeau (@cournape)

    Randy, I am sorry for the trouble you’re having with ipython. If you have any issue with EPD + ipython notebook, don’t hesitate to let us know. enpkg enstaller + enpkg ipython is all you should need to get the notebook working (and we are working to make that simpler).

    More generally, the current distribution infrastructure in python is completely inadequate for advanced needs, especially on windows and/or when you have binaries dependencies. Some of us are working to make that better, though, but it is a long journey.

    • http://www.randalolson.com Randy Olson

      Hi David!

      I’m currently exploring the EPD Free + empkg route on all OSs to see how that works. If that pans out, I will have to redact my statement about EPD. ;-)

      Thank you for your hard work! I know how difficult it can be to wrangle all of Python’s packages, especially when you need to support multiple OSs.

      Cheers, Randy

  • http://alfredessa.com Alfred Essa

    I recently installed EPD Free + Pandas on several Macs and Windows. It was quite smooth. Only hitch on a Mac is that one has to install the Command Line Tools with XCode.

    • http://www.randalolson.com Randy Olson

      Yep, the install process seems to be really streamlined now, especially following the instructions that Fernando provided in the comments here.

About this blog

The data visualizations on this blog are the result of my “data tinkering” hobby, where I tackle a new data analysis problem every week. If I find something interesting, I report my findings here to share with the world.

If you would like to use one of my graphs on your website or in a publication, please email me.

Archives

Enter your email address to subscribe to this blog and receive notifications of new posts by email.