Frustrated by Python module management 6
I don’t really do any Python development myself, but at work I do support some automated testing infrastructure (in this particular case, I’m talking about a CruiseControl.rb installation), and some of the projects that use that infrastructure use Python. The setup is so that the tests are actually executed in the CC.rb server, so I have to have Python installed there, and it has to have some basic dependencies to be able to run the tests.
A couple of times something strange happened: suddenly, those tests would start failing with no apparent reason, and looking at the logs, it looked like some dependencies were not installed (error messages such as ImportError: No module named sqlalchemy). Of course that didn’t make any sense, because SQLAlchemy is needed for the tests and they were working like a charm for weeks. I was totally and completely confused by the error message, and I tried to install SQLAlchemy again. That solved the problem, luckily, so I decided to forget about it because it wasn’t my thing anyway.
But the problems appeared again. And again. And I got another error message that was really confusing, because it looked like Python was using some old version of some module (a version that wasn’t there anymore, because the code had been updated from SVN). So I just got tired of not knowing what was going on, and decided to investigate enough to find out the root of the problem. And I found something surprising.
What I found is that the famous python setup.py develop (that everyone told me to use) actually adds the “current directory” to the list of paths where Python searches for modules, so you can develop your module and use it from anywhere. I had heard some comment on that, but I didn’t quite get what it meant, and I don’t think the person that said it realised either.
The fun thing with setup.py develop is that when you have several branches of the same project in the same machine, and you use that to make the modules available… well, I guess that knowing which versions of which modules Python will use becomes an interesting question to say the least. I’m not saying that the way it works is necessarily wrong, but I do think it is dangerous, and people shouldn’t think of it as the “normal” way of developing modules in Python. It should be used with care.
After having realised that and thought about it a bit, I still don’t understand why those modules simply “dissappeared”, but it seems that there was some corruption of /usr/lib/python2.5/site-packages/easy_install.pth or similar (that file seems to be modified when you install packages with easy_install, and it had references to the directories I ran setup.py develop from, so that’s my main suspect for now). At least I know now that I could backup a working easy_install.pth file, and restore when we have problems again, but I’m far from happy with that “solution” ;-)
Also, I’m wondering what the hell should I do in the future to prevent more problems, because using setup.py develop sounds like a terrible idea to me. I tried to set PYTHONPATH instead, but apparently I failed. Any suggestions?
EDIT: I’m finally using PYTHONPATH. I have no idea what I tried last time, but using it was easy, quick and clean. I still have no idea why the hell Python sometimes forgets where some modules are, though.
Certainly using setup.py develop on a machine on which you are running automated tests sounds like a bad idea.
Are you using the same machine for running automated tests (on several branches) and development (on several branches)?
Unless you are actually developing SQLAlchemy then you shouldn’t install it in development mode.
I would have all modules you depend on installed ‘normally’. Modules you are developing I personally wouldn’t have them installed at all, but keep them within my project structure and rely on the development / test environment to make sure they can be imported.
Polluting your test environment from your development environment should be avoided…
why not use python setup.py install? It installs to the /usr/lib/python/site-packages directory on python.
also, you should make sure you are using the same version of python for both the “python setup.py install” and running. If you have two versions, i.e. python2.4 and python2.5, you could by accident end up with a module installed for 2.4, but actually needed in 2.5.
as a debugging measure, start a python shell and do: import sys print sys.path
you’ll get the list of directories searched for your install. make sure to use the right python version like mentioned before.
PYTHONPATH should also work well (has for me). Have you exported it? i.e. export PYTHONPATH=/home/me/pythonlibs python myprog.py
or just prepend to the command line: PYTHONPATH=/home/me/pythonlibs python myprog.py
You should check out virtualenv. It lets you create a somewhat sandboxed python install for different projects. The biggest benefit is being able to have a separate place to install modules. I’ve done this in order to differentiate between projects I am working on and applications that run on my machine. This way I can verify what the dependencies really are as well as have an easy way to install two libraries at the same time. Good luck!
What is it that is so hard to understand?
If you want to you can just set the PYTHONPATH manually, the setup.py develop is just a convenience for you. I fail to see how Python package lookup is different from any other platform you are familiar with. It might work much better on Plan9 or some shit like that, but … ... :-)
Using setuptool and setup.py develop it will install the package in your python packages folder, which on Debian is by default the system folder, which will be shared by all users.
Probably the easiest for you (And Jonathan told me he already recommended it to you) is to use virtualenv – http://pypi.python.org/pypi/virtualenv
Also this discussion probably belong in an internal mailing list of some sort :)
I can has documentation?
http://docs.python.org/dev/install/index.html#inst-search-path
Hey, thanks for the comments!
Michael: No, that machine is only used for automated test runs. Several different projects though. BTW, I didn’t install SQLAlchemy in development mode, just with a regular easy_install… but for some reason Python sometimes “forgets” that it’s actually there (it gets lost from easy_install.pth somehow, but I have no idea why). When you say “installing normally”, do you mean in the system, like in /usr/lib/python/site-packages ? I’d rather not do that if possible, but installing in some “private directory” and using from there would be possible of course.
Sverre: PYTHONPATH didn’t work for me, maybe I did something wrong? Am I supposed to install things first so it works, or should I be able to just point PYTHONPATH to the directory with a fresh VCS checkout?
And thanks for the virtualenv thing, I’ll have a look!
Oh, and sorry for the comments not appearing, I was receiving tons of spam so I decided to make the comments invisible until I approved them… I didn’t really expect showing up on reddit :-P