Sometimes when using bash, you can run in to an oddity where it keeps running the "wrong" command. Let's explore.
It was a dark and stormy night...
You want to install a Python package and you're using the new Python 3.6 installed via yum:
bash-4.4$ export PATH=/QOpenSys/pkgs/bin:$PATH bash-4.4$ python3 --version Python 3.6.6
When you install
xlsxwriter, though it can't be found:
bash-4.4$ pip3 install xlsxwriter Downloading/unpacking xlsxwriter Downloading XlsxWriter-1.1.1-py2.py3-none-any.whl (142kB): 142kB downloaded Installing collected packages: xlsxwriter Successfully installed xlsxwriter Cleaning up... bash-4.4$ python3 Python 3.6.6 (default, Jul 7 2018, 22:45:07) [GCC 6.3.0] on aix6 Type "help", "copyright", "credits" or "license" for more information. >>> import xlsxwriter Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'xlsxwriter'
Hmm... You investigate a bit and discover that
pip3 is the 5733-OPS version of pip!
bash-4.4$ pip3 --version pip 1.5.6 from /QOpenSys/QIBM/ProdData/OPS/Python3.4/lib/python3.4/site-packages (python 3.4) bash-4.4$ which pip3 /QOpenSys/usr/bin/pip3
You know what you need to do, install
pip3 with yum:
bash-4.4$ yum install -y python3-pip Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package python3-pip.noarch 0:9.0.1-2 will be installed --> Finished Dependency Resolution Dependencies Resolved =============================================================================================================== Package Arch Version Repository Size =============================================================================================================== Installing: python3-pip noarch 9.0.1-2 ibm 1.9 M Transaction Summary =============================================================================================================== Install 1 Package Total size: 1.9 M Installed size: 5.9 M Downloading Packages: Running Transaction Check Running Transaction Test Transaction Test Succeeded Running Transaction Installing : python3-pip-9.0.1-2.noarch 1/1 Installed: python3-pip.noarch 0:9.0.1-2 Complete!
Crisis averted. Or was it?
But still, it's using the OPS version!
bash-4.4$ pip3 --version pip 1.5.6 from /QOpenSys/QIBM/ProdData/OPS/Python3.4/lib/python3.4/site-packages (python 3.4)
What the heck?
This is because bash caches command name lookups. So when you first run a command, it looks up the name via the
$PATH and then caches it away in a hash for later lookup later. If you install a different command with the same name in to a folder earlier on the path, it won't know and will continue to use the older version. It's deceiving because the
which command will show the right output:
bash-4.4$ which pip3 /QOpenSys/pkgs/bin/pip3
This is because
which is a separate command and not part of bash, so it doesn't know anything about bash's cache and always uses the
$PATH to find the binary.
Instead, we can use the
type bash built-in to determine where it's coming from:
bash-4.4$ type pip3 pip3 is hashed (/QOpenSys/usr/bin/pip3)
Here, pip3 has already been found at
/QOpenSys/usr/bin/pip3 and is being cached there. To clear the cache, use
bash-4.4$ hash -r bash-4.4$ type pip3 pip3 is /QOpenSys/pkgs/bin/pip3 bash-4.4$ pip3 --version pip 9.0.1 from /QOpenSys/pkgs/lib/python3.6/site-packages (python 3.6) bash-4.4$ type pip3 pip3 is hashed (/QOpenSys/pkgs/bin/pip3)
After resetting the lookup cache, pip3 is no longer hashed and bash does a normal
$PATH lookup. Only after actully running it, will it get hashed again.
Why have I never seen this before?
Note that when you change the
$PATH variable, bash is smart enough to invalidate its cache. This means it's pretty rare to hit this issue. In order to encounter it you'd have to:
- run an unqualified command which already exists
- have that command be found in a directory later on the
- install a different version of that command in to a directory earlier on the
$PATHthan the previous command
Got any more info?
Sure. The docs for the hash (and other built-ins) are here.
What about shells other than bash?
Good question! For
zsh, the command is
rehash. For other shells, I'm not sure.