Apple’s search indexer “Spotlight” likes to run and use all the disk bandwidth and most of the CPU on my machine quite regularly. It seems to correlate with large numbers of file changes, and can keep running for hours. For example, it can’t handle doing a git checkout which changes a few thousand files. Backup programs which generate a large number of short lived temporary files also seem to incapacitate the indexer (for example, BackBlaze). Unfortunately, it’s not that easy to disable globally from the GUI. Luckily Apple provides a command line tool. To disable indexing for the main volume you can do:
sudo mdutil -i off /
I also discovered that deleting the index frees a shocking amount of space (around 7G on my machine!)
sudo mdutil -Ea
If your Mac is being slow, and not sure if you have this problem you can check your favorite monitoring tool for two processes: “mds” and “mdworker”
Tracing tools for executables are surprisingly hard to come by. Profilers are plentiful, but if you’re willing to take the performance hit and want to see a complete, recorded, call tree your options are rather limited. In the HPC world, the “standard” is probably Tau, but it’s a pain to integrate with non-trivial build systems and complicated to use. I also found a tool called “etrace” which uses a lesser known feature of gcc that adds a callback at each function entrance/exit (“-finstrument-functions”). Unfortunately, it relies on some specific behavior in nm to look up function names which doesn’t seem to work anymore.
Instead of a tool linked into your binary, you could also script GDB to record each function entry. Unfortunately, it turns out that this is both surprisingly difficult to do (more on that in a future post) and amazingly slow, particularly if the original source was in Fortran.
Luckily for me, Glibc has a set of functions which provide stack unwinding and symbol lookup (creatively named “backtrace()” and “backtrace_symbols()”). Combined with the technique used by etrace, we can easily write a tool which prints the name of the calling function (or a full backtrace) every time it enteres a non-library function (which is generally more desirable than looking at every function call anyway).
My quick implementation of this is available on github. To use it all you have to do is compile your code with -finstrument-functions (gcc/g++/gfortran and icc/ifort at least) and link with my c file. When you run your new executable, redirect the output to a file.
Once you have this log, you can post process to generate a variety of things. A (not well debugged) ruby script is provided in the github repository which strips out the function names. From the output of that script, you can post process with any number of standard *NIX tools. For example, if you pipe to “c++filt -n” you can get the C++ functions demangled. It would probably also be fairly easy to feed the output into dot/graphviz or LaTex and generate a nice call graph.
If you come up with an interesting visualization or other improvement, feel free to send a pull request. I’d also love to hear if anyone knows about any easier/better ways of doing this: blog-contact at kc2vjw dot com