Quick one this, as it’s a tricky problem I keep having to Google/SO. So I’m posting here for others but mainly myself too!
Here’s the situation: you have a folder (with, ooh, let’s say 140,000 separate MinION reads, for instance…) which contains a subset of files you want to move or copy somewhere else. Normally, you’d do something simple like a wildcarded ‘cp’ command, e.g.:
1 | host:joeparker$ cp dir/*files_I_want* some/other_dir |
Unfortunately, if the list of files matched by that wildcard is sufficiently long (more than a few thousand), you’ll get an error like this:
1 | -bash: /bin/cp: Argument list too long |
In other words, you’re going to have to be more clever. Using the GUI/Finder usually isn’t an option either at this point, as the directory size will likely defeat Finder, too. The solution is pretty simple but takes a bit of tweaking to work in OSX (full credit to posts here and here that got me started).
Basically, we’re going to use the ‘find’ command to locate the files we want, then pass each one in turn as an argument to ‘cp’ using ‘find -exec’. This is a bit slower overall than doing the equivalent as our original wildcarded command but since that won’t work we’ll have to lump it! The command* is:
1 | find dir -name *files_I_want* -maxdepth 1 -exec cp {} some/other_dir \;</p> |
Simple, eh? Enjoy 🙂
*NB, In this command:
- dir is the filesystem path to start the search; ‘find’ will recursively traverse the directory tree including and below this folder;
- -name *glob* gives the files to match;
- -maxdepth is how deep to recurse (e.g. 1 = ‘don’t recurse’);
- cp is the command we’re executing on each found file (could be mv etc);
- {} is the pipe argument standing for the found file;
- some/other_dir is the destination argument to the command invoked by -exec
