parse result of `ls -l` with bash script

Question

I need to store the name of every file contained in a directory with a bash script and processes it in some way:

drwxrwxr-x  5 matteorr matteorr  4096 Jan 10 17:37 Cluster
drwxr-xr-x  2 matteorr matteorr  4096 Jan 19 10:43 Desktop
drwxrwxr-x  9 matteorr matteorr  4096 Jan 20 10:01 Developer
drwxr-xr-x 11 matteorr matteorr  4096 Dec 20 13:55 Documents
drwxr-xr-x  2 matteorr matteorr 12288 Jan 20 13:44 Downloads
drwx------ 11 matteorr matteorr  4096 Jan 20 14:01 Dropbox
drwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Music
drwxr-xr-x  2 matteorr matteorr  4096 Jan 19 22:12 Pictures
drwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Public
drwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Templates
drwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Videos

with the following command I'm able to split the result of ls -l in between all the spaces and then access the last element, which contains the name:

ls -l | awk '{split($0,array," ")} END{print array[9]}'

However it returns only the last line (i.e. Videos) so I need to iterate it over all the lines returned by the ls -l command.

• how can I do this?
• Is there a better way to approach this whole problem?

ADDED PART

To be a little more specific on what I need to do:

For all the files contained in a directory if it is a file I won't do anything, if it is a directory I should append the name of the directory to all the files it contains.

So supposing the directory Videos has the files:

-rwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 video1.mpeg
-rwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Video2.wmv

I need to rename them as follows:

-rwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 video1_Videos.mpeg
-rwxr-xr-x  2 matteorr matteorr  4096 Oct 18 18:43 Video2_Videos.wmv

Solution 1

A better way would be to use bash globbing

Just listing all files

echo *

Or doing something with them

for file in *; do
  echo "$file" # or do something else
done

Or recursively with bash 4+

shopt -s globstar
for file in **/*; do
  echo "$file" # or do something else
done 

Update to get directory name and append it to all files within it

Replace mv with an echo to test what it does. Also note ${file##*.} assumes the extension is everything after the last period, so if you had a file like file.tar.gz in directory on, below would turn it into file.tar_on.gz. As far as I know there is no easy way to handle this problem, though you could skip files with multiple . if you want)

#!/bin/bash
d="/some/dir/to/do/this/on"
name=${d##*/} #name=on
for file in "$d"/*; do
  extension=${file##*.} 
  filename=${file%.*}
  filename=${filename##*/}
  [[ -f $file ]] && mv "$file" "$d/${filename}_${name}.$extension"
done

e.g.

> ls /some/dir/to/do/this/on
video1.mpeg  Video2.wmv
> ./abovescript
> ls /some/dir/to/do/this/on
video1_on.mpeg  Video2_on.wmv

Explanation

In bash you can do this

• ${parameter#word} Removes shortest matching prefix
• ${parameter##word} Removes longest matching prefix
• ${parameter%word} Removes shortest matching suffix
• ${parameter%%word} Removes longest matching suffix

To remove everything anything (*) before and including the last period, I did below

 extension=${file##*.} 

To remove everything including and from the last period, I did below (think about shortest match here as going from right to left, e.g. * looks for any non-period text right to left, then when it finds a period it removes that whole section)

filename=${file%.*}

To remove everything up to and including the last /, I did below.

filename=${filename##*/}

Some other notes:

• "$d/${filename}_${name}.$extension" Variables can have _ so I switched syntax for a couple of variables here for it to work
• "$d"/* Expands to every file of any type (regular, dir, symlink etc...) directly in "$d"

Solution 2

What is wrong with

ls > myfile.txt

This will only list the file names (nothing else) and send them to myfile.txt

If you want to go the awk route, just do

ls -l | awk '{print $9}'

The default action of awk is to split fields on space - and this prints the 9th field for every line…

If you want to do other things with the file names, you can just extend your awk script. For example, an array with these file names could be created with

ls -l | awk '{a[NR]=$9}'

and you can use this array (called a) in further processing. If the processing requires something other than awk (from the comments I think it does), you would be better off with something that looks like

#!/bin/bash
for f in $1"/"*
do
if [ -d "$f" ] ; then
  ./listdir $f
else
  echo $f
fi
done

Save this as listdir in your current directory, and you're good to go.

./listdir .

Will list the entire directory, recursing down (with full relative path appended) as needed.

If you want this to be available "from anywhere" (it is a pretty useful command after all) you would put it somewhere in your path (and do a "rehash" command so it will be "known"); then you don't need the ./ at the start of the command.

Solution 3

Good question! Glad you asked. Parsing ls's output is rarely the right thing to do. There are myriad ways to process a list of files. It depends what you want to do with them.

Here are some examples of things you can do. I've used touch as an example command. Replace that with whatever command or commands you want to do.

1. To run a command over multiple files, often you can simply pass all the files on the command-line.

   touch /var/myapp/*
   

2. To loop over the files in the current directory:

   for file in *; do
       touch "$file"
   done
   

3. To loop over files in another directory:

   for file in /some/dir/*; do
       touch "$file"
   done
   

4. To rename files named *.txt to '*.bak', both here and in sub-directories:

   find . -name '*.txt' -exec mv {} {}.bak \;
   

5. To delete JPEGs in Bob's home directory (damn you Bob and your wandering eyes):

   find ~bob/ -name '*.jpg' -delete
   

6. To loop over files recursively and do complicated things to them:

   find /dir/to/search -print0 | while read -d $'\0' file; do
       echo "$file"
       touch "$file"
   
       if [[ -L $file ]]; then
           # $file is a symlink, do something special
       fi
   done
   

Solution 4

ls -l | awk '{split($0,array," ")} {print array[9]}'

or

ls -l | awk '{print $9}'

but why not just ls?

View more solutions

Comments

[ctrl-alt-delor]

Where is myfile.txt from? it is not in the question.

[Matteo]

thks, but how can I iterate over the list one item at the time?

[Matteo]

thks, but how can I iterate over the list on item at the time?

[Floris]

@richard - The question said "I need to store"… I decided to store it in a file.

[Floris]

@matteo - how do you want to iterate? The awk line I gave gives you one value for each iteration. You can put other expressions inside the awk script.

[ctrl-alt-delor]

Sorry, you are correct.

[Matteo]

For all the files contained in a directory if it is a file I won't do anything, if it is a directory I should append the name of the directory to all the files it contains.

[abligh]

ls puts the output in one column if redirected to something other than a terminal. Type ls | cat to check. If you really don't believe the automatic terminal detection will work, you can use ls -1 (that's a numeric one, not lowercase the letter L)

[Matteo]

Thanks for the great answer! I'm trying it right now

[Reinstate Monica Please]

@Matteo No problem. In case you have a file with a multi dot extension, see my comment above the script. Handling this would get far messier and probably involve an array with known multi-dot extension and testing against it or something similar.

[Matteo]

The script is working fine, could I just ask you to comment it a bit more? I am confused by all the #,%,.,/,* characters...

[Reinstate Monica Please]

@Matteo Updated with explanation. Also, I forgot a fairly important part in the original script. If you only want to mv regular files, you should do something like [[ -f $file ]] && mv, since * expands to all files including subdirs.

[Floris]

+1 for some very cool bash tricks. I learn something new here every day. Today, this answer was "something new". Thanks! Note - I fixed some typos. Please make sure I didn't accidentally change the meaning of your answer.

[Reinstate Monica Please]

@Floris Thanks for the fixes.

[tripleee]

Maybe note that find -print0 is a nonportable GNU find extension.