create a list of files located in multiple directories

Question

I have a set of data files (*.all) located in different folders

Documents/rawdata/Geob1005/1.all, 2.all, 3.all,...,n.all
Documents/rawdata/Geob1006/A.all, B.all, C.all,...,Z.all          and so on...

I have created a file called datalist.mb-1 in each directory which lists all the *.all files located in the same directory using:

/bin/ls -1 *.all | awk '{print $1" 58"}' >datalist.mb-1

so I get this:

Documents/rawdata/Geob1005/datalist.mb-1
Documents/rawdata/Geob1006/datalist.mb-1
and so on...

Now I need to create a "master" datalist in Documents/rawdata/ that points out to all the other datalists inside the directories of rawdata/

I need the outcome of the "master" datalist to look like this:

Geob1005/datalist.mb-1  -­1 
Geob1006/datalist.mb-1  -1  

Any idea of how to do this task?

Thanks in advance.

Solution 1

I finally managed to do it using Locate, actually I didn't need the full path but using awk was enough to just show just the path from the "geo" subdirectory.

This is how I did it:

locate ~/Documents/rawdata/*/datalist.mb-1 | awk 'BEGIN {FS="/"} {print $4 "/" $5 " -1"}' > master.txt

Solution 2

A really simple way to do this would be to use the locate command like so:

locate Documents/rawdata/Geob*/datalist.mb-1 > master.txt

That is if full paths are okay.

However, if you need the format which you described above with only the "geo" subdirectory and file you could use a simple little python script like this one:

#!/bin/python
f = open("/path/to/master.txt","r+")
paths = [x.split("/",3)[-1] + " -1" for x in f.readlines()]
f.seek(0)
f.write("".join(paths))
f.truncate()
f.close()

Slightly truncated version of napogeof's answer:

locate ~/Documents/rawdata/*/datalist.mb-1 | awk -F/ '{print $4 "/" $5 " -1"}' > master.txt

Comments

[don.joey]

Nice skills. Does it have to be in bash, or will python do? Oh, can you put your file paths and code in coding formatting (use these ``)?

[Symin]

what is the -1 at the end of Geob1005/datalist.mb-1 -­1 for, or the 58?

[napogeof]

@Symin The -1 at the end of 'Geob1005/datalist.mb-1 -1' is to also describe the format of the datalist located in that path. I am trying to follow a HowTo MB_System tutorial for bathymetry data processing. ldeo.columbia.edu/res/pi/MB-System

[napogeof]

I finally managed to do it using Locate, actually I didn't need the full path but using awk was enough to just show the path from the "geo" subdirectory. this is how I did it: Guys I finally managed to do the task using the Locate command locate ~/Documents/rawdata/*/datalist.mb-1 | awk 'BEGIN {FS="/"} {print $4 "/" $5 " -1"}' > master.txt