Loop through files in a folder in matlab

Question

I have a set of days of log files that I need to parse and look at in matlab.

The log files look like this:

LOG_20120509_120002_002.csv
(year)(month)(day)_(hour)(minute)(second)_(log part number)

The logs increment hourly, but sometimes the seconds are one or two seconds off (per hour) which means i need to ignore what they say to do loadcsv.

I also have another file:

LOG_DATA_20120509_120002.csv

which contains data for the whole hour (different data).

The overall objective is to:

 loop through each day 
     loop through each hour
         read in LOG_DATA for whole hour
         loop through each segment
             read in LOG for each segment
                 compile a table of all the data

I guess the question is then, how do i ignore the minutes of the day if they are different? I suspect it will be by looping through all the files in the folder, in which case how do i do that?

Solution 1

Looping through all the files in the folder is relatively easy:

files = dir('*.csv');
for file = files'
    csv = load(file.name);
    % Do some stuff
end

Solution 2

At first, you must specify your path, the path that your *.csv files are in there

path = 'f:\project\dataset'

You can change it based on your system.

then,

use dir function :

files = dir (strcat(path,'\*.csv'))

L = length (files);

for i=1:L
   image{i}=csvread(strcat(path,'\',file(i).name));   
   % process the image in here
end

pwd also can be used.

Comments

[Fantastic Mr Fox]

I had a feeling it would be something with dir, i couldn't quite put my finger on it though. Thanks for your response.

[Juan Sebastian Totero]

mmm this is not working for me... file = files simply copies files into file :(

[Isaac]

Try file = files'; it may require that files is a row-array.

[John Bensin]

@Isaac Keep in mind that load is designed for use with files containing MATLAB variables only, so even though it will work for CSV files that are completely numeric, if the header row contains variable names or the data are more complicated, it may be better to use textscan or csvread.

[Fantastic Mr Fox]

@JohnBensin I know this is late but that is not entirely true. load can get space separated data out of files and other specific things.

[Sibbs Gambling]

Works good. However, one potential pitfall: pay attention to the file order as a result of the file names, if the file order matters. I spent hours debugging and realized this. [facepalm]

[vigamage]

When I do this, I get an error saying Unable to read file 'abcdef.csv' : ': no such file or directory. why is that?

[Isaac]

@vigamage, are you targeting the current directory? Or are you using dir('some/path/*.csv')?

[Charlie Parker]

why do you need the ' (transpose)?

[Isaac]

@CharlieParker, because matlab for-loops treat row-vectors and column-vectors differently (or at least they did when I wrote this).

[Charlie Parker]

how do you actually deal with this without having to cd to a specific directory? I have a directory with more directories with those directory having files. It seems really dum to have to cd to directories then call dir, is there not a better solution?

[Isaac]

@CharlieParker, you can call dir directly on a subdirectory to get the contents of that subdirectory. You can also loop through the names returned by dir, and call isdir on each one to figure out if they are directories themselves. In this way you can recursively list all the files in a nested directory.

[Cris Luengo]

Instead of strcat, use fullfile: dir(fullfile(path,'*.csv')). See documentation: mathworks.com/help/matlab/ref/fullfile.html