How to read all files in a folder with spring-batch and MultiResourceItemReader?

Question

I want to configure spring-batch to read all csv files inside a specific folder sequentially.

The following does not work because the delegate will try to open a file named *.csv, which of course is invalid. What do I have to change here?

    @Bean
    public ItemReader<String> reader() {
        MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
        reader.setResources(new Resource[] {new FileSystemResource("/myfolder/*.csv")});
        reader.setDelegate(new FlatFileItemReader<>(..));
        return reader;
}

The equivalent xml configuration would be written as follows, how could I rewrite it to java only config?

<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader">
        <property name="resources" value="/mypfolder/*.csv"/>
        <property name="delegate" ref="flatFileItemReader"/>
</bean>

Solution 1

I think you should use a PathMatchingResourcePatternResolver.

Solution 2

Use PathMatchingResourcePatternResolver like this.

@Bean
public ItemReader<String> reader() {
    Resource[] resources = null;
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();   
    try {
        resources = patternResolver.getResources("/myfolder/*.csv");
    } catch (IOException e) {
        e.printStackTrace();
    }
    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(new FlatFileItemReader<>(..));
    return reader;
}

Solution 3

Alternatively @Value annotation can be used to let ResourceArrayPropertyEditor convert string to Resource[].

@Bean
@StepScope
public ItemReader<String> reader(
        @Value("file:/myfolder/*.csv") Resource[] resources
) {
    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(new FlatFileItemReader<>(..));
    return reader;
}

Solution 4

Read multiple files from Absolute file path:

 @Bean
      public MultiResourceItemReader<POJO> multiResourceItemReader() {
        MultiResourceItemReader<POJO> resourceItemReader = new MultiResourceItemReader<POJO>();
            ClassLoader cl = this.getClass().getClassLoader();
            ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);

        Resource[] resources = resolver.getResources("file:" + filePath );
        resourceItemReader.setResources(resources);
        resourceItemReader.setDelegate(reader());
        return resourceItemReader;
      }

      @Bean
      public FlatFileItemReader<POJO> reader() {
        FlatFileItemReader<POJO> reader = new FlatFileItemReader<POJO>();
        reader.setLineMapper(new DefaultLineMapper() {{
          setLineTokenizer(new DelimitedLineTokenizer() {{
            setNames(new String[]{"id1", "POJO"});
          }});
          setFieldSetMapper(new BeanWrapperFieldSetMapper<POJO>() {{
            setTargetType(POJO.class);
          }});
        }});
        return reader;
      }

View more solutions

Comments

[Ashish]

Is there any way to print the skip,commit,read,write count per file?

[Ashish]

Is there any way in this approach to print the skip,commit,read,write count per file?