Spring Integration SFTP Example with Spring Boot

Question

We are using the latest Spring Boot for a Spring app and using the latest Spring Integration for SFTP. I've been to the Spring Integration SFTP documentation site, and I took the Spring Boot Configuration as is:

 @Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
    DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
    factory.setHost("localhost");
    factory.setPort(port);
    factory.setUser("foo");
    factory.setPassword("foo");
    factory.setAllowUnknownKeys(true);
    return new CachingSessionFactory<LsEntry>(factory);
}

@Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
    SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
    fileSynchronizer.setDeleteRemoteFiles(false);
    fileSynchronizer.setRemoteDirectory("/");
    fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
    return fileSynchronizer;
}

@Bean
@InboundChannelAdapter(channel = "sftpChannel")
public MessageSource<File> sftpMessageSource() {
    SftpInboundFileSynchronizingMessageSource source =
            new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
    source.setLocalDirectory(new File("ftp-inbound"));
    source.setAutoCreateLocalDirectory(true);
    source.setLocalFilter(new AcceptOnceFileListFilter<File>());
    return source;
}

@Bean
@ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
    return new MessageHandler() {

        @Override
        public void handleMessage(Message<?> message) throws MessagingException {
            System.out.println(message.getPayload());
        }

    };
}

Let me be clear, after cutting and pasting, there are some unit tests that run. However, when loading the application context there was an error message because the Polling wasn't there.

When I googled that error, other posts on StackOverflow said I also had to add to remove this error message when loading the application context.

@Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata defaultPoller() {

    PollerMetadata pollerMetadata = new PollerMetadata();
    pollerMetadata.setTrigger(new PeriodicTrigger(60));
    return pollerMetadata;
}

When I added this code, THEN at least my build would work and the tests would run because the application context was now being loaded correctly.

Now I am looking for a code sample on how to make this work and move files? The Spring Integration SFTP examples on GitHub are ok, but not great ... far from it.

The Basic Spring Integration Example shows how to read files from an SFTP Server, if the data is configured with an application-context.xml file. Where is the example where a Spring Boot configuration is used, and then the code to read from that server, and the code for the test?

I understand that regardless of whether you use a Java class for Spring Boot configuration or an application-context.xml file ... the working code should work the same for autowired SFTP channels and some inbound channel adapter.

So here is the code, I am trying to make work:

@Component
@Profile("sftpInputFetch")
public class SFTPInputFetcher implements InputFetcher
{
    // The PollableChannel seems fine
    @Autowired
    PollableChannel sftpChannel;

    @Autowired
    SourcePollingChannelAdapter sftpChannelAdapter;

@Override
public Stream<String> fetchLatest() throws FileNotFoundException
{
    Stream<String> stream = null;
    sftpChannelAdapter.start();
    Message<?> received = sftpChannel.receive();
    File file = (File)received.getPayload();
    // get Stream<String> from file
    return stream;
}

Currently, "sftpChannelAdapter.start();" is the part I am having trouble with. This implementation does not find the "SourcePollingChannelAdapter" class.

If this was defined in the classic XML application context with an "id" then this code autowires just fine. With a Spring Boot configuration, it doesn't look like you can define an "id" for a bean.

This just stems from my lack of knowledge on how to convert from using a traditional application-context XML file WITH annotations in the code, to using a complete Spring Boot application context configuration file.

Any help with this is much appreciated. Thanks!

Solution

I don't understand the question; you said

I had to add ... to make it work

and then

Now I am looking for a code sample on how to make this work?

What is not working?

You can also use

@InboundChannelAdapter(value = "sftpChannel", poller = @Poller(fixedDelay = "5000"))

instead of adding a default poller definition.

We will fix the docs for the missing poller config.

EDIT

I just copied the code into a new boot app (with the poller config) and it works as expected.

@SpringBootApplication
public class SftpJavaApplication {

    public static void main(String[] args) {
        new SpringApplicationBuilder(SftpJavaApplication.class).web(false).run(args);
    }

    @Bean
    public SessionFactory<LsEntry> sftpSessionFactory() {
        DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
        factory.setHost("...");
        factory.setPort(22);
        factory.setUser("...");
        factory.setPassword("...");
        factory.setAllowUnknownKeys(true);
        return new CachingSessionFactory<LsEntry>(factory);
    }

    @Bean
    public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
        SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
        fileSynchronizer.setDeleteRemoteFiles(false);
        fileSynchronizer.setRemoteDirectory("foo");
        fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.txt"));
        return fileSynchronizer;
    }

    @Bean
    @InboundChannelAdapter(channel = "sftpChannel", poller = @Poller(fixedDelay = "5000"))
    public MessageSource<File> sftpMessageSource() {
        SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
                sftpInboundFileSynchronizer());
        source.setLocalDirectory(new File("ftp-inbound"));
        source.setAutoCreateLocalDirectory(true);
        source.setLocalFilter(new AcceptOnceFileListFilter<File>());
        return source;
    }

    @Bean
    @ServiceActivator(inputChannel = "sftpChannel")
    public MessageHandler handler() {
        return new MessageHandler() {

            @Override
            public void handleMessage(Message<?> message) throws MessagingException {
                System.out.println(message.getPayload());
            }

        };
    }

}

Result:

16:57:59.697 [task-scheduler-1] WARN  com.jcraft.jsch - Permanently added '10.0.0.3' (RSA) to the list of known hosts.
ftp-inbound/bar.txt
ftp-inbound/baz.txt

Comments

[Gary Russell]

I just tested it with no problems; see the edit above.

[tjholmes66]

I just edited the question to make it more clear what was "not working." The "@InboundChannelAdapter" you have NOW has the poller added to it, which was not in the sftp documentation. You also made it work? Was that with the sftp/test on github? That test shows an application context xml file ... So the classpathxmlapplication context would not work.

[tjholmes66]

In the test, once I load the application context from SpringBoot, then I should be able to autowire the channel and the inboundchannel adapter. I have done this before, but just haven't done it with a Spring Boot Java Configuration class.

[Gary Russell]

I just copied the docs to a new boot app and added the poller. The docs are fixed already see the JIRA I linked to. This answer shows how to move the file with an advice. It uses the Java DSL. With Java config add the advice to the service activator's advice chain.

[tjholmes66]

Great. Sure, that's another way of making it work ... but I don't want to use it that way. I'll keep looking at it ... I'll state again that the Spring Integration examples needs way more examples with both using an XML application context file like you have now, AND with Spring Boot and Java DSL configuration files. If there were some examples with Autowiring, that would be helpful also.

[Gary Russell]

We are working on adding more java-based configuration examples, as time permits; contributions are always welcome.

[alexkov]

Problem with @Autowired examples still exists 3 years later. It is 2nd day now I'm trying to implement Spring Integration with remote SFTP server (get files list by command [not polling] > parse files with custom parser > insert to DB > on remote server move success ones to one folder and failed to another). And I just cannot find the proper way of doing it. I can't understand how to use commands like ["ls", "get", "move"] inside my service with business logic :(

[Gary Russell]

See this answer for a more recent example using LS , GET and RM.

[alexkov]

Thanks for the link. This is a good example of using commands, but the problem for me is that these commands are inside the IntegrationFlow in config, I can't understand how I can use the result of the command in my service. Maybe I should directly autowire the IntegrationFlow into my service and extract the command execution results from it, but this sounds stupid and I doubt this will work. For example: I defined the upload gateway in config with proper message handler and injected it into my service and it just works – I use it just for uploading, I need similar for download, move and read.

[Gary Russell]

Instead of using an SI flow for that, consider just using the SftpRemoteFileTemplate directly from your code for more flexibility. It supports all these operations, and also allows you to drop down to use JSch directly for even lower level operations.

[alexkov]

Thank you again, Gary. Looks like this is what I needed. I tried several commands and it works perfectly. I'm happy! :)