I have a webscraping service that takes a bit too long (sometimes up to 2 hours). But I did a good configuration, that can handle long functions, however I got this error:
2025-02-18T10:40:00.017Z INFO 1 --- [one-saha] [cTaskExecutor-1] .i.o.s.i.a.s.WebScrappedArticlesReceiver : Calling web scraping service...
2025-02-18T10:40:00.017Z INFO 1 --- [one-saha] [cTaskExecutor-1] .i.o.s.i.a.s.WebScrappedArticlesReceiver : Calling scrapeNewsArticles...
2025-02-18T11:00:00.372Z ERROR 1 --- [one-saha] [cTaskExecutor-1] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.inspark.onesaha.sahainfo.infrastructure.adapter.scheduled.WebScrappedArticlesReceiver.scrape()
feign.FeignException$GatewayTimeout: [504 Gateway Time-out] during [POST] to [] [WebScrapingClient#scrapeNewsArticles()]: [<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.24.0 (Ubuntu)</center>
</body>
</html>
]
at feign.FeignException.serverErrorStatus(FeignException.java:289) ~[feign-core-13.5.jar!/:na]
and this is my Feign Client Configuration file:
@Configuration
public class FeignClientConfig {
private final ObjectMapper objectMapper;
public FeignClientConfig(ObjectMapper objectMapper) {
this.objectMapper = objectMapper;
}
@Bean
public Retryer feignRetryer() {
return new Retryer.Default(500, 3000, 3); // Initial interval, max interval, max attempts
}
@Bean
public Request.Options options() {
return new Request.Options(
180, TimeUnit.MINUTES, // connectTimeout in minutes (3 hours)
180, TimeUnit.MINUTES, // readTimeout in minutes
true
);
}
@Bean
Logger.Level feignLoggerLevel() {
return Logger.Level.FULL;
}
@Bean
public Encoder feignEncoder() {
return new JacksonEncoder(objectMapper);
}
@Bean
public Decoder feignDecoder() {
return new JacksonDecoder(objectMapper);
}
}
Can someone help me fix this problem ?