NiFi Instance goes down when List SFTP fails

Technical Question
#1

The NiFi instance was down, there were no recent changes in any processor group. upon examining the logs it was found that List SFTP failed. But it is not clear why would it cause the instance to go down and crash it again as soon as it restarts ?

#2

Upon further examination it was found that instance was crashing due to insufficient space on the disk. The error on ListSFTP Processor was recurring which caused continuous additions in the nifi-app.log file. The log file kept growing in size until the system crashed due to low disk space.

This can be resolved by using proper configuration settings for nifi logs in logback.xml file, located in “conf” directory.

By default the totalSizeCap property is not specified. It defines the size at which log roll back is started.

The following properties can be configured to handle roll back of log file -

  1. totalSizeCap
    Total size of all log files

  2. maxHistory
    No of files after which rollover starts

  3. maxFileSize
    max file size per log file

  4. fileNamePattern
    ${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log
    Creates a new log file every hour

  5. rollingPolicy class
    ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy

These settings ensure regular rollback of log files and prevents out of space issues on the disk.

1 Like