Maarg Data Manager Import Flow

#1
  • Default Execution Mode:
    In the OMS data manager flow, the default execution mode was QUEUE. In the Maarg data manager flow, the default mode is QUEUE.
  • Scheduling:
    In OMS, there was a processPendingDataManagerLog job scheduled to run every 5 minutes (or a custom interval) to process pending imports. In the Maarg data manager flow, there is no need for this additional job. The new Data Manager scheduler runs every 60 seconds to check for pending imports.
  • Execution Behavior:
    • The OMS Data Manager executed imports sequentially, or if execution mode was set to async, it submitted imports to the OFBiz scheduler.
    • The new flow picks and executes imports based on configuration priority.
      • Two thread pools are available: Priority and Normal (each with 3 threads).
      • If config.priority > 6, it is considered a Priority import; otherwise, it goes into the Normal queue.
  • Thread Pool Management:
    If a configuration is set as multithreaded, the scheduler uses the existing thread pool for parallel processing instead of creating new threads. This ensures proper thread management.
  • Large File Processing:
    • For multithreaded processing, large files are split into chunks based on the groupBy field.
    • If no groupBy is defined, files are split into chunks of 50K records.
    • Each chunk is processed in a separate thread.
    • After processing, all error files are merged into a single error file.
  • Parameter Storage:
    • The OMS Data Manager used runtimeData to store job parameters as XML serialized content.
    • The Maarg data manager flow introduces a dedicated entity to store Data Manager parameters.
  • Content Handling:
    • The OMS flow used contentId to identify uploaded and error files.
    • The Maarg Data Manager flow uses the DataManagerContent entity to manage uploaded, error, and log files.
  • Sequential Processing Mode:
    If a configuration is set to run sequentially:
    • The scheduler fetches all Data Manager imports (DML), marks them as queued, processes them one by one, and finally marks them as finished.
    • If multithread set to Y the sequential process use the thread from the same pool to process file.
  • Processing Order:
    The scheduler fetches pending imports ordered by config.priority and submits each config ID for processing accordingly.
1 Like