Class DownloadScheduler
The download job is invoked at a divisible fraction of the desired data freshness to accommodate potential retries. E.g., for 30-minute data freshness with up to two retries on error, the cron schedule for the job should be set to 10 minutes.
The processing of each BSA download progresses through multiple stages as described in
DownloadStage
until it reaches one of the terminal stages. Each stage is check-pointed on
completion, therefore if an invocation fails mid-process, the next invocation will skip the
completed stages. No new downloads will start as long as the most recent one is still being
processed.
When a new download is scheduled, the block list checksums from the most recent completed job
is included. If the new checksums match the previous ones, the download may be skipped and the
job should terminate in the NOP
stage. However, if the checksums have stayed unchanged
for longer than the user-provided maxNopInterval
, the download will be processed.
The BSA downloads contains server-provided checksums. If they do not match the checksums
generated on Nomulus' side, the download is skipped and the job should terminate in the
CHECKSUMS_NOT_MATCH
stage.
-
Method Summary
Modifier and TypeMethodDescriptionReturns the most recent downloadjobName
if it has been fully processed, andempty
if the download is still being processed.schedule()
Returns aDownloadSchedule
instance that describes the work to be performed by an invocation of the download action, if applicable; orOptional.empty()
when there is nothing to do.
-
Method Details
-
schedule
Returns aDownloadSchedule
instance that describes the work to be performed by an invocation of the download action, if applicable; orOptional.empty()
when there is nothing to do.For an interrupted job, work will resume from the
DownloadSchedule.stage
. -
fetchMostRecentDownloadJobIdIfCompleted
Returns the most recent downloadjobName
if it has been fully processed, andempty
if the download is still being processed.
-