cron.q

Job Scheduler

This library provides cron-like functionality within a specific kdb process.

Functions can be executed in one of the following modes:

• Execute once at a specific time
• Execute repeatedly every interval forever
• Execute repeatedly every interval until a specified time

The library uses the built in .z.ts timer functionality to work.

Two modes of operation are provides:

• 'Ticking' mode (default) - the timer is set to fire at a regular interval, checking for any jobs that should now be executed
• 'Tickless' mode - the timer is set to fire at the next interval when we need to run a job (NO_HZ kernel inspiration)

Adding Jobs

A "job" is a kdb function that should be run at the specified time. Arguments can be supplied to pass into the function at run time if required. If no arguments are required, generic null (::) should be passed as the arguments.

The functions to add a job are listed below. All return the cron job ID for reference.

• .cron.addRunOnceJob: Run the specified function once at the specified time
• .cron.addRepeatForeverJob: Run the specified function from the start time at every interval forever (unless explicitly removed from the cron system)
• .cron.addRepeatUntilTimeJob: Run the specified function from the start time at every interval until the specified end time

Notes:

1. Repeat intervals are specified as Timespans. The function .convert.msToTimespan can be useful if you want to define intervals in milliseconds in your code
2. All start and end times are rounded to milliseconds (as the maximum timer resolution is 1 ms) via .time.roundTimestampToMs
3. Start times that occur before 'now' will be rejected by default and an exception throw
   • See .cron.cfg.historicalStartTimes

Preventing Duplicate Scheduling of the Same Function and Arguments

As of kdb-common-1.5.0-2023.03.30 2 additional functions have been added:

• .cron.addUniqueRepeatForeverJob
• .cron.addUniqueRepeatUntilTimeJob

These functions protect the same function and argument combination being scheduled again when an active job for this combination already exists.

Example:

/ Add a repeating job for '.jas.run[]'
q) .cron.addUniqueRepeatUntilTimeJob[`.jas.run; ::; 2023.03.30+21:33; 2023.03.30+21:34; 0D00:00:10]
2

/ While job is active, cannot schedule it again
q).cron.addUniqueRepeatUntilTimeJob[`.jas.run; ::; 2023.03.30+20:34; 2023.03.30+20:35; 0D00:00:10]
2023.03.30 20:33:26.103 INFO  pid-593 jas 0 [/home/jas/git/kdb-common/src/cron.q:.cron.addUniqueRepeatUntilTimeJob(231):4] Cron job with matching function and arguments is active. Not adding job [ Function: .jas.run ] [ Arguments: :: ]
2

/ Once job has finished, it can be scheduled again
2023.03.30 20:35:00.008 INFO  pid-593 jas 0 [/home/jas/git/kdb-common/src/cron.q:.cron.i.runRepeat(289):13] Job has reached 'end time'. Will not schedule again [ Job: 3 ]

q).cron.addUniqueRepeatUntilTimeJob[`.jas.run; ::; 2023.03.30+20:35:10; 2023.03.30+20:36; 0D00:00:10]
4

Checking Defined Jobs

All the jobs added to the cron system are stored within .cron.jobs. By default when the cron system is initialised, a default status clear job is added to run once per day.

The nextRunTime column informs you when the job will next run. If this field is set to 0Wp, the job will not run again.

Checking Job Run Status

After each job is executed, the result is stored within the .cron.status table. Repeat jobs will log each time it is executed.

Logging can be enabled and disabled by updating the .cron.cfg.logStatus boolean as required.

Any jobs that fail to execute will be logged to the console and the "success" column of .cron.status will be set to false. The backtrace of any failed job will be stored in the result column of .cron.status and can also be logged to the console via the .cron.cfg.printBacktraceOnFailure configuration.

Cron Error Example

q) .jas.error:{ 1+`a }
/ Run function forever from now every 10 seconds
q) .cron.addRepeatForeverJob[`.jas.error;::;.time.now[];.convert.msToTimespan 10*1000]
3
q) Cron job failed to execute [ Job ID: 3 ]. Error - type
Cron job failed to execute [ Job ID: 3 ]. Error - type

Also checking .cron.status:

q)select from .cron.status where id = 2
q).cron.status
id func       expectedStartTime             startTime                     runTime              success result                                      ..
---------------------------------------------------------------------------------------------------------------------------------------------------..
                                                                                               0       ::                                          ..
2  .jas.error 2021.05.27D18:23:27.531000000 2021.05.27D18:23:27.600339000 0D00:00:00.000090000 0       `errorMsg`backtrace!("type";"  [4]  .jas.err..
2  .jas.error 2021.05.27D18:23:37.531000000 2021.05.27D18:23:37.600348000 0D00:00:00.000093000 0       `errorMsg`backtrace!("type";"  [4]  .jas.err..

Cancelling Jobs

If you wish to cancel a job at any point, use .cron.cancelJob with the job ID. This will not remove the job from .cron.jobs but will update nextRunTime so the job never runs.

Configuration

Cron Modes

The current cron mode is defined in .cron.cfg.mode and be changed at any point with the .cron.changeMode option.

In 'tickless' mode, the interval is updated whenever:

• The timer fires
• A new job is added
• A job is cancelled

Cron Modes Examples

When in ticking mode, the timer interval is fixed, matching .cron.cfg.timerInterval:

q).cron.cfg.mode
`ticking
q)\t
100i

When in tickless mode, the timer interval is set to fire when the next function must be executed:

q).cron.cfg.mode
`tickless
q)\t
14999i
q)Ran @ 2021.05.27D18:19:52.436183000
2021.05.27 18:19:52.436 TRACE pid-483 jas 0 Tickless cron timer updated [ Next Run: 0D00:00:15.000000000 (15000 ms) ]

Historical Start Times

When a new cron job is added, if the behaviour of the library when the start time specified occurs before 'now' can be configured with .cron.cfg.historicalStartTimes.

There are 3 options:

• disallowed (default): If the specified start time is before 'now' a InvalidCronJobTimeException will be thrown
• allowed: Any specified start time is accepted
• setAsNow: Any time earlier than 'now' will have their time changed to be 'now'

Note that adding a job with a start time before 'now' and using either allowed or setAsNow will cause the job to execute immediately on the next main loop iteration.