file.q

File and Folder Discovery

This library provides functionality to find files and folders within directories on disk.

.file.listFolder / .file.listFolderPaths

These functions list the contents of the specified folder. .file.listFolderPaths returns files and folders with the supplied path prepended (so they can be used with get for example). .file.listFolder returns just file and folder names.

.file.find / .file.findFilePaths

These functions return files that match a specified regular expression. Again, .file.findFilePaths will return any match files or folders with the supplied path prepended, where as .file.find will just return their names.

The regex to find files can be specified either as a symbol, which is then interpreted as match anywhere, or a kdb regex string.

Examples

Using symbol as regex:

q) srcFolder:` sv .file.getCwd[],`src;
Running system command: "echo %cd%"

q) .file.findFilePaths[`log; srcFolder]
,`:C:\Users\jasra_000\git\kdb-common/src/log.q

Using kdb string regex:

q) .file.findFilePaths["*.q"; srcFolder]
`:C:\Users\jasra_000\git\kdb-common/src/convert.q`:C:\Users\jasra_000\git\kdb-common/src/cron.q`:C:\Users\j..

Using string for exact match:

q) .file.findFilePaths["util.q"; srcFolder]
,`:C:\Users\jasra_000\git\kdb-common/src/util.q

.file.tree

This function recursively descends from the specified root folder down all child folders until no more folders are found (it behaves similarly to the Linux command tree). All discovered files are returned with the specified path prefixed.

Please note that folders that are symbolic links will be followed so you must ensure that there are no circular references within the folder structure you wish to tree.

.file.ensureDir

This function checks if the specified folder exists on disk already. If it doesn't it creates the folder before returning. It uses the OS library to create the new folder

.file.loadDir

This is a simple wrapper around system "l" with the conversion of a folder path symbol into a string.

.file.getCwd

This function returns the current working directory of the kdb process. It uses the OS library to get the raw result from the OS and then ensures it's in a file path symbol format.

Example

OS library working directory function:

q) .os.run[`pwd;::]
Running system command: "echo %cd%"
"C:\\Users\\jasra_000\\git\\kdb-common "

File library .file.getCwd

q) .file.getCwd[]
Running system command: "echo %cd%"
`:C:\Users\jasra_000\git\kdb-common

.file.isCompressed

This function returns true if the specified file has been compressed by kdb+ native compression.

.file.replace[source; target]

Replaces the target specified file or folder with the specified source. For files this is equivalent to a move, but for folders, it will delete the target folder before moving.

Example

q) .file.ls `:/tmp/hdb
`s#`2021.01.23`2021.01.24`2021.10.28`2021.10.29`2021.10.31`sym

q) .file.replace[`:/tmp/hdb/2021.10.31; `:/tmp/hdb/2021.10.29]
2021.11.02 13:12:35.234 DEBUG pid-516 jas 0 [./src/util.q:.util.system(20):1] Running system command: "rm -rvf /tmp/hdb/2021.10.29"
2021.11.02 13:12:35.249 DEBUG pid-516 jas 0 [./src/util.q:.util.system(20):1] Running system command: "mv /tmp/hdb/2021.10.31 /tmp/hdb/2021.10.29"

q) .file.ls `:/tmp/hdb
`s#`2021.01.23`2021.01.24`2021.10.28`2021.10.29`sym

.file.kdb.getType / .file.kdb.getLength / .file.kdb.getAttribute

Optimised functions for on-disk kdb files. Instead of having to read the full file, only the first 4096 bytes (or the compression map and first chunk if a compressed file, transparently handled by read1) need to be read from disk.

This results in a ~99% performance improvement with large files.

Optimised Function	Equivalent To
.file.kdb.getType	type get x
.file.kdb.getLength	count get x
.file.kdb.getAttributes	attr get x

NOTE: For splayed tables, func each .Q.V splay provides equivalent performance to this function.

.file.kdb.getLength works for almost all on-disk file types (including new-format lists). For unsupported files, the function will fallback to count get x.

See the original pull request for performance comparisons.

.file.kdb.getSummary

Provides a dictionary of the other .file.kdb.get* functions documented above in a single call.

q) set[`:/tmp/list; `s#til 1000]
`:/tmp/list

q) .file.kdb.getSummary `:/tmp/list
type     | 7h
attribute| `s
length   | 1000

.file.kdb.getSymEnumerationTarget

Provides an optimised equivalent of (key get x) to get the enumeration target of enumerated symbol lists

q)get `:/tmp/hdb/2023.02.15/tbl/sym
`sym_custom!0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 4..

q).file.kdb.getSymEnumerationTarget `:/tmp/hdb/2023.02.15/tbl/sym
`sym_custom

.file.hdb.qPar[hdbRoot; partVal]

Wrapper for .Q.par that converts any relative paths in par.txt to absolute paths and doesn't require specifying a table argument.

q) read0 `$":/tmp/hdb-par/par.txt"
"./P1"
"./P2"

q) .Q.par[`$":/tmp/hdb-par"; .z.d; `]
`:./P1/2021.11.02/

q) .file.hdb.qPar[`$":/tmp/hdb-par"; .z.d]
`:/tmp/hdb-par/./P1/2021.11.02/