Porting to other object models (JSON supersets, more or less)?

[barries]

Do folks here know of any projects using jaq as a library for more complex object models--JSON supersets more or less--either for matching by path expression or full filtering?

We're using Amazon Ion (specifically, ion_rs) for telemetry (threaded, scoped, structured binary logs) and a quick perusal of the repo makes it seem like jaq_json should be pretty easy to copy-paste-and-extend with additional datatypes like Timestamp, sexpr, etc.

I'm hoping to find examples and/or war stories of using jaq_core, etc., on a object models with more complex schemas... things that were easy, hard or surprising, example code to be inspired by, etc?

I'm writing a viewer for our telemetry files and would like to use something like jaq's core to allow the user to select entries and nodes; right now it only allows them to selecting lines of text-formatted output by scanning for plain string and regexs, and matches. What I'm picturing is sort of an interactive native jaq playground app that accepts log streams as input and allows interactive selection and, one day, transformations.

Thanks!

[01mf02]

Hi @barries! I may be of course a bit biased, but I believe that porting jaq to something like you describe it should be quite straightforward. You basically have to implement two traits on your Ion value datatype, namely jaq-{core,std}::ValT. What you describe is absolutely falling into the class of use cases I imagined when I made jaq abstract over different values, so I do not expect jaq-related surprises for your use case.

If you're doing this work for commercial purposes, then I could potentially also do this work for you as freelancer, possibly also including the playground part. (Given that I have already made the jaq playground.) If this interests you, you can contact me by mail or answer here. :)

[barries]

Hey, @01mf02! Yeah, the code made it look like you'd carefully designed it this way, and I'm glad to hear that it should be easy.

The telemetry (i.e. scoped/nested, structured binary logs) viewer is a side project I'm doing open source, otherwise I'd love to have my employer send some consulting $$$ your way. Unfortunately, sophisticated telemetry is not seen as a high enough priority to fund like that.

The "playground" analogy was about having the output update as you modify the search criteria--which could be jaq code. The app live-updates the results as you enter filter criteria, like fzf, etc. (see video). If I integrate jaq, the jaq part of it will be like its current regex mode and only update when the jaq code compiles correctly. One difference is that identifiers in jaq match complete names, whereas the string, and regexen without \> or `b`, perform partial matches, allowing the display to dynamically update as you type. Incremental identifier matching might be a fun improvement for later.

(I'm typing very slowly to let the viewer see the results--the app itself responds instantly for the size of logs we typically use)

view_logs_2025-03-14-156.mp4

Anyway, very, very cool stuff, and your paper about it is interesting!

[jobarr-amzn]

@barries that looks neat! I can attest that jaq is an excellent library and the Ion maintainers have been wondering the same thing for a while. We have an incomplete integration with jaq out for comment now for ion-cli, see amazon-ion/ion-cli#193.

Somewhere lost to obscurity (I hope!) is my attempt to build some jq functionality into the ion CLI before we knew about jaq. We're far better off with jaq's affordances for this kind of integration.

[01mf02]

The telemetry (i.e. scoped/nested, structured binary logs) viewer is a side project I'm doing open source, otherwise I'd love to have my employer send some consulting $$$ your way. Unfortunately, sophisticated telemetry is not seen as a high enough priority to fund like that.

I see, no worries. :)

The "playground" analogy was about having the output update as you modify the search criteria--which could be jaq code. The app live-updates the results as you enter filter criteria, like fzf, etc. (see video). If I integrate jaq, the jaq part of it will be like its current regex mode and only update when the jaq code compiles correctly. One difference is that identifiers in jaq match complete names, whereas the string, and regexen without \> or b, perform partial matches, allowing the display to dynamically update as you type. Incremental identifier matching might be a fun improvement for later.

(I'm typing very slowly to let the viewer see the results--the app itself responds instantly for the size of logs we typically use)
view_logs_2025-03-14-156.mp4

This definitely reminds me of jnv, which also uses jaq under the hood.
Watch out with live-updates, though: You can easily write jq filters that do not terminate, and therefore may eat your RAM quite quickly (because jaq is quite fast, it also exhausts your RAM quite fast, unfortunately ^^). For example, [recurse(.)]. For such cases, I suggest providing to your users a way to terminate the current jaq execution, like I did with the "Stop" button in the jaq playground (only visible when a program is running).

Anyway, very, very cool stuff, and your paper about it is interesting!

Grazie. :)

[01mf02]

@barries that looks neat! I can attest that jaq is an excellent library and the Ion maintainers have been wondering the same thing for a while. We have an incomplete integration with jaq out for comment now for ion-cli, see amazon-ion/ion-cli#193.

Somewhere lost to obscurity (I hope!) is my attempt to build some jq functionality into the ion CLI before we knew about jaq. We're far better off with jaq's affordances for this kind of integration.

Thank you for your kind words! I find it very cool what people are achieving with my little library. :)