Mixing defaults, references inside a record

[eloualiche]

My goal is to find a way to create simple records with sane defaults.
I am using the json in a python script to generate a bunch of source/target pair to symlink.

An example of what I would like to do in nixel is to use a schema (Link here) such that:

link1 | Link = {
    path_type = 'file,  # this is default
    source = { file = "file_src_nondefault.csv", folder = "folder_src_nondefault" },
    target = { file = "file_tgt_nondefault.csv", folder = "folder_tgt_nondefault",}
    }

which would generate the following record

"link1": {
    "path_type": "file",
    "source": { "file": "file_src_nondefault.csv", "folder": "folder_src_nondefault" },
    "target": { "file": "file_tgt_nondefault.csv", "folder": "folder_tgt_nondefault" }
}

but also allow for by default giving me file in target which comes from source

link2 | Link = {
    path_type = 'file,  # this is default
    source = { file = "file_src_nondefault.csv", folder = "folder_src_nondefault" },
    target = { folder = "folder_tgt_nondefault",}
    }

and the record (note that the file in target is source.file)

"link1": {
    "path_type": "file",
    "source": { "file": "file_src_nondefault.csv", "folder": "folder_src_nondefault" },
    "target": { "file": "file_src_nondefault.csv", "folder": "folder_tgt_nondefault" }
}

I have created some contract -- note the function that tries to fet the source.file value inside of target as a default:

Linktype = {
  path_type | [| 'file, 'files, 'folder |] | doc "type of Link"
    | default = 'other,
  source | Dyn | default = { },  # I have some other contract to validate source somewhere
  target | Dyn | default
    = std.record.filter (fun _name x => _name == "file") source &
      { folder | default = "folder_tgt_default" }
  }

I find that when I create a record that has some elements in target it erases ALL of the defaults in target.
I do not fully understand how the merge is happening.

I am totally new to this. Sorry if the question is trivial, but I could not figure it out from the documentation.
My approach to the problem might also be wrong, so feel free to let me know if the Linktype schema is not appropriate.

[jneem]

I can explain the merge behavior that you're seeing: when you merge two records that have the same field, if one of the records has a higher merge priority on that field then it "wins", and if they have the same merge priority then the values of the fields are merged recursively. For example,

{
  subfield1 | default = { foo = 1 },
  subfield2 = { foo | default = 1 },
}
& {
  subfield1 = { bar = 1 },
  subfield2 = { bar = 1 },
}

outputs

{
  "subfield1": {
    "bar": 1
  },
  "subfield2": {
    "bar": 1,
    "foo": 1
  }
}

The "subfield1" values aren't merged recursively because the "default" one just loses outright.

My other feedback is that for fields that are themselves records, it's often better to define subcontracts using "|" than to set values using "=". There a manual section about this. So I might try something like

Linktype = {
  source,
  target
    | (
      (if std.record.has_field "file" source then { file | default = source.file } else {})
      & { folder | default = "folder_tgt_default" }
    )
}

The Dyn contracts aren't necessary, by the way. They should have no effect, but in fact here they seem to trigger a bug in nickel (I'll file an issue).

[eloualiche]

Ahhhh! Thank you so much. This makes a lot of sense.
Always obvious in hindsight ... less so a few hours ago.

Out of curiosity. How would you go about including the function inside a contract?

[jneem]

Can you elaborate on what you mean by "including the function inside the contract"? In this case, you want a contract that depends on source, so I think having a function that takes source and returns a contract is pretty reasonable.

A bit of unsolicited advice that you should feel free to ignore: rather than having contracts that depend on other fields, I think it might be nicer to model this using enum variants (aka. algebraic data types). Rather than having a path_type enum adjacent to some fields whose interpretation depends on the path_type, you could have something like

Link = [|
  'file {
    source | { file | String, folder | String },
    target | { file | String | default = source.file, folder | String
#                                             ^^^
# no need for conditional contracts, because in this variant the source always has a file
  }
  'folder { source | { folder | String } ... }
  'files { ... }
|]

[eloualiche]

Ignore my first point. I thought because of the warning I needed to figure out how to wrap it inside a custom contract.

On enum variants I wrote a small example following your lead (please do bring unsolicited advice now before I go down the wrong path!).
I think this is super helpful and I can see the value of doing so for readability.

let Link = [|
  'file {
    file | String | default = "file_default.txt",
    folder | String | default = "folder_default"
  },
  'folder {
    folder | String | default = "folder_default"
  }
|]
in
{
  file1 = 'file { file = "file_nondefault.txt", folder = "folder_nondefault" } | Link,
  file2 = 'file { folder = "folder_nondefault" } | Link,

  folder1 = ('folder { folder = "folder_nondefault" } | Link),
  folder2 = 'folder {} | Link,
  # folder3 = 'folder { file = "file_nondefault.txt" } | Link , # errors
}

Is this what you had in mind.
The one thing I notice is that exporting (json) I would need to add std.enum.to_tag_and_arg or some user defined function to export it?

[yannham]

The one thing I notice is that exporting (json) I would need to add std.enum.to_tag_and_arg or some user defined function to export it?

That's correct, currently enum variants are the only primitive Nickel data type that can't be exported as it is. It is more targeted at Nickel APIs and internal markers (in the sense, values that only live during the Nickel evaluation phase) but aren't really nice for exporting data to JSON. There are possible solutions to make the situation better, but they aren't currently implemented or expected to land in the near future. If the data you're manipulating are meant to be exported, then maybe your original approach was more adapted. Enum variants are cool but for a real-life configuration I fear it's going to be harder and harder to properly apply all the conversion functions everywhere.

For context, this blog post explains why and how we added ADTs to Nickel. Some parts in the middle are fairly technical, but I believe a good chunk of the first sections and some overview in the conclusion are pretty approachable.

[eloualiche]

Ok. this makes sense. I like that approach though.
Maybe I can do the transformation ex-post myself, once the data has been validated.

Thanks a lot for your help @yannham @jneem . It was a little hard to get started but you brought some of the barriers down!