feat: file transfers analytics rules based on threshold

[juju4]

Change(s):

• Add file transfer analytics rules

Reason for Change(s):

• More data exfiltration detection based on static threshold to adapt to context
• Adapted for OpenSSH SFTP, Sharepoint and Progress MoveIt logs

Version Updated:

• No (new rules, starting 1.0.0)

Testing Completed:

• Yes

Checked that the validations are passing and have addressed any issues that are present:

• Pending CI

Note: for moveit, I put in Windows Forwarded Event folder but not sure if best. In my case, logs are in Events table collected through AMA.

[v-rbajaj]

Hi @juju4, please shorten the description of all the analytical rules to 255 characters only.

[juju4]

description was inspired of Syslog/Analytic Rules/ssh_potentialBruteForce.yaml and many other yaml files seem to be above 255.

If too long, where do we put other information? any extended description or note field?
mostly the entity mapping for array limitation information.

Also how to add grouping (aggregationKind) on selected entities?
If not already, it would be nice if the yaml rule schema was documented somewhere public as it does not seem to be 1:1 mapping of the ARM template.

[v-rbajaj]

Hi @juju4, we acknowledge your concern, let me get back to you by 10 Oct 2023.

[v-rbajaj]

Hi @juju4, sorry for delay in response.
To answer your first question, the limit for 255 characters is only for Hunting queries and not for Analytical rules, so please ignore that.

Regarding second question, we will get back to you by 12 Oct 2023.

[v-rbajaj]

Also how to add grouping (aggregationKind) on selected entities? If not already, it would be nice if the yaml rule schema was documented somewhere public as it does not seem to be 1:1 mapping of the ARM template.

Hi @juju4, for this question, you can refer to this document - https://github.com/Azure/Azure-Sentinel/wiki/Query-Style-Guide

Apart from that, I could see there are some validations which are failing in the PR.
To fix them, please change the extensions of files Solutions/Windows Forwarded Events/Analytic Rules/moveit_file_transfer_above_threshold.yml, Solutions/Windows Forwarded Events/Analytic Rules/moveit_file_transfer_folders_above_threshold.yml to .yaml

[juju4]

file extension renaming done.

On eventGroupingSettings, this is not the grouping that I was looking for.
I was searching for setting to group repeated alerts based on selected entities, typically Hostname and Account.
Not grouping based on multiple KQL results.
Would this be exist in the yaml definition?

In ARM template, it would be groupingConfiguration as per https://learn.microsoft.com/en-us/azure/templates/microsoft.securityinsights/2022-01-01-preview/alertrules?pivots=deployment-language-bicep#groupingconfiguration with matchingMethod = Selected

[v-rbajaj]

file extension renaming done.

On eventGroupingSettings, this is not the grouping that I was looking for. I was searching for setting to group repeated alerts based on selected entities, typically Hostname and Account. Not grouping based on multiple KQL results. Would this be exist in the yaml definition?

In ARM template, it would be groupingConfiguration as per https://learn.microsoft.com/en-us/azure/templates/microsoft.securityinsights/2022-01-01-preview/alertrules?pivots=deployment-language-bicep#groupingconfiguration with matchingMethod = Selected

Hi @juju4, for groupingConfiguration, please refer Azure-Sentinel/Solutions/Microsoft 365 Defender/Analytic Rules/PossibleWebpBufferOverflow.yaml at 990bc461901c3f3792c2b26c9c35e01e407a8199 · Azure/Azure-Sentinel (github.com)

[v-atulyadav]

Hi @juju4, please check comments above from @v-rbajaj and act accordingly. Thanks

[juju4]

Tests all green now

[juju4]

Thanks!