Skip to content

Latest commit

 

History

History
668 lines (506 loc) · 28.2 KB

README.md

File metadata and controls

668 lines (506 loc) · 28.2 KB

scm-engine

SCM Engine allow for easy Merge Request automation within your GitLab projects.

Automatically add / remove labels depending on files changes, age of the Merge Request, who contributes, and pretty much anything else you could want, thanks to the usage of expr-lang.

SCM engine can be run either as a regular CI job in your pipeline, or be triggered through the Webhook system, allowing for versatile and flexible deployments.

Installation

Docker

docker run --rm ghcr.io/jippi/scm-engine

homebrew tap

brew install jippi/tap/scm-engine

apt

echo 'deb [trusted=yes] https://pkg.jippi.dev/apt/ * *' | sudo tee /etc/apt/sources.list.d/scm-engine.list
sudo apt update
sudo apt install scm-engine

yum

echo '[scm-engine]
name=scm-engine
baseurl=https://pkg.jippi.dev/yum/
enabled=1
gpgcheck=0' | sudo tee /etc/yum.repos.d/scm-engine.repo
sudo yum install scm-engine

snapcraft

sudo snap install scm-engine

scoop

scoop bucket add scm-engine https://github.com/jippi/scoop-bucket.git
scoop install scm-engine

aur

yay -S scm-engine-bin

deb, rpm and apk packages

Download the .deb, .rpm or .apk packages from the releases page and install them with the appropriate tools.

go install

go install github.com/jippi/scm-engine/cmd@latest

Usage

GitLab-CI pipeline

Using scm-engine within a GitLab CI pipeline is straight forward.

  1. Add a .scm-engine.yml file in the root of your project.

  2. Create a CI/CD Variable

    1. Name must be SCM_ENGINE_TOKEN
    2. Value must a Project Access Token
      1. Must have api scope.
      2. Must have developer or maintainer role access so it can edit Merge Requests.
    3. Mask should be checked.
    4. Protected should NOT be checked.
    5. Expand variable reference should NOT be checked.
  3. Setup a CI job using the scm-engine Docker image that will run when a pipeline is created from a Merge Request Event.

    scm-engine::evaluate::on-merge-request-event:
      image: ghcr.io/jippi/scm-engine:latest
      rules:
        - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      script:
        - scm-engine evaluate
    
    scm-engine::evaluate::on-schedule:
      image: ghcr.io/jippi/scm-engine:latest
      rules:
        - if: $CI_PIPELINE_SOURCE == "schedule"
      script:
        - scm-engine evaluate all
  4. Done! Every Merge Request change should now re-run scm-engine and apply your label rules

Commands

evaluate

Evaluate the SCM engine rules against a specific Merge Request.

NAME:
   scm-engine evaluate - Evaluate a Merge Request

USAGE:
   scm-engine evaluate [command options] [id, id, ...]

OPTIONS:
   --project value                                                GitLab project (example: 'gitlab-org/gitlab') [$GITLAB_PROJECT, $CI_PROJECT_PATH]
   --id value, --merge-request-id value, --pull-request-id value  The pull/merge to process, if not provided as a CLI flag [$CI_MERGE_REQUEST_IID]
   --help, -h                                                     show help

GLOBAL OPTIONS:
   --config value     Path to the scm-engine config file (default: ".scm-engine.yml") [$SCM_ENGINE_CONFIG_FILE]
   --api-token value  GitHub/GitLab API token [$SCM_ENGINE_TOKEN]
   --base-url value   Base URL for the SCM instance (default: "https://gitlab.com/") [$GITLAB_BASEURL, $CI_SERVER_URL]
   --dry-run          Dry run, don't actually _do_ actions, just print them (default: false)
   --help, -h         show help
   --version, -v      print the version

server

Point your GitLab webhook at the /gitlab endpoint.

Support the following events, and they will both trigger an Merge Request evaluation

  • Comments - A comment is made or edited on an issue or merge request.
  • Merge request events - A merge request is created, updated, or merged.

Tip

You have access to the raw webhook event payload via webhook_event.* fields in Expr script fields when using server mode. See the GitLab Webhook Events documentation for available fields.

NAME:
   scm-engine server - Start HTTP server for webhook event driven usage

USAGE:
   scm-engine server [command options]

OPTIONS:
   --webhook-secret value  Used to validate received payloads. Sent with the request in the X-Gitlab-Token HTTP header [$SCM_ENGINE_WEBHOOK_SECRET]
   --listen value          Port the HTTP server should listen on (default: "0.0.0.0:3000") [$SCM_ENGINE_LISTEN]
   --help, -h              show help

GLOBAL OPTIONS:
   --config value     Path to the scm-engine config file (default: ".scm-engine.yml") [$SCM_ENGINE_CONFIG_FILE]
   --api-token value  GitHub/GitLab API token [$SCM_ENGINE_TOKEN]
   --base-url value   Base URL for the SCM instance (default: "https://gitlab.com/") [$GITLAB_BASEURL, $CI_SERVER_URL]
   --dry-run          Dry run, don't actually _do_ actions, just print them (default: false)
   --help, -h         show help
   --version, -v      print the version

Configuration file

The default configuration filename is .scm-engine.yml, either in current working directory, or if you are in a Git repository, the root of the project.

The file path can be changed via --config CLI flag and $SCM_ENGINE_CONFIG_FILE environment variable.

Examples

Note

A quick demo of what SCM Engine can do. More details documentation further down the document.

The script field is a expr-lang expression, a safe, fast, and intuitive expression evaluator.

actions:
  - name: Warn if the Merge Request haven't had commit activity for 21 days and will be closed
    if: |
      merge_request.state != "closed"
      && merge_request.time_since_last_commit > duration("21d")
      && merge_request.time_since_last_commit < duration("28d")
      && not merge_request.has_label("do-not-close")
    then:
      - action: comment
        message: |
          :wave: Hello!

          This Merge Request has not seen any commit activity for 21 days.
          We will automatically close the Merge request after 28 days to keep our project clean.

          To disable this behavior, add the `do-not-close` label to the Merge Request in the right menu or add a comment with `/label ~"do-not-close"`.

  - name: Close the Merge Request if it haven't had commit activity for 28 days
    if: |
      merge_request.state != "closed"
      && merge_request.time_since_last_commit > duration("28d")
      && not merge_request.has_label("do-not-close")
    then:
      - action: close
      - action: comment
        message: |
          :wave: Hello!

          This Merge Request has not seen any commit activity for 28 days.
          To keep our project clean, we will close the Merge request now.

          To disable this behavior, add the `do-not-close` label to the Merge Request in the right menu or add a comment with `/label ~"do-not-close"`.

label:
    # Add a label named "lang/go"
  - name: lang/go
    # using the "conditional" strategy
    strategy: conditional
    # and a description (optional)
    description: "Modified Go files"
    # and the color $indigo
    color: "$indigo"
    # if files matching "*.go" was modified
    script: merge_request.modified_files("*.go")

    # Generate list of labels via script
  - strategy: generate
    # With a description (optional)
    description: "Modified this service directory"
    # With the color $pink
    color: "$pink"
    # From this script, returning a list of labels
    script: >
      // Generate a list of all file paths that was changed in the Merge Request inside pkg/service/
      merge_request.modified_files_list("pkg/service/")

      // Remove the filename from the path "pkg/service/example/file.go" => "pkg/service/example"
      | map({ filepath_dir(#) })

      // Remove the prefix "pkg/" from the path "pkg/service/example" => "service/example"
      | map({ trimPrefix(#, "pkg/") })

      // Remove duplicate values from the output
      | uniq()

actions[] (list)

The actions key is a list of actions that can be taken on a Merge Request.

actions[].name

The name of the action, this is purely for debugging and your convenience. It's encouraged to be descriptive of the actions.

actions[].if

A key controlling if the action should executed or not.

The if field must be a valid Expr-lang expression returning a boolean.

actions[].then[] (list)

The list of operations to take if the action.if returned true.

actions[].then[].action

This key controls what kind of action that should be taken.

  • close to close the Merge Request.
  • reopen to reopen the Merge Request.
  • lock_discussion to prevent further discussions on the Merge Request.
  • unlock_discussion to allow discussions on the Merge Request.
  • approve to approve the Merge Request.
  • unapprove to approve the Merge Request.
  • comment to add a comment to the Merge Request (requires the message field)

actions[].then[].message

Required field for action: comment.

The message that will be commented on the Merge Request.

label[] (list)

The label key is a list of the labels you want to manage.

These keys are shared between the conditional and generate label strategy. (more above these below!)

label[].name

  • When using label.strategy: conditional

    REQUIRED The name of the label to create.

  • When using label.strategy: generate

    OMITTED The name field must not be set when using the generate strategy.

label[].script (required)

Tip

See the SCM engine expr-lang documentation for more information about functions and attributes available.

The script field is an expr-lang expression, a safe, fast, and intuitive expression evaluator.

Depending on the label.strategy used, the behavior of the script changes, read more about this below.

label[].strategy (optional)

SCM Engine supports two strategies for managing labels, each changes the behavior of the script.

  • conditional (default, if type key is omitted), where you provide the name of the label, and a script that returns a boolean for wether the label should be added to the Merge Request.

    The script must return a boolean value, where true mean add the label and false mean remove the label.

  • generate, where your script generates the list of labels that should be added to the Merge Request.

    The script must return a list of strings, where each label returned will be added to the Merge Request.

label[].strategy: conditional use-cases

Use the conditional strategy when you want to add/remove a label on a Merge Request depending on something. It's the default strategy, and the most simple one to use.

label[].strategy: conditional examples

Note

The script field is a expr-lang expression, a safe, fast, and intuitive expression evaluator.

label:
    # Add a "lang/go" label if any "*.go" files was changed
  - name: lang/go
    color: "$indigo"
    script: merge_request.modified_files("*.go")

    # Add a "lang/markdown" label if any "*.md" files was changed
  - name: lang/markdown
    color: "$indigo"
    script: merge_request.modified_files("*.md")

    # Add a "type/documentation" label if any files was changed within the "docs/" folder
  - name: type/documentation
    color: "$green"
    script: merge_request.modified_files("docs/")

    # Add a "go::tests" scoped & prioritized label with value "missing" if no "*_test.go" files was changed
  - name: go::tests::missing
    color: "$red"
    priority: 999
    script: not merge_request.modified_files("*_test.go")

    # Add a "go::tests" scoped & prioritized label with value "OK" if any "*_test.go" files was changed
  - name: go::tests::ok
    color: "$green"
    priority: 999
    script: merge_request.modified_files("*_test.go")
label[].strategy: generate use-cases

Use the generate strategy if you want to manage dynamic labels, for example, depending on the file structure within your project.

label[].strategy: generate examples

The script field is a expr-lang expression, a safe, fast, and intuitive expression evaluator.

Thanks to the dynamic nature of the generate strategy, it has fantastic flexibility, at the cost of greater flexibility.

label:
    # Generate list of labels via script.
    #
    # Image you have a project where you have multiple "service" directories
    #
    # * pkg/service/example/file.go
    # * pkg/service/scm/gitlab/file.go
    # * pkg/service/scm/github/file.go
    #
    # and you want to generate a labels like this
    #
    # * service/example
    # * service/scm/gitlab
    # * service/scm/github
    #
    # depending on what directories are having files changed in a Merge Request.
  - strategy: generate
    description: "Modified this service directory"
    color: "$pink"
    script: >
      // Generate a list of all file paths that was changed in the Merge Request inside pkg/service/
      merge_request.modified_files_list("pkg/service/")

      // Remove the filename from the path "pkg/service/example/file.go" => "pkg/service/example"
      | map({ filepath_dir(#) })

      // Remove the prefix "pkg/" from the path "pkg/service/example" => "service/example"
      | map({ trimPrefix(#, "pkg/") })

      // Remove duplicate values from the output
      | uniq()

label[].color (required)

Note

When used on strategy: generate labels, all generated labels will have the same color.

color is a mandatory field, controlling the background color of the label when viewed in the User Interface.

You can either provide your own #hex value or use the Twitter Bootstrap color variables, for example $blue-500 and $teal.

label[].description (optional)

Note

When used on strategy: generate labels, all generated labels will have the same description.

An optional key that control the description field for the label within GitLab.

Descriptions are shown in the User Interface when you hover any label.

label[].priority (optional)

Note

When used on strategy: generate labels, all generated labels will have the same priority.

An optional key that controls the label priority.

label[].skip_if (optional)

An optional key controlling if the label should be skipped (meaning no removal or adding of labels).

The skip_if field must be a valid Expr-lang expression returning a boolean, where true means skip and false means process.

Script (Expr lang) information

Tip

The Expr Language Definition is a great resource to learn more about the language. This guide will only cover SCM Engine specific extensions and information.

Attributes

Note

Missing an attribute? The schema/gitlab.schema.graphqls file are what is used to query GitLab, adding the missing field to the right type should make it accessible. Please open an issue or Pull Request if something is missing.

Important

SCM Engine uses snake_case for fields instead of camelCase

Tip

You have access to the raw webhook event payload via webhook_event.* attributes (not listed below) in Expr script fields when using server mode. See the GitLab Webhook Events documentation for available fields. The attributes are named exactly as documented in the GitLab documentation.

The following attributes are available in script fields.

They can be accessed exactly as shown in this list.

  • group.description (string) Description of the namespace
  • group.emails_disabled (optional bool) Indicates if a group has email notifications disabled
  • group.full_name (string) Full name of the namespace
  • group.full_path (string) Full path of the namespace
  • group.id (string) ID of the namespace
  • group.mentions_disabled (optional bool) Indicates if a group is disabled from getting mentioned
  • group.name (string) Name of the namespace
  • group.path (string) Path of the namespace
  • group.visibility (optional string) Visibility of the namespace
  • group.web_url (string) Web URL of the group
  • merge_request.approvals_left (optional int) Number of approvals left
  • merge_request.approvals_required (optional int) Number of approvals required
  • merge_request.approved (bool) Indicates if the merge request has all the required approvals
  • merge_request.auto_merge_enabled (bool) Indicates if auto merge is enabled for the merge request
  • merge_request.auto_merge_strategy (optional string) Selected auto merge strategy
  • merge_request.commit_count (optional int) Number of commits in the merge request
  • merge_request.conflicts (bool) Indicates if the merge request has conflicts
  • merge_request.created_at (time) Timestamp of when the merge request was created
  • merge_request.description (optional string) Description of the merge request (Markdown rendered as HTML for caching)
  • merge_request.diff_stats[].additions (int) Number of lines added to this file
  • merge_request.diff_stats[].deletions (int) Number of lines deleted from this file
  • merge_request.diff_stats[].path (string) File path, relative to repository root
  • merge_request.discussion_locked (bool) Indicates if comments on the merge request are locked to members only
  • merge_request.diverged_from_target_branch (bool) Indicates if the source branch is behind the target branch
  • merge_request.downvotes (int) Number of downvotes for the merge request
  • merge_request.draft (bool) Indicates if the merge request is a draft
  • merge_request.first_commit.author_email (optional string) Commit author’s email
  • merge_request.first_commit.author_name (optional string) Commit authors name
  • merge_request.first_commit.authored_date (optional time) Timestamp of when the commit was authored
  • merge_request.first_commit.committed_date (optional time) Timestamp of when the commit was committed
  • merge_request.first_commit.committer_email (optional string) Email of the committer
  • merge_request.first_commit.committer_name (optional string) Name of the committer
  • merge_request.first_commit.description (optional string) Description of the commit message
  • merge_request.first_commit.full_title (optional string) Full title of the commit message
  • merge_request.first_commit.id (optional string) ID (global ID) of the commit
  • merge_request.first_commit.message (optional string) Raw commit message
  • merge_request.first_commit.sha (string) SHA1 ID of the commit
  • merge_request.first_commit.short_id (string) Short SHA1 ID of the commit
  • merge_request.first_commit.title (optional string) Title of the commit message
  • merge_request.first_commit.web_url (string) Web URL of the commit
  • merge_request.force_remove_source_branch (optional bool) Indicates if the project settings will lead to source branch deletion after merge
  • merge_request.id (string) ID of the merge request
  • merge_request.iid (string) Internal ID of the merge request
  • merge_request.labels[].color (string) Background color of the label
  • merge_request.labels[].description (string) Description of the label (Markdown rendered as HTML for caching)
  • merge_request.labels[].id (string) Label ID
  • merge_request.labels[].title (string) Content of the label
  • merge_request.last_commit.author_email (optional string) Commit author’s email
  • merge_request.last_commit.author_name (optional string) Commit authors name
  • merge_request.last_commit.authored_date (optional time) Timestamp of when the commit was authored
  • merge_request.last_commit.committed_date (optional time) Timestamp of when the commit was committed
  • merge_request.last_commit.committer_email (optional string) Email of the committer
  • merge_request.last_commit.committer_name (optional string) Name of the committer
  • merge_request.last_commit.description (optional string) Description of the commit message
  • merge_request.last_commit.full_title (optional string) Full title of the commit message
  • merge_request.last_commit.id (optional string) ID (global ID) of the commit
  • merge_request.last_commit.message (optional string) Raw commit message
  • merge_request.last_commit.sha (string) SHA1 ID of the commit
  • merge_request.last_commit.short_id (string) Short SHA1 ID of the commit
  • merge_request.last_commit.title (optional string) Title of the commit message
  • merge_request.last_commit.web_url (string) Web URL of the commit
  • merge_request.merge_status_enum (string) Merge status of the merge request
  • merge_request.merge_when_pipeline_succeeds (optional bool) Indicates if the merge has been set to auto-merge
  • merge_request.mergeable (bool) Indicates if the merge request is mergeable
  • merge_request.mergeable_discussions_state (optional bool) Indicates if all discussions in the merge request have been resolved, allowing the merge request to be merged
  • merge_request.merged_at (optional time) Timestamp of when the merge request was merged, null if not merged
  • merge_request.prepared_at (optional time) Timestamp of when the merge request was prepared
  • merge_request.should_be_rebased (bool) Indicates if the merge request will be rebased
  • merge_request.should_remove_source_branch (optional bool) Indicates if the source branch of the merge request will be deleted after merge
  • merge_request.source_branch (string) Source branch of the merge request
  • merge_request.source_branch_exists (bool) Indicates if the source branch of the merge request exists
  • merge_request.source_branch_protected (bool) Indicates if the source branch is protected
  • merge_request.squash (bool) Indicates if the merge request is set to be squashed when merged. Project settings may override this value. Use squash_on_merge instead to take project squash options into account
  • merge_request.squash_on_merge (bool) Indicates if the merge request will be squashed when merged
  • merge_request.state (string) State of the merge request
  • merge_request.target_branch (string) Target branch of the merge request
  • merge_request.target_branch_exists (bool) Indicates if the target branch of the merge request exists
  • merge_request.time_between_first_and_last_commit (optional duration)
  • merge_request.time_since_first_commit (optional duration)
  • merge_request.time_since_last_commit (optional duration)
  • merge_request.title (string) Title of the merge request
  • merge_request.updated_at (time) Timestamp of when the merge request was last updated
  • merge_request.upvotes (int) Number of upvotes for the merge request.
  • merge_request.user_discussions_count (optional int) Number of user discussions in the merge request
  • merge_request.user_notes_count (optional int) User notes count of the merge request
  • project.archived (bool) Indicates the archived status of the project
  • project.created_at (time) Timestamp of the project creation
  • project.description (string) Short description of the project
  • project.full_path (string) Full path of the project
  • project.id (string) ID of the project
  • project.issues_enabled (bool) Indicates if Issues are enabled for the current user
  • project.labels[].color (string) Background color of the label
  • project.labels[].description (string) Description of the label (Markdown rendered as HTML for caching)
  • project.labels[].id (string) Label ID
  • project.labels[].title (string) Content of the label
  • project.last_activity_at (time) Timestamp of the project last activity
  • project.name (string) Name of the project (without namespace)
  • project.name_with_namespace (string) Full name of the project with its namespace
  • project.path (string) Path of the project
  • project.topics ([]string) List of project topics
  • project.visibility (string) Visibility of the project

Functions

merge_request.modified_files

Returns wether any of the provided files patterns have been modified in the Merge Request.

The file patterns use the .gitignore format.

merge_request.modified_files("*.go", "docs/") == true

merge_request.modified_files_list

Returns an array of files matching the provided (optional) pattern thas has been modified in the Merge Request.

The file patterns use the .gitignore format.

merge_request.modified_files_list("*.go", "docs/") == ["example/file.go", "docs/index.md"]

merge_request.has_label

Returns wether any of the provided label exist on the Merge Request.

merge_request.has_label("my-label-name")

duration

Returns the time.Duration value of the given string str.

Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h", "d" and "w".

duration("1h").Seconds() == 3600

uniq

Returns a new array where all duplicate values has been removed.

(["hello", "world", "world"] | uniq) == ["hello", "world"]

filepath_dir

filepath_dir returns all but the last element of path, typically the path's directory. After dropping the final element,

Dir calls Clean on the path and trailing slashes are removed.

If the path is empty, filepath_dir returns ".". If the path consists entirely of separators, filepath_dir returns a single separator.

The returned path does not end in a separator unless it is the root directory.

filepath_dir("example/directory/file.go") == "example/directory"

limit_path_depth_to

limit_path_depth_to takes a path structure, and limits it to the configured maximum depth. Particularly useful when using generated labels from a directory structure, and want to to have a label naming scheme that only uses path of the path.

limit_path_depth_to("path1/path2/path3/path4", 2), == "path1/path2"
limit_path_depth_to("path1/path2", 3), == "path1/path2"