[Fleet] Create API to report status of integrations synchronization #216178

criamico · 2025-03-27T13:37:06Z

Closes #192363

Summary

Add endpoint that compares integrations installed on remote cluster with integrations in ccr index fleet-synced-integrations-ccr-<outputId>. Feature flag: enableSyncIntegrationsOnRemote

Use the ccr info api to check that the ccr index is active
Compare the content of the two indices and report the sync status for each integration:

GET kbn:/api/fleet/remote_synced_integrations/status

{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": true
    },
    {
      "package_name": "auth0",
      "package_version": "1.21.0",
      "updated_at": "2025-03-26T12:06:26.268Z",
      "sync_status": false,
      "error": "Installation status: not_installed" 
    },
]

Check that the integrations have custom assets and report their sync status (follow up of [Fleet] install custom assets on remote cluster #215504)

Testing

Setup local env with the guide added in dev_docs (preview)

Install some integrations on local cluster, wait that they are synced on remote
From remote cluster dev tools, run

GET kbn:/api/fleet/remote_synced_integrations/status

To verify that custom assets are synced choose an integration, for instance system
From the package policy select a var, advanced options and add a custom mapping and a custom pipeline. In my example I used system

Run the endpoint again and you should see the status of custom assets too:

{
  "integrations": [
    {
      "package_name": "akamai",
      "package_version": "2.28.0",
      "updated_at": "2025-03-27T10:29:52.485Z",
      "sync_status": "completed"
    },
    {
      "package_name": "elastic_agent",
      "package_version": "2.2.0",
      "updated_at": "2025-03-26T14:06:29.216Z",
      "sync_status": "completed"
    },
    {
      "package_name": "synthetics",
      "package_version": "1.4.1",
      "updated_at": "2025-03-26T14:06:31.909Z",
      "sync_status": "completed"
    },
    {
      "package_name": "system",
      "package_version": "1.67.3",
      "updated_at": "2025-03-28T10:08:00.602Z",
      "sync_status": "completed"
    }
  ],
  "custom_assets": {
    "component_template:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "component_template",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    },
    "ingest_pipeline:logs-system.auth@custom": {
      "name": "logs-system.auth@custom",
      "type": "ingest_pipeline",
      "package_name": "system",
      "package_version": "1.67.3",
      "sync_status": "completed"
    }
  }
}

Checklist

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

criamico · 2025-03-27T13:37:31Z

@elasticmachine merge upstream

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts

juliaElastic · 2025-03-28T08:39:43Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts

+    if (!installedIntegrations) {
+      return {
+        items: [],
+        error: `No integrations installed on remote`


it's not necessarily an error if the main cluster doesn't have any integrations either

x-pack/platform/plugins/shared/fleet/server/routes/remote_synced_integrations/index.ts

x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md

x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/custom_assets.ts

criamico · 2025-04-01T09:22:36Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts

+
+  try {
+    const installedPipelines = await getPipeline(esClient, abortController);
+    const installedComponentTemplates = await getComponentTemplate(esClient, abortController);


I went for the solution of fetching all the pipelines and component templates and keep them on memory, instead of doing a call for each one in the loop below. @juliaElastic do you think it could become a performance issue?

We should only compare pipelines and component templates that match '*@custom', like it's done in custom_assets.ts. The other assets are already installed when the package is installed, no need to compare them.

juliaElastic · 2025-04-01T09:37:54Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts

+    Object.entries(ccrCustomAssets).forEach(([ccrCustomName, ccrCustomAsset]) => {
+      if (ccrCustomAsset.type === 'ingest_pipeline') {
+        const installedAsset = installedPipelines[ccrCustomAsset?.name];
+        if (isEqual(installedAsset?.processors, ccrCustomAsset?.pipeline?.processors)) {


pipelines have an optional version which we can use to compare like here

kibana/x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/custom_assets.ts

Line 241 in b438a71

(existingPipeline.version && existingPipeline.version < customAsset.pipeline.version) ||

we should probably have a common logic to compare here and custom_assets.ts

criamico · 2025-04-01T15:33:28Z

@elasticmachine merge upstream

elasticmachine · 2025-04-01T16:05:36Z

Pinging @elastic/fleet (Team:Fleet)

...form/plugins/shared/fleet/dev_docs/diagrams/remote_clusters/remote_clusters_architecture.png

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --update'

criamico · 2025-04-03T20:03:27Z

@juliaElastic based on the new requirements in #217025 we should be able to query by output_id:

Create API that queries remote kibana sync status API by output ID (to be used by the UI to show status)
e.g. /api/fleet/remote_synced_integrations/{output_id}/status -> https://{remote_kibana}/api/fleet/remote_synced_integrations/status

In a previous commit it was already by output_id. Do you think we'll need to keep the general status? Otherwise I'll change it directly in this PR.

criamico · 2025-04-04T06:26:40Z

@elasticmachine merge upstream

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --update'

juliaElastic · 2025-04-04T07:15:10Z

@juliaElastic based on the new requirements in #217025 we should be able to query by output_id:

Create API that queries remote kibana sync status API by output ID (to be used by the UI to show status)
e.g. /api/fleet/remote_synced_integrations/{output_id}/status -> https://{remote_kibana}/api/fleet/remote_synced_integrations/status

In a previous commit it was already by output_id. Do you think we'll need to keep the general status? Otherwise I'll change it directly in this PR.

We can keep as is in the current pr, as it collects the status in the remote cluster. The new API by output_id will only call the remote API using the remote output kibana url and API key.

criamico · 2025-04-04T12:48:42Z

@elasticmachine merge upstream

juliaElastic · 2025-04-07T07:23:39Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

+    return { info: res.follower_indices[0] };
+  } catch (err) {
+    if (err?.body?.error?.type === 'index_not_found_exception')
+      throw new IndexNotFoundError(`Index not found`);


should we return the error message instead of throwing an error?

It's handled here: https://github.com/elastic/kibana/pull/216178/files#diff-f8de6a6d308d65b9d61c400a2fdbebe1078e67eaf036537c028476260be254f5R312-R315

I left the throw block and handled outside because we might need this function elsewhere, this is a basic utility for the ccr case.

okay, then shouldn't we return the error in the catch there? https://github.com/elastic/kibana/pull/216178/files#diff-f8de6a6d308d65b9d61c400a2fdbebe1078e67eaf036537c028476260be254f5R324

Fixed, I added some unit tests to cover these cases

juliaElastic · 2025-04-07T07:32:00Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

+      };
+    } else if (
+      installedPipeline?.version &&
+      installedPipeline.version < ccrCustomAsset.pipeline.version


the version comparison should be done before the equality check, so we can skip the equality check if the version is not equal

juliaElastic · 2025-04-07T07:44:35Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

+    if (ccrCustomAsset.is_deleted === true && installedPipeline) {
+      return {
+        ...result,
+        sync_status: 'failed' as SyncStatus.FAILED,


should this be synchronizing (the deletion might not have happened yet) unless we know there was an error deleting?

juliaElastic · 2025-04-07T07:45:23Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

+    if (ccrCustomAsset.is_deleted === true && installedCompTemplate) {
+      return {
+        ...result,
+        sync_status: 'failed' as SyncStatus.FAILED,


same here, should these be synchronizing unless we know there was an error deleting?

criamico · 2025-04-07T10:17:47Z

@elasticmachine merge upstream

juliaElastic · 2025-04-07T10:35:06Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

@@ -255,6 +250,11 @@ const compareCustomAssets = ({
        sync_status: 'failed' as SyncStatus.FAILED,


this should be synchronizing too

elasticmachine · 2025-04-07T11:55:22Z

💚 Build Succeeded

Buildkite Build
Commit: f884e10

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	1199	1201	+2

History

💚 Build #290276 succeeded 317ed18
💔 Build #290206 failed 2b4cb2e
💔 Build #290141 failed 4519bd6
💔 Build #290074 failed 0907204
💛 Build #289957 was flaky 8129bf4
💔 Build #289827 failed 3a0db47

cc @criamico

juliaElastic · 2025-04-07T13:05:19Z

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/compare_synced_integrations.ts

+        return {
+          ...ccrIntegration,
+          sync_status: 'failed' as SyncStatus.FAILED,
+          error: `Installation status: ${localIntegrationSO?.attributes.install_status}`,


can we return the error message if install_status: install_failed from latest_install_failed_attempts or latest_executed_state?

kibana/x-pack/platform/plugins/shared/fleet/common/types/models/epm.ts

Line 674 in 0c5fa84

latest_install_failed_attempts?: InstallFailedAttempt[];

juliaElastic

LGTM, thanks for the updates

.../platform/plugins/shared/fleet/dev_docs/diagrams/remote_clusters/remote_clusters_diagram.png

criamico added 4 commits March 19, 2025 09:59

[Fleet] Create API to report status of integrations synchronization

bdc0100

use follower info api

ce1de43

Merge main

fe5c7f1

Get working endpoint

d7fbe56

criamico added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 27, 2025

criamico self-assigned this Mar 27, 2025

elasticmachine and others added 2 commits March 27, 2025 15:37

Merge branch 'main' into 192363_integrations_sync_status

6e1f6b8

Add top level error

525d0d5

juliaElastic reviewed Mar 28, 2025

View reviewed changes

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts Outdated Show resolved Hide resolved

juliaElastic reviewed Mar 28, 2025

View reviewed changes

.../platform/plugins/shared/fleet/server/tasks/sync_integrations/sync_integrations_on_remote.ts Outdated Show resolved Hide resolved

juliaElastic reviewed Mar 28, 2025

View reviewed changes

x-pack/platform/plugins/shared/fleet/server/routes/remote_synced_integrations/index.ts Outdated Show resolved Hide resolved

criamico added 2 commits March 28, 2025 10:57

Address code review comments

84e10d6

Update dev docs

796b101

juliaElastic reviewed Mar 31, 2025

View reviewed changes

x-pack/platform/plugins/shared/fleet/dev_docs/local_setup/remote_clusters_ccr.md Outdated Show resolved Hide resolved

Add comparison for custom assets

b438a71

criamico commented Apr 1, 2025

View reviewed changes

x-pack/platform/plugins/shared/fleet/server/tasks/sync_integrations/custom_assets.ts Outdated Show resolved Hide resolved

criamico commented Apr 1, 2025

View reviewed changes

juliaElastic reviewed Apr 1, 2025

View reviewed changes

criamico changed the title ~~192363 integrations sync status~~ [Fleet] Create API to report status of integrations synchronization Apr 1, 2025

Address comments and add unit tests

f022ee1

Merge branch 'main' into 192363_integrations_sync_status

682a307

criamico added v9.1.0 release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Apr 1, 2025

criamico marked this pull request as ready for review April 1, 2025 16:05

criamico requested a review from a team as a code owner April 1, 2025 16:05

juliaElastic reviewed Apr 3, 2025

View reviewed changes

...form/plugins/shared/fleet/dev_docs/diagrams/remote_clusters/remote_clusters_architecture.png Outdated Show resolved Hide resolved

kibanamachine and others added 3 commits April 3, 2025 14:30

[CI] Auto-commit changed files from 'make api-docs'

8129bf4

Code review pt.2: check is_deleted, remove wrong condition

0907204

elasticmachine and others added 3 commits April 4, 2025 07:26

Merge branch 'main' into 192363_integrations_sync_status

6e10aa5

[CI] Auto-commit changed files from 'make api-docs'

4519bd6

criamico added 2 commits April 4, 2025 11:02

Fix unit test

c186486

Update docs and diagram

2b4cb2e

Merge branch 'main' into 192363_integrations_sync_status

317ed18

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Address code review comments pt.3, add more tests

fb1eb39

Merge branch 'main' into 192363_integrations_sync_status

f884e10

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Replace failed state with synchronizing when versions do not match

fb34535

juliaElastic reviewed Apr 7, 2025

View reviewed changes

Add failure reason from latest_install_failed_attempts

fc6a8c8

juliaElastic approved these changes Apr 7, 2025

View reviewed changes

juliaElastic reviewed Apr 7, 2025

View reviewed changes

.../platform/plugins/shared/fleet/dev_docs/diagrams/remote_clusters/remote_clusters_diagram.png Outdated Show resolved Hide resolved

Update diagram

e51b00c

criamico enabled auto-merge (squash) April 7, 2025 16:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Create API to report status of integrations synchronization #216178

[Fleet] Create API to report status of integrations synchronization #216178

criamico commented Mar 27, 2025 •

edited

Loading

criamico commented Mar 27, 2025

juliaElastic Mar 28, 2025

criamico Apr 1, 2025 •

edited

Loading

juliaElastic Apr 1, 2025 •

edited

Loading

juliaElastic Apr 1, 2025

criamico commented Apr 1, 2025

elasticmachine commented Apr 1, 2025

criamico commented Apr 3, 2025

criamico commented Apr 4, 2025

juliaElastic commented Apr 4, 2025

criamico commented Apr 4, 2025

juliaElastic Apr 7, 2025

criamico Apr 7, 2025

juliaElastic Apr 7, 2025

criamico Apr 7, 2025

juliaElastic Apr 7, 2025

juliaElastic Apr 7, 2025

juliaElastic Apr 7, 2025

criamico commented Apr 7, 2025

juliaElastic Apr 7, 2025

elasticmachine commented Apr 7, 2025

juliaElastic Apr 7, 2025

juliaElastic left a comment

		@@ -255,6 +250,11 @@ const compareCustomAssets = ({
		sync_status: 'failed' as SyncStatus.FAILED,

[Fleet] Create API to report status of integrations synchronization #216178

Are you sure you want to change the base?

[Fleet] Create API to report status of integrations synchronization #216178

Conversation

criamico commented Mar 27, 2025 • edited Loading

Summary

Testing

Checklist

criamico commented Mar 27, 2025

Choose a reason for hiding this comment

criamico Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

juliaElastic Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

criamico commented Apr 1, 2025

elasticmachine commented Apr 1, 2025

criamico commented Apr 3, 2025

criamico commented Apr 4, 2025

juliaElastic commented Apr 4, 2025

criamico commented Apr 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

criamico commented Apr 7, 2025

Choose a reason for hiding this comment

elasticmachine commented Apr 7, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

History

Choose a reason for hiding this comment

juliaElastic left a comment

Choose a reason for hiding this comment

criamico commented Mar 27, 2025 •

edited

Loading

criamico Apr 1, 2025 •

edited

Loading

juliaElastic Apr 1, 2025 •

edited

Loading