Skip to content

Commit 60b02ba

Browse files
fivetran-catfritzfivetran-catfritz
authored
Feature: Union schema compatibility (#14)
* MagicBot/add-union-schema updates * revisions to magicbot pr * update identifiers * update casts * update config enabled * update changelog & readme * update changelog and add auto releaser * update test * regen docs
1 parent 7c0b3c3 commit 60b02ba

File tree

46 files changed

+621
-253
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+621
-253
lines changed

‎.github/workflows/auto-release.yml

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: 'auto release'
2+
on:
3+
pull_request:
4+
types:
5+
- closed
6+
branches:
7+
- main
8+
9+
jobs:
10+
call-workflow-passing-data:
11+
if: github.event.pull_request.merged
12+
uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
13+
secrets: inherit

‎CHANGELOG.md

+41
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,44 @@
1+
# dbt_google_play_source v0.4.0
2+
[PR #14](https://github.com/fivetran/dbt_google_play_source/pull/14) includes the following updates:
3+
4+
## 🚨 Breaking Changes 🚨
5+
- Updated the source identifier format for consistency with other packages and for compatibility with the `fivetran_utils.union_data` macro. The identifier variables now are:
6+
7+
previous | current
8+
--------|---------
9+
`stats_installs_app_version_identifier` | `google_play_stats_installs_app_version_identifier`
10+
`stats_crashes_app_version_identifier` | `google_play_stats_crashes_app_version_identifier`
11+
`stats_ratings_app_version_identifier` | `google_play_stats_ratings_app_version_identifier`
12+
`stats_installs_device_identifier` | `google_play_stats_installs_device_identifier`
13+
`stats_ratings_device_identifier` | `google_play_stats_ratings_device_identifier`
14+
`stats_installs_os_version_identifier` | `google_play_stats_installs_os_version_identifier`
15+
`stats_ratings_os_version_identifier` | `google_play_stats_ratings_os_version_identifier`
16+
`stats_crashes_os_version_identifier` | `google_play_stats_crashes_os_version_identifier`
17+
`stats_installs_country_identifier` | `google_play_stats_installs_country_identifier`
18+
`stats_ratings_country_identifier` | `google_play_stats_ratings_country_identifier`
19+
`stats_store_performance_country_identifier` | `google_play_stats_store_performance_country_identifier`
20+
`stats_store_performance_traffic_source_identifier` | `google_play_stats_store_performance_traffic_source_identifier`
21+
`stats_installs_overview_identifier` | `google_play_stats_installs_overview_identifier`
22+
`stats_crashes_overview_identifier` | `google_play_stats_crashes_overview_identifier`
23+
`stats_ratings_overview_identifier` | `google_play_stats_ratings_overview_identifier`
24+
`earnings_identifier` | `google_play_earnings_identifier`
25+
`financial_stats_subscriptions_country_identifier` | `google_play_financial_stats_subscriptions_country_identifier`
26+
27+
- If you are using the previous identifier, be sure to update to the current version!
28+
29+
## Feature update 🎉
30+
- Unioning capability! This adds the ability to union source data from multiple google_play connectors. Refer to the [README](https://github.com/fivetran/dbt_google_play_source/blob/main/README.md#union-multiple-connectors) for more details.
31+
- Added a `source_relation` column in each staging model for tracking the source of each record.
32+
- Updated tests to account for the new `source_relation` column.
33+
- Added casting to staging columns that are used in any downstream COALESCEs, UNIONs, or JOINs to prevent datatype conflicts.
34+
- Metric fields, such as `crashes`, `*_installs`, `*_subscriptions`, etc., are now cast as the datatype determined by the `dbt.type_bigint()` macro for your warehouse.
35+
36+
## Under the hood 🚘
37+
- Updated tmp models to union source data using the `fivetran_utils.union_data` macro.
38+
- Applied the `fivetran_utils.source_relation` macro in each staging model to determine the `source_relation`.
39+
- Updated `src_apple_store.yml` to use the current method to enable/disable sources.
40+
- Included auto-releaser GitHub Actions workflow to automate future releases.
41+
142
# dbt_google_play_source v0.3.1
243
[PR #12](https://github.com/fivetran/dbt_google_play_source/pull/12) includes the following updates:
344
## 🪲 Bug fixes:

‎README.md

+16-4
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Include the following google_play_source package version in your `packages.yml`
3434
```yaml
3535
packages:
3636
- package: fivetran/google_play_source
37-
version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
37+
version: [">=0.4.0", "<0.5.0"] # we recommend using ranges to capture non-breaking changes automatically
3838
```
3939
## Step 3: Define database and schema variables
4040
By default, this package runs using your destination and the `google_play` schema. If this is not where your google_play data is (for example, if your google_play schema is named `google_play_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -60,8 +60,20 @@ In order to map longform territory names to their ISO country codes, we have ada
6060
You will need to `dbt seed` the `google_play__country_codes` [file](https://github.com/fivetran/dbt_google_play_source/blob/main/seeds/google_play__country_codes.csv) just once.
6161

6262
## (Optional) Step 6: Additional configurations
63-
<details><summary>Expand to view configurations</summary>
64-
63+
<details open><summary>Expand/collapse configurations</summary>
64+
65+
### Union multiple connectors
66+
If you have multiple google_play connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `google_play_union_schemas` OR `google_play_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
67+
68+
```yml
69+
vars:
70+
google_play_union_schemas: ['google_play_usa','google_play_canada'] # use this if the data is in different schemas/datasets of the same database/project
71+
google_play_union_databases: ['google_play_usa','google_play_canada'] # use this if the data is in different databases/projects but uses the same schema name
72+
```
73+
Please be aware that the native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
74+
75+
To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
76+
6577
### Change the build schema
6678
By default, this package builds the google_play staging models within a schema titled (`<target_schema>` + `_google_play_source`) in your destination. If this is not where you would like your google_play staging data to be written to, add the following configuration to your root `dbt_project.yml` file:
6779

@@ -77,7 +89,7 @@ If an individual source table has a different name than the package expects, add
7789

7890
```yml
7991
vars:
80-
<default_source_table_name>_identifier: your_table_name
92+
google_play_<default_source_table_name>_identifier: your_table_name
8193
```
8294

8395
</details>

‎dbt_project.yml

+12-12
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,26 @@
11
name: 'google_play_source'
2-
version: '0.3.1'
2+
version: '0.4.0'
33
config-version: 2
44
require-dbt-version: [">=1.3.0", "<2.0.0"]
55
vars:
66
google_play_source:
7-
stats_installs_app_version: "{{ source('google_play', 'stats_installs_app_version') }}"
7+
earnings: "{{ source('google_play', 'earnings') }}"
8+
financial_stats_subscriptions_country: "{{ source('google_play', 'financial_stats_subscriptions_country') }}"
89
stats_crashes_app_version: "{{ source('google_play', 'stats_crashes_app_version') }}"
9-
stats_ratings_app_version: "{{ source('google_play', 'stats_ratings_app_version') }}"
10-
stats_installs_device: "{{ source('google_play', 'stats_installs_device') }}"
11-
stats_ratings_device: "{{ source('google_play', 'stats_ratings_device') }}"
12-
stats_installs_os_version: "{{ source('google_play', 'stats_installs_os_version') }}"
13-
stats_ratings_os_version: "{{ source('google_play', 'stats_ratings_os_version') }}"
1410
stats_crashes_os_version: "{{ source('google_play', 'stats_crashes_os_version') }}"
11+
stats_crashes_overview: "{{ source('google_play', 'stats_crashes_overview') }}"
12+
stats_installs_app_version: "{{ source('google_play', 'stats_installs_app_version') }}"
1513
stats_installs_country: "{{ source('google_play', 'stats_installs_country') }}"
14+
stats_installs_device: "{{ source('google_play', 'stats_installs_device') }}"
15+
stats_installs_os_version: "{{ source('google_play', 'stats_installs_os_version') }}"
16+
stats_installs_overview: "{{ source('google_play', 'stats_installs_overview') }}"
17+
stats_ratings_app_version: "{{ source('google_play', 'stats_ratings_app_version') }}"
1618
stats_ratings_country: "{{ source('google_play', 'stats_ratings_country') }}"
19+
stats_ratings_device: "{{ source('google_play', 'stats_ratings_device') }}"
20+
stats_ratings_os_version: "{{ source('google_play', 'stats_ratings_os_version') }}"
21+
stats_ratings_overview: "{{ source('google_play', 'stats_ratings_overview') }}"
1722
stats_store_performance_country: "{{ source('google_play', 'stats_store_performance_country') }}"
1823
stats_store_performance_traffic_source: "{{ source('google_play', 'stats_store_performance_traffic_source') }}"
19-
stats_installs_overview: "{{ source('google_play', 'stats_installs_overview') }}"
20-
stats_crashes_overview: "{{ source('google_play', 'stats_crashes_overview') }}"
21-
stats_ratings_overview: "{{ source('google_play', 'stats_ratings_overview') }}"
22-
earnings: "{{ source('google_play', 'earnings') }}"
23-
financial_stats_subscriptions_country: "{{ source('google_play', 'financial_stats_subscriptions_country') }}"
2424
models:
2525
google_play_source:
2626
+schema: google_play_source

‎docs/catalog.json

+1-1
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)