Skip to content

Commit 26e9a5a

Browse files
ndthuanndthuan
committed
Initial version
1 parent d84ff82 commit 26e9a5a

13 files changed

+966
-0
lines changed

‎.github/workflows/main.yml

+39
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Go
2+
3+
on:
4+
push:
5+
branches: [ master ]
6+
pull_request:
7+
branches: [ master ]
8+
9+
jobs:
10+
11+
build:
12+
name: Build
13+
runs-on: ubuntu-latest
14+
strategy:
15+
matrix:
16+
go-version: [1.18, 1.19, 1.20, 1.21, 1.22]
17+
steps:
18+
- uses: actions/checkout@v4
19+
with:
20+
fetch-depth: 10
21+
22+
- name: Set up Go
23+
uses: actions/setup-go@v2
24+
with:
25+
go-version: "${{ matrix.go-version }}"
26+
id: go
27+
28+
- name: Get dependencies
29+
run: go get -v -t -d ./...
30+
31+
- uses: gwatts/go-coverage-action@v2
32+
id: coverage
33+
with:
34+
# Optional coverage threshold
35+
# use fail-coverage to determine what should happen below this threshold
36+
coverage-threshold: 100
37+
38+
# collect coverage for all packages beyond the one under test
39+
cover-pkg: ./...

‎.idea/.gitignore

+8
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/kvsync.iml

+9
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/modules.xml

+8
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎.idea/vcs.xml

+6
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md

+139
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# KVSync
2+
3+
KVSync is a Go package that provides a simple and efficient way to synchronize your GORM models with a key-value store.
4+
5+
There can be multiple key definitions for each model. For example, you can have a key for fetching by ID, a key for fetching by UUID, and a composite key for fetching by both ID and UUID. For each key, the model data is replicated accordingly to the key-value store.
6+
7+
## Installation
8+
9+
To install KVSync, use the `go get` command:
10+
11+
```bash
12+
go get github.com/ndthuan/kvsync
13+
```
14+
15+
## Sync Setup
16+
17+
### Define Synced Models
18+
19+
Implement `kvsync.Syncable` and provide sync keys mapped by a name for fetching later. Each key is unique on the key-value store.
20+
21+
```go
22+
type SyncedUser struct {
23+
gorm.Model
24+
UUID string
25+
Username string
26+
}
27+
28+
func (u SyncedUser) SyncKeys() map[string]string {
29+
return map[string]string{
30+
"id": fmt.Sprintf("user:id:%d", u.ID),
31+
"uuid": fmt.Sprintf("user:uuid:%s", u.UUID),
32+
"composite": fmt.Sprintf("user:composite:%d_%s", u.ID, u.UUID),
33+
}
34+
}
35+
36+
```
37+
38+
### Configure Key-Value Store
39+
40+
With Redis for example, you can use the provided `RedisStore`. Steps:
41+
- Init GORM DB instance
42+
- Init Redis client
43+
- Create a new `RedisStore` instance
44+
- Create a new `KVSync` instance
45+
- Register GORM callbacks
46+
47+
```go
48+
package main
49+
50+
import (
51+
"context"
52+
"fmt"
53+
"github.com/ndthuan/kvsync"
54+
"github.com/redis/go-redis/v9"
55+
"gorm.io/gorm"
56+
"time"
57+
)
58+
59+
func main() {
60+
db, err := gorm.Open(...)
61+
if err != nil {
62+
panic(fmt.Sprintf("Failed to connect to database: %v", err))
63+
}
64+
65+
clusterClient := redis.NewClusterClient(&redis.ClusterOptions{
66+
Addrs: []string{},
67+
})
68+
69+
// Create a new RedisStore
70+
store := &kvsync.RedisStore{
71+
Client: clusterClient,
72+
Expiration: time.Hour * 24 * 365, // Set the expiration time for the keys
73+
Prefix: "kvsync:", // Optional, defaults to "kvsync:"
74+
Marshaler: &kvsync.BSONMarshalingAdapter{}, // Optional, BSONMarshalingAdapter (using Mongo's BSON) is the default and recommended
75+
}
76+
77+
ctx, cancel := context.WithCancel(context.Background())
78+
defer cancel()
79+
80+
kvSync := kvsync.NewKVSync(ctx, kvsync.Options{
81+
Store: store,
82+
Workers: 4,
83+
ReportCallback: func(r kvsync.Report) {
84+
if r.Err == nil {
85+
actualDoneCount++
86+
}
87+
},
88+
})
89+
90+
// Register the GORM callbacks for automated synchronization
91+
db.Callback().Create().After("gorm:create").Register("kvsync:create", kvSync.GormCallback())
92+
db.Callback().Update().After("gorm:update").Register("kvsync:update", kvSync.GormCallback())
93+
94+
}
95+
```
96+
97+
### And create/update your model as usual
98+
99+
```go
100+
// Create a new SyncedUser
101+
db.Create(&SyncedUser{
102+
UUID: "test-uuid",
103+
Username: "test-username",
104+
})
105+
// The SyncedUser is automatically synchronized with the key-value store
106+
```
107+
108+
## Fetching Synced Models
109+
110+
You can fetch the model by any of the keys you defined. You must provide a struct with non-zero values for the keys you want to fetch by.
111+
112+
By ID
113+
```go
114+
user := SyncedUser{
115+
Model: gorm.Model{ID: 1},
116+
}
117+
kvSync.Fetch(&user, "id")
118+
```
119+
120+
By UUID
121+
```go
122+
user := SyncedUser{
123+
UUID: "test-uuid",
124+
}
125+
kvSync.Fetch(&user, "uuid")
126+
```
127+
128+
By composite key
129+
```go
130+
user := SyncedUser{
131+
Model: gorm.Model{ID: 1},
132+
UUID: "test-uuid",
133+
}
134+
kvSync.Fetch(&user, "composite")
135+
```
136+
137+
## License
138+
139+
KVSync is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information.

‎go.mod

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
module github.com/ndthuan/kvsync
2+
3+
go 1.18
4+
5+
require (
6+
github.com/alicebob/miniredis/v2 v2.33.0
7+
github.com/redis/go-redis/v9 v9.5.3
8+
github.com/stretchr/testify v1.9.0
9+
go.mongodb.org/mongo-driver v1.15.0
10+
gorm.io/driver/sqlite v1.5.5
11+
gorm.io/gorm v1.25.10
12+
)
13+
14+
require (
15+
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect
16+
github.com/cespare/xxhash/v2 v2.2.0 // indirect
17+
github.com/davecgh/go-spew v1.1.1 // indirect
18+
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
19+
github.com/jinzhu/inflection v1.0.0 // indirect
20+
github.com/jinzhu/now v1.1.5 // indirect
21+
github.com/mattn/go-sqlite3 v1.14.17 // indirect
22+
github.com/pmezard/go-difflib v1.0.0 // indirect
23+
github.com/yuin/gopher-lua v1.1.1 // indirect
24+
gopkg.in/yaml.v3 v3.0.1 // indirect
25+
)

‎go.sum

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZpUGpz5+4FfNmIU+FmZg2P3Xaj2v2bfNWmk=
2+
github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc=
3+
github.com/alicebob/miniredis/v2 v2.33.0 h1:uvTF0EDeu9RLnUEG27Db5I68ESoIxTiXbNUiji6lZrA=
4+
github.com/alicebob/miniredis/v2 v2.33.0/go.mod h1:MhP4a3EU7aENRi9aO+tHfTBZicLqQevyi/DJpoj6mi0=
5+
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
6+
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
7+
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
8+
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
9+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
10+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
11+
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
12+
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
13+
github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
14+
github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E=
15+
github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc=
16+
github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
17+
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
18+
github.com/mattn/go-sqlite3 v1.14.17 h1:mCRHCLDUBXgpKAqIKsaAaAsrAlbkeomtRFKXh2L6YIM=
19+
github.com/mattn/go-sqlite3 v1.14.17/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg=
20+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
21+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
22+
github.com/redis/go-redis/v9 v9.5.3 h1:fOAp1/uJG+ZtcITgZOfYFmTKPE7n4Vclj1wZFgRciUU=
23+
github.com/redis/go-redis/v9 v9.5.3/go.mod h1:hdY0cQFCN4fnSYT6TkisLufl/4W5UIXyv0b/CLO2V2M=
24+
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
25+
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
26+
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
27+
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
28+
go.mongodb.org/mongo-driver v1.15.0 h1:rJCKC8eEliewXjZGf0ddURtl7tTVy1TK3bfl0gkUSLc=
29+
go.mongodb.org/mongo-driver v1.15.0/go.mod h1:Vzb0Mk/pa7e6cWw85R4F/endUC3u0U9jGcNU603k65c=
30+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
31+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
32+
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
33+
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
34+
gorm.io/driver/sqlite v1.5.5 h1:7MDMtUZhV065SilG62E0MquljeArQZNfJnjd9i9gx3E=
35+
gorm.io/driver/sqlite v1.5.5/go.mod h1:6NgQ7sQWAIFsPrJJl1lSNSu2TABh0ZZ/zm5fosATavE=
36+
gorm.io/gorm v1.25.10 h1:dQpO+33KalOA+aFYGlK+EfxcI5MbO7EP2yYygwh9h+s=
37+
gorm.io/gorm v1.25.10/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8=

0 commit comments

Comments
 (0)