feat: parse allocation csvs #194

swimricky · 2023-10-20T12:55:49Z

No description provided.

vercel · 2023-10-20T12:55:57Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
grant-program	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 27, 2023 6:13pm

swimricky · 2023-10-20T12:56:34Z

frontend/utils/db.ts

@@ -74,6 +75,9 @@ export async function addClaimInfosToDatabase(
    })
  )

+  console.log(`\n\nbuilt merkle tree\n\n`)


TODO: clean up comments.
right now this insert is taking a long time so will look into optimizing

Yeah makes sense, I am sure you can batch the queries

guibescos · 2023-10-23T11:07:04Z

frontend/scripts/setup.sh

      usage
      exit
    else
      test=1
    fi
    shift
    ;;
+    -c|--csv)
+    if [ "$dev" -eq 1 ] || [ "$test" -eq 1 ]; then


Would something like if $dev + $test + $csv != 1 be simpler?

guibescos · 2023-10-23T11:14:57Z

frontend/scripts/populate_from_csv.ts

+    fs.readFileSync(path.resolve(CSV_DIR, defi_csv), 'utf-8'),
+    {
+      header: true,
+      skipEmptyLines: true,


Is this needed? we might want to code to crash if there are empty lines

i originally added this for handling any empty lines at the end but open to removing this and throwing errors/crashing

guibescos · 2023-10-23T11:15:17Z

frontend/scripts/populate_from_csv.ts

+  )
+  assert(
+    defiCsvClaims.meta.fields?.includes('alloc'),
+    "CSV file does not have required 'alloc' column"


should it check that there's a chain column

Also might be useful to refactor this into a hasColumns function

guibescos · 2023-10-23T11:22:26Z

frontend/scripts/populate_from_csv.ts

+        assert(
+          !curChainsForAddr.includes(chainAndAlloc[0]),
+          `CurChainsForAddr: ${curChainsForAddr}. Address ${key} has duplicate chain ${chainAndAlloc[0]}`
+        )


I think it would be good to have logic here that ignores dev addresses if they are already in the defi claims csv

i was planning on just removing them from the defi_dev.csv address but can change to this instead

guibescos · 2023-10-23T11:29:21Z

frontend/scripts/populate_from_csv.ts

 async function main() {
  await clearDatabase(pool)
+  const { claimInfos, evmBreakdownAddresses, solanaBreakdownData } = parseCsvs()
+  if (DEBUG) {
+    const [maxUser, maxAmount] = getMaxUserAndAmount(claimInfos)


Good sanity checks

guibescos · 2023-10-23T11:32:40Z

frontend/scripts/populate_from_csv.ts

+  assert(
+    discordClaimsAddrSet.size ===
+      discordDevClaims.length + discordClaims.length,
+    'Discord claims and discord dev claims have duplicate addresses'


Same comment as for defi_dev, maybe it's fine just skipping the dev claims that are already in the main table instead of crashing

guibescos · 2023-10-23T11:34:17Z

frontend/scripts/populate_from_csv.ts

+      evmBreakDowns.push({
+        chain,
+        identity,
+        amount: truncateAllocation(alloc),


Maybe this conversion should happen inside parseCsv and parseCsv should return a EvmBreakdownRow[] and a SolanaBreakdownRow[]

swimricky · 2023-10-26T16:30:42Z

@guibescos i left some console logging statements since this is just a script that will only be used internally but if you want me to remove them lmk and i can clean those up

guibescos · 2023-10-26T18:44:28Z

frontend/utils/db.ts

@@ -10,6 +10,8 @@ import { getMaxAmount } from '../claim_sdk/claim'
 import * as anchor from '@coral-xyz/anchor'
 import { MerkleTree } from '../claim_sdk/merkleTree'
 import { BN } from 'bn.js'
+const sql = require('sql') as any
+// import { SQL, define } from 'sql';


Commented out code

guibescos · 2023-10-26T18:59:18Z

frontend/.env.sample

@@ -30,3 +30,4 @@ DISPENSER_GUARD=[128,47,82,29,168,187,15,46,104,227,206,117,172,71,34,235,23,50,
 FUNDER_KEYPAIR=[145,197,43,77,224,103,196,174,132,195,48,31,177,97,237,163,15,196,217,142,181,204,104,107,98,82,213,0,155,140,218,180,30,119,201,38,51,176,207,221,193,222,235,244,163,250,125,66,68,196,45,208,212,201,232,178,100,163,24,21,106,83,66,174]

 PROGRAM_ID=Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS


Let's add DEPLOYER_WALLET here

guibescos · 2023-10-26T19:01:17Z

frontend/scripts/populate_from_csv.ts

@@ -17,7 +19,12 @@ import BN from 'bn.js'
 import NodeWallet from '@coral-xyz/anchor/dist/cjs/nodewallet'
 import { EvmBreakdownRow } from '../utils/db'
 import assert from 'assert'
+import path from 'path'
+import { SolanaBreakdown } from 'utils/api'


guibescos · 2023-10-26T19:01:42Z

frontend/scripts/populate_from_csv.ts

@@ -31,19 +38,43 @@ const FUNDER_KEYPAIR = Keypair.fromSecretKey(
 )
 const CLUSTER = envOrErr('CLUSTER')


guibescos · 2023-10-26T19:03:38Z

frontend/scripts/populate_from_csv.ts

+
+const NFT_CLAIMS = 'nft.csv'
+
+const ECOSYSTEM_LIST = [


Can we use Ecosystems from ../claim_sdk/claim

guibescos · 2023-10-26T19:14:01Z

frontend/scripts/populate_from_csv.ts

+  })
+
+  // need solana breakdown between nft & defi
+  const nftCsvClaims = Papa.parse(


Perhaps refactor into parseNftClaims

guibescos · 2023-10-26T19:20:15Z

frontend/utils/db.ts

-    )
+  let claimInfoChunks = []
+  let chunkSize = 100
+  let chunkCounts = [...Array(Math.ceil(claimInfos.length / chunkSize))]


You don't need this variable, only the index is important

guibescos · 2023-10-26T19:21:33Z

frontend/utils/db.ts

-      ]
-    )
+  let claimInfoChunks = []
+  let chunkSize = 100


Feels like this should be const

guibescos

I like it! Dropped many minor comments.

* Add change * Optimize csvs

swimricky commented Oct 20, 2023

View reviewed changes

vercel bot deployed to Preview October 20, 2023 12:56 View deployment

swimricky added 2 commits October 20, 2023 08:58

feat: parse allocation csvs

6ec336c

refactor: clean up

fb1b3c1

swimricky force-pushed the parse-csvs branch from f235c6b to fb1b3c1 Compare October 20, 2023 13:04

vercel bot deployed to Preview October 20, 2023 13:06 View deployment

guibescos reviewed Oct 23, 2023

View reviewed changes

feat: address feedback

0a8e779

swimricky requested a review from guibescos October 26, 2023 16:29

swimricky marked this pull request as ready for review October 26, 2023 16:30

fix: script fix

a22fd5f

vercel bot deployed to Preview October 26, 2023 16:33 View deployment

guibescos reviewed Oct 26, 2023

View reviewed changes

guibescos approved these changes Oct 26, 2023

View reviewed changes

Parse csvs 2 (#207)

fd862d2

* Add change * Optimize csvs

vercel bot deployed to Preview October 27, 2023 18:00 View deployment

fix: address feedback

802c905

vercel bot deployed to Preview October 27, 2023 18:11 View deployment

fix: fix script

5b2d116

vercel bot deployed to Preview October 27, 2023 18:13 View deployment

swimricky merged commit c6e8e32 into main Oct 27, 2023
3 checks passed

swimricky deleted the parse-csvs branch October 27, 2023 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: parse allocation csvs #194

feat: parse allocation csvs #194

swimricky commented Oct 20, 2023

vercel bot commented Oct 20, 2023 •

edited

Loading

swimricky Oct 20, 2023

guibescos Oct 23, 2023

guibescos Oct 23, 2023

guibescos Oct 23, 2023

swimricky Oct 24, 2023 •

edited

Loading

guibescos Oct 23, 2023

guibescos Oct 23, 2023

guibescos Oct 23, 2023 •

edited

Loading

swimricky Oct 24, 2023

guibescos Oct 23, 2023

guibescos Oct 23, 2023

guibescos Oct 23, 2023 •

edited

Loading

swimricky commented Oct 26, 2023

guibescos Oct 26, 2023 •

edited

Loading

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos Oct 26, 2023

guibescos left a comment

		@@ -30,3 +30,4 @@ DISPENSER_GUARD=[128,47,82,29,168,187,15,46,104,227,206,117,172,71,34,235,23,50,
		FUNDER_KEYPAIR=[145,197,43,77,224,103,196,174,132,195,48,31,177,97,237,163,15,196,217,142,181,204,104,107,98,82,213,0,155,140,218,180,30,119,201,38,51,176,207,221,193,222,235,244,163,250,125,66,68,196,45,208,212,201,232,178,100,163,24,21,106,83,66,174]

		PROGRAM_ID=Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS

		@@ -31,19 +38,43 @@ const FUNDER_KEYPAIR = Keypair.fromSecretKey(
		)
		const CLUSTER = envOrErr('CLUSTER')

feat: parse allocation csvs #194

feat: parse allocation csvs #194

Conversation

swimricky commented Oct 20, 2023

vercel bot commented Oct 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swimricky Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guibescos Oct 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guibescos Oct 23, 2023 • edited Loading

Choose a reason for hiding this comment

swimricky commented Oct 26, 2023

guibescos Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guibescos left a comment

Choose a reason for hiding this comment

vercel bot commented Oct 20, 2023 •

edited

Loading

swimricky Oct 24, 2023 •

edited

Loading

guibescos Oct 23, 2023 •

edited

Loading

guibescos Oct 23, 2023 •

edited

Loading

guibescos Oct 26, 2023 •

edited

Loading