Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare electron 35 #22

Merged
merged 36 commits into from
Mar 12, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
05a690a
update
bigorn0 Mar 3, 2025
ce7e3b9
Upgrade tooling and patches
bigorn0 Mar 10, 2025
49aebf0
Fix kerberos patch
bigorn0 Mar 10, 2025
01b92b3
bump node to 22
bigorn0 Mar 10, 2025
8a48114
Support locally linked modules or workspaces
bigorn0 Mar 10, 2025
a94daec
Revert back winapi for now
bigorn0 Mar 10, 2025
f9e65f4
try bundle dependencies
bigorn0 Mar 10, 2025
3d30c35
rollback npm env
bigorn0 Mar 10, 2025
ae77ee8
Bump counter
bigorn0 Mar 10, 2025
b7adea9
add apt update
bigorn0 Mar 10, 2025
c176d30
Remove test on ssh-wrapper
bigorn0 Mar 10, 2025
9ced787
Fix couchbase patch
bigorn0 Mar 10, 2025
315387d
Rollback ubuntu version
bigorn0 Mar 10, 2025
7b9045e
add apt update
bigorn0 Mar 10, 2025
6a0a88c
Try to fix studio
bigorn0 Mar 10, 2025
7a75ed9
revert deps
bigorn0 Mar 10, 2025
e355e11
Add sharp native modules
bigorn0 Mar 11, 2025
6578ace
Filter out sharp-libvips for now
bigorn0 Mar 11, 2025
a031010
run patch-package first
bigorn0 Mar 11, 2025
602a240
tweak exports
bigorn0 Mar 11, 2025
034c0a6
Fix sharp patch and repackaging of dylibs
bigorn0 Mar 12, 2025
3fc8e81
Bump build counter
bigorn0 Mar 12, 2025
c34e6e3
Add sharp loading test
bigorn0 Mar 12, 2025
8182c78
Rename test
bigorn0 Mar 12, 2025
613639b
Update
bigorn0 Mar 12, 2025
af757b2
Fix desktop trampoline patch
bigorn0 Mar 12, 2025
6a36177
improve trampoline test coverage
bigorn0 Mar 12, 2025
d518147
Remove sharp for now
bigorn0 Mar 12, 2025
9fb02fd
add some initial loging
bigorn0 Mar 12, 2025
a045812
put back patch after
bigorn0 Mar 12, 2025
648faa8
Bump
bigorn0 Mar 12, 2025
1738832
Ensure npm 11
bigorn0 Mar 12, 2025
71f4bd4
Fix typo
bigorn0 Mar 12, 2025
8197d46
try to replace xvfb on wayland
bigorn0 Mar 12, 2025
36eee71
revert to installing xvfb
bigorn0 Mar 12, 2025
0d0abfd
revert to xvfb
bigorn0 Mar 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/build-prebuilds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: Install krb5
run: |
sudo apt-get update
Expand Down Expand Up @@ -59,7 +59,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: install python setuptools for workaround https://github.com/nodejs/node-gyp/issues/2869
run: brew install python-setuptools
- name: Install toolkit
Expand Down Expand Up @@ -99,7 +99,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: install python setuptools for workaround https://github.com/nodejs/node-gyp/issues/2869
run: pip install setuptools
- name: Install toolkit
Expand Down Expand Up @@ -136,7 +136,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: Install MIT Kerberos
run: choco install -y --limit-output mitkerberos --ia=ADDLOCAL=all --ignore-package-exit-codes
- name: Get MIT Kerberos path
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/discover-native-modules.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- run: npm ci
- name: Set Electron Major version in env
run: echo "ELECTRON_MAJOR_VERSION=$(node ./bin/get-electron-major-version.js)" >> $GITHUB_ENV
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish-studio-native-modules.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: Install toolkit
run: npm ci
- name: Set Electron Major version in env
Expand Down
28 changes: 21 additions & 7 deletions .github/workflows/test-native-modules copy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,14 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
- name: Install krb5
run: sudo apt-get install -y libkrb5-dev xvfb
node-version: 22
- name: install latest npm
run: |
npm install -g npm@11
- name: Install krb5 and Xvfb
run: |
sudo apt-get update
sudo apt-get install -y libkrb5-dev xvfb
- name: Install toolkit
run: npm ci
- name: Set Electron Major version in env
Expand Down Expand Up @@ -46,7 +51,10 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: install latest npm
run: |
npm install -g npm@11
- name: Install MIT Kerberos MSI
run: |
choco install --install-arguments="ADDLOCAL=all" --ignoredetectedreboot --ignorepackagecodes --confirm mitkerberos
Expand All @@ -73,7 +81,7 @@ jobs:
# Use the empty string ("") to ignore status or conclusion in the search
workflow_conclusion: success
name: modulesToBuildForElectron-${{ env.ELECTRON_MAJOR_VERSION }}
- name: Build on Windows x64
- name: Test native modules on Windows x64
run: npm run -w test test
test-mac-intel:
# Pinned down version to have intel arch
Expand All @@ -83,7 +91,10 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: install latest npm
run: |
npm install -g npm@11
- name: Install toolkit
run: npm ci
- name: Set Electron Major version in env
Expand Down Expand Up @@ -116,7 +127,10 @@ jobs:
- uses: actions/setup-node@v4
with:
cache: npm
node-version: 20
node-version: 22
- name: install latest npm
run: |
npm install -g npm@11
- name: Install toolkit
run: npm ci
- name: Set Electron Major version in env
Expand Down
1 change: 0 additions & 1 deletion .npmrc
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,4 @@ ignore-scripts=true
save-exact=true

//registry.npmjs.org/:_authToken=${NODE_AUTH_TOKEN}
always-auth=true
access=public
2 changes: 1 addition & 1 deletion bin/get-electron-major-version.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import semver from 'semver';
import electron from 'electron/package.json' assert { type: 'json' };
import electron from 'electron/package.json' with { type: 'json' };

console.log(semver.major(electron.version));
1 change: 1 addition & 0 deletions bin/prepare.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { writeFile } from 'node:fs/promises';
import { ROOT_DIR } from '#root';
import { resolve } from 'node:path';


const modulesBuildMetadata = await getNativeModulesMetadata();
await installAllPrebuildsProvidedAsNPMDependencies(modulesBuildMetadata);
await runPatchPackage();
Expand Down
16 changes: 9 additions & 7 deletions bin/publish-studio-natives.js
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
import pkg from '../modulesToBuild.json' assert { type: 'json' };
import pkg from '../modulesToBuild.json' with { type: 'json' };
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import { publishToNPM } from '../lib/publish.js';

const deps = pkg.map(({ name, rescopedModuleName, version }) => {
return { [name]: `npm:${rescopedModuleName}@${version}`};
return { [name]: `npm:${rescopedModuleName}@${version}` };
});

const windowsSpecificDep = [{'@hackolade/winapi-detect-rdp': '1.3.0'}];
const windowsSpecificDep = [{ '@hackolade/winapi-detect-rdp': '1.3.0' }];

const { default: studioNativesPkg } = await import('../studio-natives-modules/package.json', {
assert: { type: 'json' },
with: { type: 'json' },
});
studioNativesPkg.dependencies = [...deps, ...windowsSpecificDep].reduce((acc, dep) => {return {...dep, ...acc}}, {});
studioNativesPkg.dependencies = [...deps, ...windowsSpecificDep].reduce((acc, dep) => {
return { ...dep, ...acc };
}, {});


console.log("- studio native modules pkg:", studioNativesPkg);
console.log('- studio native modules pkg:', studioNativesPkg);
await writeFile(resolve('./studio-natives-modules/package.json'), JSON.stringify(studioNativesPkg));

await publishToNPM(resolve('./studio-natives-modules'));
2 changes: 1 addition & 1 deletion bin/rebuild.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { build } from '#lib/build.js';
import modules from '../modulesToBuild.json' assert { type: 'json' };
import modules from '../modulesToBuild.json' with { type: 'json' };
import { LOGGER } from '#lib/logger.js';
import { ROOT_DIR } from '#root';
import { runPatchPackage } from '../lib/install.js';
Expand Down
9 changes: 5 additions & 4 deletions lib/build-metadata.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import pkg from '../package.json' assert { type: 'json' };
import electron from 'electron/package.json' assert { type: 'json' };
import pkg from '../package.json' with { type: 'json' };
import electron from 'electron/package.json' with { type: 'json' };
import semver from 'semver';
import { join, resolve, sep } from 'node:path';

Expand Down Expand Up @@ -30,6 +30,7 @@ export const targetsDetails = [
targetPlatform: 'win32',
targetArch: 'x64',
},

];

export const setModuleNameWithoutScopeUnderNewScope = name => `${npmScope}/${name}`.replaceAll('-napi', '').replaceAll('-glibc', '');
Expand Down Expand Up @@ -136,7 +137,7 @@ export async function getNativeModulesMetadata() {

const nativeModulesWithProjectPath = Object.entries(nativeModules).map(async ([moduleName, meta]) => {
const baseDirectories = moduleInstalledPaths.filter(path => {
return path.endsWith(`${sep}${moduleName.replaceAll('/', sep, 'g')}`);
return path.endsWith(`${sep}${moduleName.replaceAll('/', sep, 'g')}`)
});

const [moduleBaseDir] = baseDirectories;
Expand Down Expand Up @@ -211,7 +212,7 @@ export function sanitizeModulePackageJSON({
if (modulePkg.dependencies && modulePkg.dependencies['node-addon-api']) {
delete modulePkg.dependencies['node-addon-api']; // only useful for build phase
}

delete modulePkg.gypfile;
delete modulePkg.bugs;
delete modulePkg.tsd;
Expand Down
18 changes: 13 additions & 5 deletions lib/fs.js
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
import { cp as fsCP, mkdir as fsMKDIR, rename, readdir as fsREADDIR, rm as fsRM } from 'node:fs/promises';
import {join} from 'node:path';

export const defaultOptions = { force: true, recursive: true };

export async function cp({ src, dest }) {
await fsCP(src, dest, defaultOptions);
export async function cp({ src, dest, options }) {
await fsCP(src, dest, {...defaultOptions, ...options});
}

export async function mkdir(dir) {
Expand All @@ -15,13 +16,20 @@ export async function mv({ src, dest }) {
}

export async function readdir(dir) {
return await fsREADDIR(dir, {
const entries = await fsREADDIR(dir, {
withFileTypes: true,
});

const allEntriesRecursively = await Promise.all(entries.map(async (fsEntry) =>{
if(fsEntry.isDirectory()){
return readdir(join(dir, fsEntry.name));
}
return fsEntry;
}));

return allEntriesRecursively.flat();
}

export async function rm(fileOrDir) {
await fsRM(fileOrDir, defaultOptions);
}


6 changes: 4 additions & 2 deletions lib/install.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@ export async function getOfficialPrebuildsAsNpmOptionalDependencies(optionalDepe
}

export async function runPatchPackage() {
log(`installing modules patches...`);
await exec({
command: npxCommand,
parameters: ['patch-package', '--error-on-warn', '--error-on-fail'],
options: {shell: true},
});
log(`modules patches installed!`);
}

export function setNPMConfForNAPI(existingRuntimeEnv, targetPlatform, targetArch) {
Expand Down Expand Up @@ -70,8 +72,8 @@ export async function installAllPrebuildsProvidedAsNPMDependencies(modulesBuildM
// Which typically blocks us from being able to install all targets like we need for cross building the application
// The only way to workaround that is to force npm to ignore any check or issue during installation, which is in general a bad idea but we don't have any other way.
// We also don't want to have this installation being persisted in our top level package.json file nor in the lock file: thus the use of --no-save
await exec({
command: npmCommand,
await exec({
command: npmCommand,
parameters: ['install', '--force', '--no-save', ...prebuildsToInstall],
options: {shell: true},
});
Expand Down
2 changes: 1 addition & 1 deletion lib/module.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export async function listModulesInBaseDirectory(baseDirectory) {
);

const topLevelModules = installedModules
.filter(fsEntry => fsEntry.isDirectory() && !fsEntry.name.startsWith('@'))
.filter(fsEntry => (fsEntry.isDirectory() || fsEntry.isSymbolicLink()) && !fsEntry.name.startsWith('@'))
.map(fsEntry => path.join(PROJECT_NODE_MODULES, fsEntry.name));

return [...topLevelModules, ...scopedModules].flat();
Expand Down
36 changes: 27 additions & 9 deletions lib/pack.js
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,35 @@ async function relocatePrebuildAndBinaries({
files: filesFromConfig,
};
} else {
log('relocatePrebuildAndBinaries baseDir: %o', baseDir);
const pkgFiles = await readdir(baseDir);
log('relocatePrebuildAndBinaries pkgFiles: %o', pkgFiles);

const [prebuild] = pkgFiles
.filter(entry => entry.isFile() && entry.name.endsWith('.node'))
.map(entry => join(baseDir, entry.name));
.map(entry => join(entry.parentPath, entry.name));

if(prebuild){
log('relocatePrebuildAndBinaries detected prebuild: %o', prebuild);

const prebuildName = 'prebuild.node';
const exports = {"./prebuild.node": "./prebuild.node"};
await cp({ src: prebuild, dest: join(baseDir, prebuildName) });

const prebuildName = 'prebuild.node';
await cp({ src: prebuild, dest: join(baseDir, prebuildName) });
const files = [prebuildName];

const files = [prebuildName];
return { files, main: prebuildName, exports };
}

// Handle new sharp case where the prebuild is a statically built dylib
const [dylib] = pkgFiles
.filter(entry => entry.isFile() && entry.name.endsWith('.dylib'))
.map(entry => join(entry.parentPath, entry.name));

return { files, main: prebuildName };
if(dylib){
log('relocatePrebuildAndBinaries detected dylib: %o', dylib);
return {};
}
}
} else {
return {};
Expand All @@ -59,7 +77,7 @@ export async function pack({
await rm(destBaseDir);
await mkdir(destBaseDir);

await cp({ src: srcBaseDir, dest: destBaseDir });
await cp({ src: srcBaseDir, dest: destBaseDir, options: {dereference: true } });
await rm(join(destBaseDir, 'deps'))

const isWindows = packageJSONData?.name?.includes('-win32-');
Expand All @@ -71,8 +89,9 @@ export async function pack({
relocatePrebuilds,
});
log('prepare package.json for module in directory: %o ', destBaseDir);
log('prepare package.json using files: %o ', filesFromConfig);
await writeFile(join(destBaseDir, 'package.json'), JSON.stringify({ ...packageJSONData, ...prebuildMeta,}));
const pkgData = { ...packageJSONData, ...prebuildMeta,}
log('prepare package.json for module in directory: %o ', pkgData);
await writeFile(join(destBaseDir, 'package.json'), JSON.stringify(pkgData));

modulesToPublish.push(destBaseDir);
return destBaseDir;
Expand All @@ -82,7 +101,6 @@ export async function rescopeOfficialPrebuildsFromPackage({ buildMetadata, resco
const toPublish = [];

for (const meta of buildMetadata) {
log(">>>>>>>>>>>> RESCOPE: %O", meta);
if (!rescopeOnlyPrebuilds) {
const modulePkg = await readPackageData(meta.moduleBaseDir);
const newJSONPkg = sanitizeModulePackageJSON({
Expand Down
Loading