By Toru Kobayashi (@koba04)
This article shows how to manage multiple packages in a single repository.
What is Monorepo?
In this article, what we call "Monorepo" is a way to manage multiple npm packages in a single repository.
Babel, Jest, and Create React App adopt Monorepo using Lerna, a tool I will later introduce in this article. React is also managing multiple packages in a single repository, but without Lerna.
Monorepo is not only for OSS libraries. Shopify manages many packages as Monorepo, and you can see various packages like AST utilities and React Components in the packages/
directory of Shopify/quilt repository.
Following are the Pros and Cons of Monorepo.
Pros
- You can make it easy to develop multiple packages dependent on each other
- You don't need to link each package using
npm link
- You don't need to link each package using
- You can manage npm package dependencies in a single repository
- This is especially useful when you are using a dependency management tool like Renovate
Cons
- You need to manage issues and pull requests in a single repository
- You need to understand a workflow using Monorepo tools
We have npm packages that help Kintone customization development and manage them in separate repositories. This has some disadvantages, like the cost for updating dependencies with Renovate and developing a package having a mutual dependency.
Currently, we are developing a new tool to improve the developer experience of Kintone customization development. It was a good opportunity to adopt the Monrepo architecture, so I created a new repository as Monorepo.
You can see the repository from the following link. Now, the repository manages only a few packages, but we are going to add more to it.
https://github.com/kintone/js-sdk
First, we will introduce Lerna and Yarn workspaces.
Lerna
Lerna is "a tool for managing JavaScript projects with multiple packages", as the official website says, which is a tool to manage multiple npm packages in a single repository.
Lerna provides the following features.
- Manage multiple npm packages in a single repository
lerna bootstrap
command installs all package's dependencies- Hoist duplicated dependencies
- "Hoist" means to install duplicated dependencies into the root directory rather than each package directory
lerna publish
command publishes npm packages that have changes- You can choose to manage all your packages as a single version, or manage each version separately
lerna run
command runs the same npm-scripts in each npm package at once- Import existing git repositories to a Monorepo
Yarn Workspaces
https://classic.yarnpkg.com/lang/en/
Yarn is a JavaScript package manager. You can manage npm packages with Yarn in the same way as npm. This article doesn't mention the difference between Yarn and npm, so please refer to the following link if you want to know more about it.
https://classic.yarnpkg.com/en/docs/migrating-from-npm
*This article uses Yarn v1
Yarn has a feature called Yarn Workspaces, which is a way to manage multiple npm packages. npm doesn't have the feature.
https://classic.yarnpkg.com/en/docs/workspaces
Note: npm has a plan to support Monorepo at v7. The following is the RFC for this.
https://github.com/npm/rfcs/pull/103
Yarn Workspaces provide the following features.
- Manage multiple npm packages on a single repository
yarn install
command installs all package's dependencies- Hoist duplicated dependencies
yarn workspaces
command run the same npm-scripts in each npm package at onceyarn.lock
file manages all dependencies for all packages
You can see that Yarn Workspaces and Lerna have similar features.
As Yarn's documentation mentioned, "Yarn’s workspaces are the low-level primitives that tools like Lerna can (and do!) use", Yarn Workspaces provide lower-level APIs than Lerna.
Using Lerna and Yarn Workspaces together
Lerna provides an option for you to use it with Yarn Workspaces.
To use Yarn instead of the npm client, you have to specify "npmClient": "yarn"
in lerna.json
, a setting file of Lerna.
Many projects use Lerna with Yarn Workspaces.
You might wonder why many projects use Yarn Workspaces, not only Lerna, even though Lerna provides features similar to those of Yarn. This question makes sense. You can do what you want with Lerna and npm client. However, Yarn Workspaces provide more intuitive ways to manage Monorepo because Yarn supports Monorepo at a package manager level.
For example, Yarn Workspaces create a single yarn.lock
file in the root directory, and use it to manage all dependencies between multiple packages, instead of creating one for each. This makes diffs minimum when updating dependencies.
How to use
This section describes the way to use Lerna and Yarn Workspaces to manage Monorepo.
Settings
First, you have to set the private
and workspaces
fields in package.json
in the root directory.
The private
field should be true
, and the workspaces
field should be directories in the repository where you put your packages.
You can specify directories like examples
for the workspaces
field, so we use the examples
directory to put demo and sample scripts in.
// package.json // Add private and workspaces fields { : "private": true, "workspaces": [ "packages/*", "examples/*" ], : }
Install Lerna and run lerna init
.
% yarn add --dev lerna % yarn lerna init --independent lerna notice cli v3.19.0 lerna info Updating package.json lerna info Creating lerna.json lerna info Creating packages directory lerna success Initialized Lerna files ✨ Done in 0.98s.
We use the --independent
flag because we'd like to manage the versions of the packages separately.
If you don't set the flag, all npm packages have the same version.
Next, we configure lerna.json
for Yarn Workspaces.
{ "npmClient": "yarn", // Add "useWorkspaces": true, "version": "independent" }
Lerna has the useWorkSpaces option. This option enables us to reuse the setting for Yarn Workspaces as Lerna's workspace setting.
Of course, you can manage different packages between Lerna and Yarn Workspaces.
Jest adopts this way, and they use Yarn Workspaces to manage the packages
、website
、examples
directories, and Lerna to manage the packages
directory.
- https://github.com/facebook/jest/blob/44a960de28035e7590b21c25de44af5f0f1796df/package.json#L114-L120
- https://github.com/facebook/jest/blob/44a960de28035e7590b21c25de44af5f0f1796df/lerna.json#L4-L6
We use useWorkSpaces
because we'd like to process the examples
directory with Lerna.
Adding a new package
lerna create
is a command to add a new package to be managed in Monorepo.
% npx lerna create @kintone/rest-api-client
Installing an npm package
To install an npm package, we can use the yarn workspace
command in the root directory or the yarn add package-name
command in the package directory where you'd like to install the package.
% cd packages/rest-api-client % yarn add form-data # or % yarn workspace @kintone/rest-api-client add form-data
Yarn updates package.json
for the package and yarn.lock
placed in the root directory, and installs the npm package in the node_modules/
directory in the root directory.
Installing an npm package shared between multiple packages
Yarn Workspaces makes it possible to share dependencies installed in the root directory between all packages, useful for devDependencies
like TypeScript, ESLint, and Jest. It leads to implicit dependencies because they are not listed in each package.json
, but it reduces the cost to manage these npm packages in each package directory.
To install shared npm packages, you have to add the -W
flag to the yarn add
command.
% yarn add -W --dev typescript prettier eslint
Publishing
lerna publish
is a command to publish npm packages.
If your commit logs follow the Conventional Commits style, you can use the --conventional-commits
option to determine a published version. With this option, Lerna generates a CHANGELOG.md
for a new version.
% npx lerna publish --conventional-commits lerna notice cli v3.19.0 lerna info versioning independent lerna info Looking for changed packages since @kintone/rest-api-client@1.0.0 lerna info getChangelogConfig Successfully resolved preset "conventional-changelog-angular" Changes: - @kintone/rest-api-client: 1.0.0 => 1.1.0 - @kintone/cutomize-uploader: 3.0.1 => 3.0.2 ? Are you sure you want to publish these packages? (ynH) : Successfully published: - @kintone/rest-api-client@1.1.0 - @kintone/customize-uploader@3.0.2 lerna success published 2 packages
When publishing a new package to the npm registry, you can use the from-package
option.
% npx lerna publish from-package --conventional-commits
If you have a private package that you don't publish, you have to specify "private": true
in package.json
.
Setup after git clone
If you are using Lerna without Yarn Workspaces, you have to run the lerna bootstrap
command to set up a repository, but with Yarn Workspaces, the yarn install
command does everything for you.
% yarn install
Run npm-scripts in multiple packages
lerna run
is a command to run npm-scripts in all packages managed by Lerna.
For example, the lerna run test
command runs npm test
in all packages. The --stream
option prints each output on the terminal.
The following is a result of running lerna run
with the --stream
option.
% npx lerna run test --stream lerna notice cli v3.20.2 lerna info versioning independent lerna info Executing command in 2 packages: "yarn run test" @kintone/customize-uploader: yarn run v1.22.4 @kintone/customize-uploader: $ jest --rootDir src @kintone/customize-uploader: PASS src/__tests__/init.test.ts @kintone/customize-uploader: PASS src/__tests__/util.test.ts @kintone/customize-uploader: PASS src/__tests__/import.test.ts @kintone/customize-uploader: Test Suites: 4 passed, 4 total @kintone/customize-uploader: Tests: 7 passed, 7 total @kintone/customize-uploader: Snapshots: 0 total @kintone/customize-uploader: Time: 1.948s, estimated 2s @kintone/customize-uploader: Ran all test suites. @kintone/customize-uploader: Done in 2.95s. @kintone/rest-api-client: yarn run v1.22.4 @kintone/rest-api-client: $ jest --rootDir src @kintone/rest-api-client: PASS src/client/__tests__/BulkRequestClient.test.ts @kintone/rest-api-client: PASS src/__tests__/KintoneAllRecordsError.test.ts @kintone/rest-api-client: PASS src/__tests__/url.test.ts @kintone/rest-api-client: PASS src/client/__tests__/File.test.ts @kintone/rest-api-client: PASS src/__tests__/KintoneRestAPIError.test.ts @kintone/rest-api-client: PASS src/__tests__/KintoneRestAPIClient.test.ts @kintone/rest-api-client: PASS src/__tests__/KintoneRequestConfigBuilder.test.ts @kintone/rest-api-client: PASS src/client/__tests__/RecordClient.test.ts @kintone/rest-api-client: PASS src/client/__tests__/AppClient.test.ts @kintone/rest-api-client: Test Suites: 9 passed, 9 total @kintone/rest-api-client: Tests: 243 passed, 243 total @kintone/rest-api-client: Snapshots: 0 total @kintone/rest-api-client: Time: 3.37s @kintone/rest-api-client: Ran all test suites. @kintone/rest-api-client: Done in 3.86s. lerna success run Ran npm script 'test' in 2 packages in 7.5s: lerna success - @kintone/customize-uploader lerna success - @kintone/rest-api-client
The yarn workspaces
command, provided by Yarn Workspaces, is also able to run an npm-scripts in all packages like this.
yarn workspaces run test
Defining a naming rule for npm-scripts is a good practice to manage multiple packages. We have the following rules for all packages in a Monorepo repository.
https://github.com/kintone/js-sdk/blob/master/CONTRIBUTING.md#create-a-new-package
- build
- lint
- test
- test:ci (a test script for CI environment)
- prerelease (run scripts that have to run before publishing a package)
We also have a test to verify if these scripts are registered in each package.
https://github.com/kintone/js-sdk/blob/a603caabadc695a34f3202eb45699e505b58eb80/tests/npmScripts.test.ts
Migration from an existing repository
If you already have a repository, you need to migrate it into a Monorepo. Lerna has the import
command for this purpose.
To import an existing repository, you have to run the lerna import path/to/existing-repository
command.
Unfortunately, the import might fail if the existing repository has any merge conflict commits as Troubleshooting mentions.
According to the Troubleshooting, this can be avoided with the --flatten
option, but this option imports commit logs based on merge commits, so you cannot keep commit logs completely. If you don't want to import by merge commits, you can rebase all commits in an existing repository, and then run the lerna import
command again.
@kintone/rest-api-client
is imported to kintone/js-sdk
in this way.
After importing an existing package, you have to push a tag for the last commit with which you published the package. The tag format should look like package-name@version
because Lerna calculates diffs for the next release based on this format of a tag.
Run scripts in sample packages with ts-node
Currently, we have a package to manage sample scripts for rest-api-client
in the examples
directory.
This used to be included in rest-api-client
itself.
We used ts-node
to run the scripts without compiling TypeScript ahead of time and wanted to keep the same experiences even after the package was separated.
So we use the paths
option of compilerOptions
to be able to run scripts without compiling.
"baseUrl": "../../", "paths": { "@kintone/rest-api-client": ["packages/rest-api-client/src"] },
To enable the paths
option with ts-node
, we use the tsconfig-paths package.
Conclusion
We've introduced how to use Lerna and Yarn Workspaces to manage multiple packages in a Monorepo. Now, we are considering adopting the Project References feature that TypeScript provides to compile multiple packages effectively. I'll introduce it in another entry.