Kintone Engineering Blog

Learn about Kintone's engineering efforts. Kintone is provided by Cybozu Inc., a Tokyo-based public company founded in 1997.

Managing Monorepo using Lerna and Yarn workspaces

By Toru Kobayashi (@koba04)

This article shows how to manage multiple packages in a single repository.

What is Monorepo?

In this article, what we call "Monorepo" is a way to manage multiple npm packages in a single repository.

Babel, Jest, and Create React App adopt Monorepo using Lerna, a tool I will later introduce in this article. React is also managing multiple packages in a single repository, but without Lerna.

Monorepo is not only for OSS libraries. Shopify manages many packages as Monorepo, and you can see various packages like AST utilities and React Components in the packages/ directory of Shopify/quilt repository.

Following are the Pros and Cons of Monorepo.

Pros

  • You can make it easy to develop multiple packages dependent on each other
    • You don't need to link each package using npm link
  • You can manage npm package dependencies in a single repository
    • This is especially useful when you are using a dependency management tool like Renovate

Cons

  • You need to manage issues and pull requests in a single repository
  • You need to understand a workflow using Monorepo tools

We have npm packages that help Kintone customization development and manage them in separate repositories. This has some disadvantages, like the cost for updating dependencies with Renovate and developing a package having a mutual dependency.

Currently, we are developing a new tool to improve the developer experience of Kintone customization development. It was a good opportunity to adopt the Monrepo architecture, so I created a new repository as Monorepo.

You can see the repository from the following link. Now, the repository manages only a few packages, but we are going to add more to it.

https://github.com/kintone/js-sdk

First, we will introduce Lerna and Yarn workspaces.

Lerna

https://lerna.js.org/

Lerna is "a tool for managing JavaScript projects with multiple packages", as the official website says, which is a tool to manage multiple npm packages in a single repository.

Lerna provides the following features.

  • Manage multiple npm packages in a single repository
  • lerna bootstrap command installs all package's dependencies
  • Hoist duplicated dependencies
    • "Hoist" means to install duplicated dependencies into the root directory rather than each package directory
  • lerna publish command publishes npm packages that have changes
    • You can choose to manage all your packages as a single version, or manage each version separately
  • lerna run command runs the same npm-scripts in each npm package at once
  • Import existing git repositories to a Monorepo

Yarn Workspaces

https://classic.yarnpkg.com/lang/en/

Yarn is a JavaScript package manager. You can manage npm packages with Yarn in the same way as npm. This article doesn't mention the difference between Yarn and npm, so please refer to the following link if you want to know more about it.

https://classic.yarnpkg.com/en/docs/migrating-from-npm

*This article uses Yarn v1

Yarn has a feature called Yarn Workspaces, which is a way to manage multiple npm packages. npm doesn't have the feature.

https://classic.yarnpkg.com/en/docs/workspaces

Note: npm has a plan to support Monorepo at v7. The following is the RFC for this.
https://github.com/npm/rfcs/pull/103

Yarn Workspaces provide the following features.

  • Manage multiple npm packages on a single repository
  • yarn install command installs all package's dependencies
  • Hoist duplicated dependencies
  • yarn workspaces command run the same npm-scripts in each npm package at once
  • yarn.lock file manages all dependencies for all packages

You can see that Yarn Workspaces and Lerna have similar features.

As Yarn's documentation mentioned, "Yarn’s workspaces are the low-level primitives that tools like Lerna can (and do!) use", Yarn Workspaces provide lower-level APIs than Lerna.

Using Lerna and Yarn Workspaces together

Lerna provides an option for you to use it with Yarn Workspaces. To use Yarn instead of the npm client, you have to specify "npmClient": "yarn" in lerna.json, a setting file of Lerna.

Many projects use Lerna with Yarn Workspaces.

You might wonder why many projects use Yarn Workspaces, not only Lerna, even though Lerna provides features similar to those of Yarn. This question makes sense. You can do what you want with Lerna and npm client. However, Yarn Workspaces provide more intuitive ways to manage Monorepo because Yarn supports Monorepo at a package manager level.

For example, Yarn Workspaces create a single yarn.lock file in the root directory, and use it to manage all dependencies between multiple packages, instead of creating one for each. This makes diffs minimum when updating dependencies.

How to use

This section describes the way to use Lerna and Yarn Workspaces to manage Monorepo.

Settings

First, you have to set the private and workspaces fields in package.json in the root directory. The private field should be true, and the workspaces field should be directories in the repository where you put your packages. You can specify directories like examples for the workspaces field, so we use the examples directory to put demo and sample scripts in.

// package.json
// Add private and workspaces fields
{
    :
    "private": true,
    "workspaces": [
        "packages/*",
        "examples/*"
    ],
    :
}

Install Lerna and run lerna init.

% yarn add --dev lerna
% yarn lerna init --independent
lerna notice cli v3.19.0
lerna info Updating package.json
lerna info Creating lerna.json
lerna info Creating packages directory
lerna success Initialized Lerna files
✨  Done in 0.98s.

We use the --independent flag because we'd like to manage the versions of the packages separately. If you don't set the flag, all npm packages have the same version.

Next, we configure lerna.json for Yarn Workspaces.

{
    "npmClient": "yarn", // Add
    "useWorkspaces": true,
    "version": "independent"
}

Lerna has the useWorkSpaces option. This option enables us to reuse the setting for Yarn Workspaces as Lerna's workspace setting.

Of course, you can manage different packages between Lerna and Yarn Workspaces. Jest adopts this way, and they use Yarn Workspaces to manage the packages、website、examples directories, and Lerna to manage the packages directory.

We use useWorkSpaces because we'd like to process the examples directory with Lerna.

Adding a new package

lerna create is a command to add a new package to be managed in Monorepo.

% npx lerna create @kintone/rest-api-client

Installing an npm package

To install an npm package, we can use the yarn workspace command in the root directory or the yarn add package-name command in the package directory where you'd like to install the package.

% cd packages/rest-api-client
% yarn add form-data
# or
% yarn workspace @kintone/rest-api-client add form-data

Yarn updates package.json for the package and yarn.lock placed in the root directory, and installs the npm package in the node_modules/ directory in the root directory.

Installing an npm package shared between multiple packages

Yarn Workspaces makes it possible to share dependencies installed in the root directory between all packages, useful for devDependencies like TypeScript, ESLint, and Jest. It leads to implicit dependencies because they are not listed in each package.json, but it reduces the cost to manage these npm packages in each package directory. To install shared npm packages, you have to add the -W flag to the yarn add command.

% yarn add -W --dev typescript prettier eslint

Publishing

lerna publish is a command to publish npm packages.

If your commit logs follow the Conventional Commits style, you can use the --conventional-commits option to determine a published version. With this option, Lerna generates a CHANGELOG.md for a new version.

% npx lerna publish --conventional-commits
lerna notice cli v3.19.0
lerna info versioning independent
lerna info Looking for changed packages since @kintone/rest-api-client@1.0.0
lerna info getChangelogConfig Successfully resolved preset "conventional-changelog-angular"

Changes:
    - @kintone/rest-api-client: 1.0.0 => 1.1.0
    - @kintone/cutomize-uploader: 3.0.1 => 3.0.2

? Are you sure you want to publish these packages? (ynH)
:
Successfully published:
    - @kintone/rest-api-client@1.1.0
    - @kintone/customize-uploader@3.0.2
lerna success published 2 packages

When publishing a new package to the npm registry, you can use the from-package option.

% npx lerna publish from-package --conventional-commits

If you have a private package that you don't publish, you have to specify "private": true in package.json.

Setup after git clone

If you are using Lerna without Yarn Workspaces, you have to run the lerna bootstrap command to set up a repository, but with Yarn Workspaces, the yarn install command does everything for you.

% yarn install

Run npm-scripts in multiple packages

lerna run is a command to run npm-scripts in all packages managed by Lerna. For example, the lerna run test command runs npm test in all packages. The --stream option prints each output on the terminal.

The following is a result of running lerna run with the --stream option.

% npx lerna run test --stream
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Executing command in 2 packages: "yarn run test"
@kintone/customize-uploader: yarn run v1.22.4
@kintone/customize-uploader: $ jest --rootDir src
@kintone/customize-uploader: PASS src/__tests__/init.test.ts
@kintone/customize-uploader: PASS src/__tests__/util.test.ts
@kintone/customize-uploader: PASS src/__tests__/import.test.ts
@kintone/customize-uploader: Test Suites: 4 passed, 4 total
@kintone/customize-uploader: Tests:       7 passed, 7 total
@kintone/customize-uploader: Snapshots:   0 total
@kintone/customize-uploader: Time:        1.948s, estimated 2s
@kintone/customize-uploader: Ran all test suites.
@kintone/customize-uploader: Done in 2.95s.
@kintone/rest-api-client: yarn run v1.22.4
@kintone/rest-api-client: $ jest --rootDir src
@kintone/rest-api-client: PASS src/client/__tests__/BulkRequestClient.test.ts
@kintone/rest-api-client: PASS src/__tests__/KintoneAllRecordsError.test.ts
@kintone/rest-api-client: PASS src/__tests__/url.test.ts
@kintone/rest-api-client: PASS src/client/__tests__/File.test.ts
@kintone/rest-api-client: PASS src/__tests__/KintoneRestAPIError.test.ts
@kintone/rest-api-client: PASS src/__tests__/KintoneRestAPIClient.test.ts
@kintone/rest-api-client: PASS src/__tests__/KintoneRequestConfigBuilder.test.ts
@kintone/rest-api-client: PASS src/client/__tests__/RecordClient.test.ts
@kintone/rest-api-client: PASS src/client/__tests__/AppClient.test.ts
@kintone/rest-api-client: Test Suites: 9 passed, 9 total
@kintone/rest-api-client: Tests:       243 passed, 243 total
@kintone/rest-api-client: Snapshots:   0 total
@kintone/rest-api-client: Time:        3.37s
@kintone/rest-api-client: Ran all test suites.
@kintone/rest-api-client: Done in 3.86s.
lerna success run Ran npm script 'test' in 2 packages in 7.5s:
lerna success - @kintone/customize-uploader
lerna success - @kintone/rest-api-client

The yarn workspaces command, provided by Yarn Workspaces, is also able to run an npm-scripts in all packages like this.

yarn workspaces run test

Defining a naming rule for npm-scripts is a good practice to manage multiple packages. We have the following rules for all packages in a Monorepo repository.

https://github.com/kintone/js-sdk/blob/master/CONTRIBUTING.md#create-a-new-package

  • build
  • lint
  • test
  • test:ci (a test script for CI environment)
  • prerelease (run scripts that have to run before publishing a package)

We also have a test to verify if these scripts are registered in each package.

https://github.com/kintone/js-sdk/blob/a603caabadc695a34f3202eb45699e505b58eb80/tests/npmScripts.test.ts

Migration from an existing repository

If you already have a repository, you need to migrate it into a Monorepo. Lerna has the import command for this purpose.

To import an existing repository, you have to run the lerna import path/to/existing-repository command.

Unfortunately, the import might fail if the existing repository has any merge conflict commits as Troubleshooting mentions.

According to the Troubleshooting, this can be avoided with the --flatten option, but this option imports commit logs based on merge commits, so you cannot keep commit logs completely. If you don't want to import by merge commits, you can rebase all commits in an existing repository, and then run the lerna import command again. @kintone/rest-api-client is imported to kintone/js-sdk in this way.

After importing an existing package, you have to push a tag for the last commit with which you published the package. The tag format should look like package-name@version because Lerna calculates diffs for the next release based on this format of a tag.

Run scripts in sample packages with ts-node

Currently, we have a package to manage sample scripts for rest-api-client in the examples directory. This used to be included in rest-api-client itself. We used ts-node to run the scripts without compiling TypeScript ahead of time and wanted to keep the same experiences even after the package was separated. So we use the paths option of compilerOptions to be able to run scripts without compiling.

    "baseUrl": "../../",
    "paths": {
      "@kintone/rest-api-client": ["packages/rest-api-client/src"]
    },

To enable the paths option with ts-node, we use the tsconfig-paths package.

Conclusion

We've introduced how to use Lerna and Yarn Workspaces to manage multiple packages in a Monorepo. Now, we are considering adopting the Project References feature that TypeScript provides to compile multiple packages effectively. I'll introduce it in another entry.