Sharing Code Between Projects: Lessons Learned In The Trenches

Sharing Code Between Projects: Lessons Learned In The Trenches

Sharing Code Between Projects: Lessons Learned In The Trenches

Jonathan Saring

2018-04-25T14:40:51+02:00
2018-04-25T13:20:26+00:00

About a year ago, we came to a crossroad that changed the way we build software today. Like many other teams, we were working on a few things at a time, developing different projects for our web and mobile applications, with shared ingredients in the form of common Node.js code between our back-end repositoriess and microservices, and common React UI components with some slight visual and functional differences between our apps.

As our team grew and code lines multiplied, we began to realize that with every passing day we were writing the same code over and over again. Over time, it became harder to maintain our code base and develop new features with the same speed and efficiency.

Finally, we decided to find a solution that would enable us to share and sync common components of code between our projects. Here is what we learned along our journey, which eventually gave birth to Bit.

Sharing code components between projects

Code components as building blocks. (Large preview)

Common Code In The Wild

While Git is great for collaborating on a single repository, sharing code between multiple projects can be more challenging than we think.

To get started, we looked into our own code base to learn how many times we duplicated our own integration to our user service. The unbelievable result was no less than 86 instances. After the initial shock, we started thinking that this must also be happening elsewhere.

Code shared in multiple projects

Using the same code in different places. (Large preview)

We asked some friends working in a few different organizations of various sizes to run a simple copy-and-paste detection on their code base, looking for duplicates of code longer than 100 lines. The result blew us away: On average, more than 30% of their code base was duplicated.

Finally, we decided to look deep into the open-source projects on GitHub, checking for both duplications and re-implementations of a simple isString function in the 10,000 most popular JavaScript GitHub projects.

Amazingly, we found this function was implemented in more than 100 different ways and duplicated over 1,000 times in only 10,000 repositories. Later studies claim that over 50% of the code on GitHub is actually duplicated. We realized we were not the only ones facing this issue.

Looking For A Solution

Before building Bit, we looked for a tool that would help us turn the smaller components that our apps are built from into building blocks that could be shared between our projects and synced across our code base. We also wanted to organize them and make them discoverable for our team. Here’s a short summary of what we learned.

A Micro-Package Arsenal With NPM

At first, we considered publishing all of our UI components, utility functions and smaller modules as packages to NPM. This seemed like the obvious solution for modularity for our software’s building blocks. However, we quickly learned that this solution came with huge overhead.

Trying to publish a few files from our project to NPM forced us to split our repository and create new ones just to share this code. When dealing with hundreds of components, this meant having to maintain and make changes across hundreds of repositories.

We would also have to refactor our code base, removing the newly created packages from their original repositories, boilerplating the packages in the new repositories and so on.

Even then, we had now a simple way to organize these packages and make them easily discoverable to our entire team. Another major problem was the coupling between the packages and the owners of their origin repositories, which made it nearly impossible for other people to quickly make updates to the packages while working on their own projects.

This kind of overhead was too much for us to handle. So, we quickly decided to look for a better way to share our code.

Lerna Monorepos

The next option we came up with was to use Lerna in order to refactor our code base into a few multi-package repositories, often referred to as “monorepos”.

The Hydra of Lerna

Lerna multi-package repository. (Large preview)

The upside of this solution was that it would allow us to maintain and publish all our packages from a single repository. However, this option, too, came with a set of drawbacks, particularly when working with smaller components.

Choosing this option meant we would still have to effectively keep multiple packages with multiple package.json files, multiple build and test environments and a complicated dependency tree to handle between them. Updating these packages must also go through the main repository, still making it hard to modify these package from other projects when working with a few separate monorepos.

For example, take the popular Material-UI React UI library. Even though it uses Lerna to publish five different packages from the same repository, you would still have to install the entire library to use each of its components. Making changes would still have to go through that project as well, and discoverability for these component didn’t improve.

Monorepos can be great for some cases (such as testing or building a project as a whole) and can definitely work for some teams. However, refactoring your entire code base just to share common code between projects while still having to struggle with the issues mentioned above made us drop this option as well.

Shared Libraries

This option was quickly dropped, too. In a lot of way, it resembles using a CD-ROMs instead of an iTunes playlist. First, it made no sense to force an entire library of React components and an entire utility library and so on on each of our projects.

Secondly, every project using it would be tightly coupled to the development of this library, making it impossible to adjust its components for each project. This becomes most painful when sharing common Node.js code between our microservices, which would now be coupled to the library.

Thirdly, discoverability within the library is bound to be poor and would involve a lot of work with its documentation and usage in different edge cases.

Because it makes very little sense to couple and slow down our development, we try to minimize the use of these libraries as much as possible. Even popular JavaScript utility libraries such as Lodash are working hard to make their smaller components independently available via NPM.

Git Submodules

Finally, we turned back time and looked into working with Git submodules.

Git enables you to make one repository a subdirectory of another repository, creating a single working tree for the entire project, so that a repository can utilize code from another repository.

As for many other teams, this solution did not last for us. First, submodules only work on the master branch, which causes problems for rapid development. Secondly, submodules increase coupling between projects, which makes it hard to work on cross-repository assignments. Finally, a submodule repository is oblivious to its own nesting and the existence of dependent repositories.

After trying these different solutions, we realized that it shouldn’t be this complicated. There really should be a simpler way to organize, share and develop components of code from different projects. So, we decided to build it, and called it Bit.

Building Bit

Our vision for a solution was simple: turn our components and modules into building blocks that can be easily isolated from any project, organized in the cloud and used in any project.

Bit sharing workflow

Isolate, share and organize your reusable code. (Large preview)

When building it, we set a few guidelines for what we needed from the project.

  • Make it seamless to isolate and share code components from any project, without having to create new repositories or manually configure build and test environments and dependencies for each component.
  • Enable two-way development, so that each component could be changed and updated from any project, while changes would be synced across our code base.
  • Make it simple to organize and share our components, while making them discoverable for our entire team with useful visual information.

After hard work and extensive research, in 2017 we released the first version of Bit to GitHub.

How It Works

Bit’s workflow is made of three simple steps:

  1. The first is to simply tell Bit which components of code you would like to share from your project, and it will immediately start tracking them in all of the projects you share them in.
  2. You can then tag a version for these components so that Bit automatically defines and locks their dependency tree for both file and package dependencies, and creates an isolated environment for each component to build and test in isolation.
  3. Finally, you can share the components to the cloud (or your own remote server), where they will be organized, will be made discoverable and can be installed with NPM or Yarn like any other package.

You don’t have to create new repositories, split your code base or refactor a single line of code.


Source: Smashing Magazine