How do I build a docker image with my unpublished extension?

I’ve built an extension that works correctly when using yarn run start in the browser-app directory (I used generator-theia-extension and started from the “blank” option). I’m now trying to build a docker image. I’ve forked the theia-apps/theia-docker repo. As I understand it, I need to include my extension in the dependencies in package.json. But my extension has not been published in a public repository (nor do I plan to).

I’ve tried adding my extension to the same directory as the Dockerfile, and including it as a local dependency like so:

In the Dockerfile, I added this line just before the yarn build:

COPY my-extension /tmp/my-extension

I added the dependency to latest.package.json and next.package.json:

"my-extension": "file:/tmp/my-extension"

But after building with docker build . I get the following error:

error Can't add "my-extension": invalid package version ""

I thought maybe I need to include the subdirectory package, so I tried changing the dependency:

"my-extension": "file:/tmp/my-extension/my-extension"

The build gets further along, but it gives the following error:

Module not found: Error: Can't resolve 'my-extension/lib/browser/my-extension-frontend-module' in '/home/theia/src-gen/frontend  

Is there some other way I can make this work?

@bendavis I believe you can use yarn workspaces so your extension is discoverable when you build your application: https://classic.yarnpkg.com/en/docs/workspaces/. It is the approach I personally use (since I have unpublished extensions), and my custom extensions are discovered during the build process.

@vince-fugnitto thanks, I’ll try that out. However, I’m confused on which package I should include in the 2nd workspace. Should it be the root of the extension project, or the my-extension subdirectory? Or should it be the browser-app directory?

It depends on how you setup your project, for example, we use yarn workspaces in the main repository of the framework as well:

This field helps the repo discover all extensions under packages, dev-packages, and examples be discoverable (without the need to necessarily be published) when building both our example examples/browser and examples/electron applications.

My project was created using generator-theia-extension, which has a directory structure like this:

- my-extension/
  - package.json
  - my-extension/
    - package.json
  - browser-app/
    - package.json
  - electron-app/
    - package.json

I’m not sure which one of those directories actually contains the extension itself. Is it the my-extension subdir, or the project root itself?

[edit] Also, how do I organize the workspace directories? From the yarn documentation, it seems like I should put the theia package.json in its own directory, and my extension in another, and then have a package.json in the root that points to the workspaces. But then, how do I actually build theia? When try this and run yarn theia build, it says: error Command "theia" not found.

@vince-fugnitto, I’ve tried every way I can think of to structure things with no luck. I’m using theia-docker as a starting point. What do I need to change in order to add my extension to this docker image?

What I’ve tried so far:

  1. I added my-extension (whose package.json defines theiaExtensions) to the same dir as the Dockerfile

  2. I moved latest.package.json in a subdir called “theia” and renamed it to package.json

  3. I created a new package.json in the same dir as the Dockerfile, with the following contents:

    {"private": "true", "workspaces": ["theia", "my-extension"]}
    
  4. I changed theia/package.json and added the dependency:

    {"my-extension": "1.0.0"}
    

After doing this, I’m not sure how to build theia. I’m using the same build command as in the Dockerfile. I first run yarn --pure-lockfile (which completes rather quickly). But when I run yarn theia build, it says “error Command "theia" not found”. So, after running yarn --pure-lockfile it does not seem to install theia correctly.

What am I missing?

There is development and production setup. Theia docker is production setup which consumes packages from npmjs and install only deps required at runtime.

For development setup you should follow: https://theia-ide.org/docs/authoring_extensions/

You can use yarn offline mirror to create cache of node_modules from development setup including local packages as well as packages from npmjs. Later in docker you can have base layer mounting such mirror and do install from it instead of npmjs. The rest will be like in theia docker image.

@akosyakov I am looking for how to make a production setup. I have already followed that tutorial to make the extension.

Hi @bendavis. The reason for this I think is that, for a production build, all dev-dependencies are stripped, including @theia/cli, which provides the “theia” part of yarn theia build. You can work-around that by depending on @theia/cli as a production dependency instead of a dev-dependency.

I think a setup such as described by @akosyakov would be ideal, but there are simpler ones that can be used too. Unfortunately there are no open-source examples of such that I know-of.

@vince-fugnitto maybe we could add a small Theia extension, e.g. to customize something, in the theia-apps repo and then use it in at least one image, as a simple example of how this can work? One idea would be to add a “more info about this image” Help menu item that one can click to open the theia-apps readme?

@marcdumais-work When I run yarn --pure-lockfile on the package.json from theia-apps/theia-docker (without a yarn workspace), it installs @theia/cli, so I don’t think that’s it. The Dockerfile in theia-apps/theia-docker calls the “theia build” command without any problems.

The reason for the error was because the package.json did not include name and version properties. After adding them, the yarn theia command is accessible. However, now when using a yarn workspace, the build command exits immediately with no errors, effectively doing nothing.

Here’s how you can reproduce what I’m seeing:

(note, I’m not including my extension in this example just to demonstrate how the package.json behaves within a workspace)

without yarn workspace

$ mkdir /tmp/theia-test-1 && cd /tmp/theia-test-1
$ curl https://raw.githubusercontent.com/theia-ide/theia-apps/master/theia-docker/latest.package.json -o package.json
$ yarn --pure-lockfile
$ yarn theia build

When you run the above, it successfully builds theia. However:

with yarn workspace

$ mkdir /tmp/theia-build-test && cd /tmp/theia-build-test
$ mkdir theia
$ curl https://raw.githubusercontent.com/theia-ide/theia-apps/master/theia-docker/latest.package.json -o ./theia/package.json

After downloading, edit theia/package.json to include "name": "theia" and "version": "latest". Then create the workspace root package.json:

$ echo '{"private":true, "workspaces": ["theia"]}' > package.json
$ yarn --pure-lockfile
$ yarn theia build

When you run the above, you’ll notice the yarn --pure-lockfile command finishes much more quickly than it did in the previous example. When you run yarn theia build, it seems to exit immediately with no errors and does nothing.

@bendavis for your second use-case, the build completes quickly since you are not actually building the application, the application in your case exists in theia-build-test/theia. The root package.json describes the workspace, but it is not what should be built with yarn unless you include build scripts which target the theia folder. I can’t tell accurately without looking at your setup (a reproducible example would help a lot), but that’s what I noticed about your yarn workspaces example.

I was finally able to figure it out. For anyone else coming across this, here’s how I got it to work:

  1. Use one of the theia docker images as a starting point (eg, theia-docker). We’ll call this the “docker root”.

  2. Make a subdirectory called extensions, and copy your built extension there

  3. Make a note the version number in your extension’s package.json.

  4. Add the following to the Dockerfile just before RUN yarn --pure-lockfile:

    RUN mkdir extensions
    COPY extensions/my-extension/my-extension ./extensions/my-extension
    RUN cat package.json \
        | jq '. + {"workspaces": ["extensions/*"]}' \
        | jq '.dependencies = .dependencies + { \
               "my-extension": "1.0.0" \
              }' \
        > package.json.tmp && mv package.json.tmp package.json
    

    Note that when the extension is copied to the container (within the docker file), it should copy the package that defines theiaExtensions in the package.json.

  5. Add “jq” to the RUN apk add command in the Dockerfile

  6. Run docker build -t myimagename

2 Likes

Hi @bendavis,

I was working on an example for you. I see you’ve found your own way, that’s very similar. I’ll still post it. A little tweak is that you do not need to specify your extension’s version (or be required to update it along). This works just as well:

"my-extension": "./my-extension"

See below for what I was originally going to post.


Thanks for the step-by-step instructions - very useful. Indeed my suggestion did not help.

Concentrating on the practical, I have come-up with a quick Docker example, based on the theia-docker example image, where i add an arbitrary extension in-place. I use the hello-world extension generated by the theia-generator-extension. See here:

You can replicate these changes or fetch my commit. Then just generate your own hello-world on top:

$ cd theia-apps/theia-docker
$ npm install -g yo generator-theia-extension
$ mkdir my-extension && cd my-extension
# Generate standalone extension. Will be quicker without example app
$ yo theia-extension  -s   # chose hello-world extension
# build extension - not done automatically for standalone
$ cd my-extension && yarn
$ cd ../..
$ docker build --no-cache -t theia-docker:test .
$ docker run --init -it -p 3000:3000  theia-docker:test

Connect to localhost:3000 using a web browser. Confirm the extension is present by clicking on menu item “Edit -> Say Hello”

Hope this helps,

Marc

1 Like

@marcdumais-work this is very helpful, thank you!

yeah, I meant the setup when one does not need to pull development time dependencies into the production.