How to add e2e tests for the electron-app in development mode?

Hello:

I’m trying to add some UI test(e2e test) to my Theia application and I found the basic-example.espec.ts of Theia:

// *****************************************************************************
// Copyright (C) 2017 Ericsson and others.
//
// This program and the accompanying materials are made available under the
// terms of the Eclipse Public License v. 2.0 which is available at
// http://www.eclipse.org/legal/epl-2.0.
//
// This Source Code may also be made available under the following Secondary
// Licenses when the conditions for such availability set forth in the Eclipse
// Public License v. 2.0 are satisfied: GNU General Public License, version 2
// with the GNU Classpath Exception which is available at
// https://www.gnu.org/software/classpath/license.html.
//
// SPDX-License-Identifier: EPL-2.0 OR GPL-2.0 WITH Classpath-exception-2.0
// *****************************************************************************

import * as chai from 'chai';
import * as path from 'path';
import { app, BrowserWindow } from 'electron';

const expect = chai.expect;

describe.skip('basic-example-spec', () => {

    const mainWindow: Electron.BrowserWindow = new BrowserWindow({ show: false });
    mainWindow.on('ready-to-show', () => mainWindow.show());

    describe('01 #start example app', () => {
        it('should start the electron example app', async () => {
            if (!app.isReady()) {
                await new Promise(resolve => app.on('ready', resolve));
            }

            require('../src-gen/backend/main'); // start the express server

            mainWindow.webContents.openDevTools();
            mainWindow.loadURL(`file://${path.join(__dirname, 'index.html')}`);

            // eslint-disable-next-line no-unused-expressions
            expect(mainWindow.isVisible()).to.be.true;
        });
    });
});

But it looks like a demo without a concrete implementation since it use describe.skip(). Even if I modified the describe.skip() to describe(), paths ../src-gen/backend/main and file://${path.join(__dirname, 'index.html')} need to be modified to work properly.

Then I found the app.spec.js of Theia-blueprint:

const os = require("os");
const path = require("path");
const { remote } = require("webdriverio");
const { expect } = require("chai");

const THEIA_LOAD_TIMEOUT = 15000; // 15 seconds

function getBinaryPath() {
  const distFolder = path.join(__dirname, "..", "dist");
  switch (os.platform()) {
    case "linux":
      return path.join(
        distFolder,
        "linux-unpacked",
        "theia-blueprint"
      );
    case "win32":
      return path.join(
        distFolder,
        "win-unpacked",
        "TheiaBlueprint.exe"
      );
    case "darwin":
      return path.join(
        distFolder,
        "mac",
        "TheiaBlueprint.app",
        "Contents",
        "MacOS",
        "TheiaBlueprint"
      );
    default:
      return undefined;
  }
};

// Utility for keyboard shortcuts that execute commands where
// the key combination is the same on all platforms *except that*
// the Command key is used instead of Control on MacOS. Note that
// sometimes MacOS also uses Control. This is not handled, here
function macSafeKeyCombo(keys) {
  if (os.platform() === "darwin" && keys.includes("Control")) {
    // Puppeteer calls the Command key "Meta"
    return keys.map((k) => k === "Control" ? "Meta" : k);
  }
  return keys;
};

describe("Theia App", function() {
  // In mocha, 'this' is a common context between sibling beforeEach, afterEach, it, etc methods within the same describe.
  // Each describe has its own context.
  beforeEach(async function() {
    const binary = getBinaryPath();
    if (!binary) {
      throw new Error("Tests are not supported for this platform.")
    }

    // Start app and store connection in context (this)
    this.browser = await remote({
      // Change to info to get detailed events of webdriverio
      logLevel: "info",
      capabilities: {
        browserName: "chrome",
        "goog:chromeOptions": {
          // Path to built and packaged theia
          binary: binary,
          // Hand in workspace to load as runtime parameter
          args: [path.join(__dirname, "workspace")],
        },
      },
    });

    const appShell = await this.browser.$("#theia-app-shell");

    // mocha waits for returned promise to resolve
    // Theia is loaded once the app shell is present
    return appShell.waitForExist({
      timeout: THEIA_LOAD_TIMEOUT,
      timeoutMsg: "Theia took too long to load.",
    });
  });

  afterEach(async function() {
    try {
      await this.browser.closeWindow();
    } catch (err) {
      // Workaround: Puppeteer cannot properly connect to electron and throws an error.
      // However, the window is closed and that's all we want here.
      if (`${err}`.includes("Protocol error (Target.createTarget)")) {
        return;
      }
      // Rethrow for unexpected errors to fail test.
      throw err;
    }
  });

  it("Correct window title", async function() {
    const windowTitle = await this.browser.getTitle();
    expect(windowTitle).to.include("workspace — Theia");
  });

  it("Builtin extensions", async function() {
    // Wait a bit to make sure key handlers are registered.
    await new Promise((r) => setTimeout(r, 2000));

    // Open extensions view
    await this.browser.keys(macSafeKeyCombo(["Control", "Shift", "x"]));
    const builtinContainer = await this.browser.$(
      "#vsx-extensions-view-container--vsx-extensions\\:builtin"
    );

    // Expand builtin extensions
    const builtinHeader = await builtinContainer.$(".theia-header.header");
    await builtinHeader.click();

    // Wait for expansion to finish
    const builtin = await this.browser.$(
      "#vsx-extensions\\:builtin .theia-TreeContainer"
    );
    await builtin.waitForExist();

    // Get names of all builtin extensions
    const extensions = await builtin.$$(".theia-vsx-extension .name");
    const extensionNames = await Promise.all(
      extensions.map((e) => e.getText())
    );

    // Exemplary check a few extensions
    expect(extensionNames).to.include("Debugger for Java");
    expect(extensionNames).to.include("TypeScript Language Basics (built-in)");
  });
});

It uses the webdriverio but the getBinaryPath function means that the test can only work in production mode and each time I made a change to the source code, I have to package the application before run the test.

Is there any way to implement e2e tests in development mode? :thinking:

Hi @inlann,

We recently added a playwright Theia testing framework. I believe it’s fairly straightforward to use for browser apps, but I’m not sure how easy it is to setup for Electron. @JonasHelming can you provide more info?

@planger

I haven’t tried it myself yet but there is experimental support for Electron in Playwright. If that works with a Theia electron app too, it should be possible to use the page object model of Theia also with this way of starting the app (see Playwright documentation below).

However, there are a few page objects that will likely not work out of the box, such as the main menu and the context menu, as those work differently between Electron and Browser. For those features, dedicated page objects for electron would be needed. As other page objects depend on the main menu and context menu page object, the support for Electron of the Theia Playwright page object model will eventually be rather limited in the current state, I’m afraid.

That’s really helpful :blush: I’ll try it!

Hello @planger

I cloned the Theia and ran the script in root directory: yarn test:playwright but some test cases seem to fail (Due to character limitations, I have omitted the output of some successful test case.):

mac@Titan theia-master % yarn test:playwright
yarn run v1.22.10
$ concurrently --success first -c gray,blue -k -n theia,playwright "yarn browser start" "yarn -s --cwd examples/playwright ui-tests-ci"
$ yarn -s --cwd examples/browser start
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ yarn clean && yarn build
$ rimraf lib
[theia] native node modules are already rebuilt for browser
$ tsc --incremental && yarn lint && npx playwright install chromium
[theia] Configuring to accept webviews on '^.+\.webview\..+$' hostname.
...
[theia] root INFO Deploy plugins list: 250.8 ms [Finished 2.738 s after backend start]
$ eslint -c ./.eslintrc.js --ext .ts ./src ./tests
[playwright] 
[playwright] Running 48 tests using 1 worker
[playwright] 
  ✘  ../../tests/theia-app.test.ts:26:9 › Theia Application › should load (1m)
  ✘  ../../tests/theia-app.test.ts:26:9 › Theia Application › should load (1m)
  ✘  ../../tests/theia-app.test.ts:30:9 › Theia Application › should show main content panel (144ms)
  ✘  ../../tests/theia-app.test.ts:30:9 › Theia Application › should show main content panel (141ms)
  ✘  ../../tests/theia-explorer-view.test.ts:34:9 › Theia Explorer View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-explorer-view.test.ts:34:9 › Theia Explorer View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-main-menu.test.ts:31:9 › Theia Main Menu › should show the main menu bar (0ms)
...
  ✘  ../../tests/theia-main-menu.test.ts:31:9 › Theia Main Menu › should show the main menu bar (0ms)
...
  ✘  ../../tests/theia-preference-view.test.ts:30:9 › Preference View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-preference-view.test.ts:30:9 › Preference View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-problems-view.test.ts:30:9 › Theia Problems View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-problems-view.test.ts:30:9 › Theia Problems View › should be visible and active after being opened (0ms)
...
  ✘  ../../tests/theia-quick-command.test.ts:34:9 › Theia Quick Command › should show quick command palette (0ms)
...
  ✘  ../../tests/theia-quick-command.test.ts:34:9 › Theia Quick Command › should show quick command palette (0ms)
...
  ✘  ../../tests/theia-status-bar.test.ts:34:9 › Theia Status Bar › should show status bar (0ms)
...

  ✘  ../../tests/theia-status-bar.test.ts:34:9 › Theia Status Bar › should show status bar (0ms)
...
  ✘  ../../tests/theia-text-editor.test.ts:32:9 › Theia Text Editor › should be visible and active after opening "sample.txt" (0ms)
...
  ✘  ../../tests/theia-text-editor.test.ts:32:9 › Theia Text Editor › should be visible and active after opening "sample.txt" (0ms)
...
  ✘  ../../tests/theia-workspace.test.ts:25:9 › Theia Workspace › should be initialized empty by default (1m)
  ✘  ../../tests/theia-workspace.test.ts:25:9 › Theia Workspace › should be initialized empty by default (1m)
  ✘  ../../tests/theia-workspace.test.ts:32:9 › Theia Workspace › should be initialized with the contents of a file location (1m)
  ✘  ../../tests/theia-workspace.test.ts:32:9 › Theia Workspace › should be initialized with the contents of a file location (1m)
  ✘  ../../tests/theia-workspace.test.ts:41:9 › Theia Workspace › should be initialized with the contents of multiple file locations (1m)
  ✘  ../../tests/theia-workspace.test.ts:41:9 › Theia Workspace › should be initialized with the contents of multiple file locations (1m)
[playwright] 
[playwright] 
[playwright]   1) ../../tests/theia-app.test.ts:26:9 › Theia Application › should load ==========================
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       25 |
[playwright]       26 |     test('should load', async () => {
[playwright]     > 27 |         app = await TheiaApp.load(page);
[playwright]          |               ^
[playwright]       28 |     });
[playwright]       29 |
[playwright]       30 |     test('should show main content panel', async () => {
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-app.test.ts:27:15
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-app-Theia-Application-should-load/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       25 |
[playwright]       26 |     test('should load', async () => {
[playwright]     > 27 |         app = await TheiaApp.load(page);
[playwright]          |               ^
[playwright]       28 |     });
[playwright]       29 |
[playwright]       30 |     test('should show main content panel', async () => {
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-app.test.ts:27:15
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-app-Theia-Application-should-load-retry1/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]   2) ../../tests/theia-app.test.ts:30:9 › Theia Application › should show main content panel =======
[playwright] 
[playwright]     TypeError: Cannot read property 'isMainContentPanelVisible' of undefined
[playwright] 
[playwright]       29 |
[playwright]       30 |     test('should show main content panel', async () => {
[playwright]     > 31 |         expect(await app.isMainContentPanelVisible()).toBe(true);
[playwright]          |                          ^
[playwright]       32 |     });
[playwright]       33 |
[playwright]       34 | });
[playwright] 
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-app.test.ts:31:26
[playwright]         at FixtureRunner.resolveParametersAndRunHookOrTest (/Users/mac/Work/theia-master/node_modules/@playwright/test/lib/fixtures.js:306:12)
[playwright]         at WorkerRunner._runTestWithBeforeHooks (/Users/mac/Work/theia-master/node_modules/@playwright/test/lib/workerRunner.js:499:7)
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-app-Theia-Application-should-show-main-content-panel/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     TypeError: Cannot read property 'isMainContentPanelVisible' of undefined
[playwright] 
[playwright]       29 |
[playwright]       30 |     test('should show main content panel', async () => {
[playwright]     > 31 |         expect(await app.isMainContentPanelVisible()).toBe(true);
[playwright]          |                          ^
[playwright]       32 |     });
[playwright]       33 |
[playwright]       34 | });
[playwright] 
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-app.test.ts:31:26
[playwright]         at FixtureRunner.resolveParametersAndRunHookOrTest (/Users/mac/Work/theia-master/node_modules/@playwright/test/lib/fixtures.js:306:12)
[playwright]         at WorkerRunner._runTestWithBeforeHooks (/Users/mac/Work/theia-master/node_modules/@playwright/test/lib/workerRunner.js:499:7)
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-app-Theia-Application-should-show-main-content-panel-retry1/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]   3) ../../tests/theia-explorer-view.test.ts:34:9 › Theia Explorer View › should be visible and active after being opened 
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   4) ../../tests/theia-main-menu.test.ts:31:9 › Theia Main Menu › should show the main menu bar ====
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   5) ../../tests/theia-preference-view.test.ts:30:9 › Preference View › should be visible and active after being opened 
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   6) ../../tests/theia-problems-view.test.ts:30:9 › Theia Problems View › should be visible and active after being opened 
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   7) ../../tests/theia-quick-command.test.ts:34:9 › Theia Quick Command › should show quick command palette 
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   8) ../../tests/theia-status-bar.test.ts:34:9 › Theia Status Bar › should show status bar =========
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   9) ../../tests/theia-text-editor.test.ts:32:9 › Theia Text Editor › should be visible and active after opening "sample.txt" 
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded in beforeAll hook.
[playwright] 
[playwright]   10) ../../tests/theia-workspace.test.ts:25:9 › Theia Workspace › should be initialized empty by default 
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       24 |
[playwright]       25 |     test('should be initialized empty by default', async () => {
[playwright]     > 26 |         const app = await TheiaApp.load(page);
[playwright]          |                     ^
[playwright]       27 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       28 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       29 |         expect(fileStatElements.length).toBe(0);
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:26:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-empty-by-default/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       24 |
[playwright]       25 |     test('should be initialized empty by default', async () => {
[playwright]     > 26 |         const app = await TheiaApp.load(page);
[playwright]          |                     ^
[playwright]       27 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       28 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       29 |         expect(fileStatElements.length).toBe(0);
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:26:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-empty-by-default-retry1/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]   11) ../../tests/theia-workspace.test.ts:32:9 › Theia Workspace › should be initialized with the contents of a file location 
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       32 |     test('should be initialized with the contents of a file location', async () => {
[playwright]       33 |         const ws = new TheiaWorkspace(['tests/resources/sample-files1']);
[playwright]     > 34 |         const app = await TheiaApp.load(page, ws);
[playwright]          |                     ^
[playwright]       35 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       36 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       37 |         // resources/sample-files1 contains one folder and one file
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:34:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-with-the-contents-of-a-file-location/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       32 |     test('should be initialized with the contents of a file location', async () => {
[playwright]       33 |         const ws = new TheiaWorkspace(['tests/resources/sample-files1']);
[playwright]     > 34 |         const app = await TheiaApp.load(page, ws);
[playwright]          |                     ^
[playwright]       35 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       36 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       37 |         // resources/sample-files1 contains one folder and one file
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:34:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-with-the-contents-of-a-file-location-retry1/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]   12) ../../tests/theia-workspace.test.ts:41:9 › Theia Workspace › should be initialized with the contents of multiple file locations 
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       41 |     test('should be initialized with the contents of multiple file locations', async () => {
[playwright]       42 |         const ws = new TheiaWorkspace(['tests/resources/sample-files1', 'tests/resources/sample-files2']);
[playwright]     > 43 |         const app = await TheiaApp.load(page, ws);
[playwright]          |                     ^
[playwright]       44 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       45 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       46 |         // resources/sample-files1 contains one folder and one file
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:43:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-with-the-contents-of-multiple-file-locations/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
[playwright]     Retry #1 ---------------------------------------------------------------------------------------
[playwright] 
[playwright]     Timeout of 60000ms exceeded.
[playwright] 
[playwright]     page.waitForSelector: Target closed
[playwright]     =========================== logs ===========================
[playwright]     waiting for selector ".theia-ApplicationShell" to be visible
[playwright]     ============================================================
[playwright] 
[playwright]       41 |     test('should be initialized with the contents of multiple file locations', async () => {
[playwright]       42 |         const ws = new TheiaWorkspace(['tests/resources/sample-files1', 'tests/resources/sample-files2']);
[playwright]     > 43 |         const app = await TheiaApp.load(page, ws);
[playwright]          |                     ^
[playwright]       44 |         const explorer = await app.openView(TheiaExplorerView);
[playwright]       45 |         const fileStatElements = await explorer.visibleFileStatNodes(DOT_FILES_FILTER);
[playwright]       46 |         // resources/sample-files1 contains one folder and one file
[playwright] 
[playwright]         at Function.load (/Users/mac/Work/theia-master/examples/playwright/src/theia-app.ts:38:20)
[playwright]         at /Users/mac/Work/theia-master/examples/playwright/tests/theia-workspace.test.ts:43:21
[playwright] 
[playwright]     attachment #1: screenshot (image/png) ----------------------------------------------------------
[playwright]     test-results/theia-workspace-Theia-Workspace-should-be-initialized-with-the-contents-of-multiple-file-locations-retry1/test-failed-1.png
[playwright]     ------------------------------------------------------------------------------------------------
[playwright] 
...
[playwright] 
[playwright]   Slow test file: ../../tests/theia-workspace.test.ts (6m)
[playwright]   Slow test file: ../../tests/theia-app.test.ts (2m)
[playwright]   Consider splitting slow test files to speed up parallel execution
[playwright] 
[playwright]   12 failed
[playwright]     ../../tests/theia-app.test.ts:26:9 › Theia Application › should load ===========================
[playwright]     ../../tests/theia-app.test.ts:30:9 › Theia Application › should show main content panel ========
[playwright]     ../../tests/theia-explorer-view.test.ts:34:9 › Theia Explorer View › should be visible and active after being opened 
[playwright]     ../../tests/theia-main-menu.test.ts:31:9 › Theia Main Menu › should show the main menu bar =====
[playwright]     ../../tests/theia-preference-view.test.ts:30:9 › Preference View › should be visible and active after being opened 
[playwright]     ../../tests/theia-problems-view.test.ts:30:9 › Theia Problems View › should be visible and active after being opened 
[playwright]     ../../tests/theia-quick-command.test.ts:34:9 › Theia Quick Command › should show quick command palette 
[playwright]     ../../tests/theia-status-bar.test.ts:34:9 › Theia Status Bar › should show status bar ==========
[playwright]     ../../tests/theia-text-editor.test.ts:32:9 › Theia Text Editor › should be visible and active after opening "sample.txt" 
[playwright]     ../../tests/theia-workspace.test.ts:25:9 › Theia Workspace › should be initialized empty by default 
[playwright]     ../../tests/theia-workspace.test.ts:32:9 › Theia Workspace › should be initialized with the contents of a file location 
[playwright]     ../../tests/theia-workspace.test.ts:41:9 › Theia Workspace › should be initialized with the contents of multiple file locations 
[playwright]   36 skipped
[playwright] yarn -s --cwd examples/playwright ui-tests-ci exited with code 1
--> Sending SIGTERM to other processes..
[theia] yarn browser start exited with code 1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

The script test:playwright executes concurrently --success first -c gray,blue -k -n theia,playwright "yarn browser start" "yarn -s --cwd examples/playwright ui-tests-ci" so I don’t need to start a browser application myself.

What can I do to fix failed test cases?

Hi @inlann,

hm, there is something fundamental that is not working, because the tests Theia Application > should load, etc. are basically just checking whether the workbench comes up.

If you just run yarn browser start and visit localhost:3000, does everything look fine?
Do the tests pass if you run yarn -s --cwd examples/playwright ui-tests-ci, once you verified in the browser everything works?

Which OS do you use? There are known issues on MacOS, which we are currently working on.

Hi @planger

I’m using MacOS. You are right. My yarn browser start only shows a Cannot GET /. :cry:

The Cannot GET / error exists in 1.23.0 release version downloaded from Github

Hello,
How I can use this theia-playwright framework https://github.com/eclipse-theia/theia/tree/master/examples/playwright in our Theia app to launch the electron application?
Thanks in advance.

Hello @AJ
Thanks for your reply :smile: Are you using Theia 1.23.0 since your link is the latest version?

Hello @inlann ,

Thanks for the reply. we are using Theia version 1.20.0. and still now not supporting the browser version. OS ubuntu 18.04

Running all the tests in Theia own repository are successful for me by following the instructions here: https://github.com/eclipse-theia/theia/blob/master/examples/playwright/docs/GETTING_STARTED.md

But I copied the src and test folder inside our repository and tried to run only the https://github.com/eclipse-theia/theia/blob/master/examples/playwright/tests/theia-app.test.ts

It fails on below line:

const app = await TheiaApp.load(page, ws);

and pointing to this source:

    at Function.loadOrReload (src/theia-app.ts:48:24)
    at Function.load (src/theia-app.ts:36:9)

seems like it’s looking for a valid url.

    await page.goto(url);

Is this failing as we still don’t support browser version of our app or there is something different that i can do to launch the electron app?

Thanks in advance

Any insightful information that can help me ?
Thanks

@AJ I believe as the following comment mentioned, electron support for the playwright tests is still experimental, and some page objects may fail.

is something different that i can do to launch the electron app

The documentation from https://playwright.dev/docs/api/class-electron mentions how to launch if required:

cc @planger @JonasHelming

I haven’t tried it myself, but I think you’d need to provide a dedicated page object for an Electron app instead of using the TheiaApp page object as is for Electron, so that it’ll invoke electron.launch() and not page.goto(). Once that works, there’ll be more things that require adjustment, such as context menu and main menu. However, I’m not sure whether Playwright already provides an API for interacting with those in Electron.

Thanks all for the hints.

I was able to get playwright test to run using one of the example test module (theia-main-menu.test.ts file).

I didn’t have to create a new theia-app.ts, but I do have to comment out the line in theia-app.ts to not call “await this.loadOrReload(this.page, ‘/#’ + this.workspace.urlEncodedPath)”. Ideally, there should be a separate TheiaApp class for electron or make the existing TheiaApp class to handle both electron and browser.

In the theia-main-menu.test.s beforeAll function, I replaced the code with the following:

const electronApp = await electron.launch({
    args: ['./'],
    cwd: '/<path to theia>/examples/electron'
 });
 const app = await TheiaApp.load(await electronApp.firstWindow());
 menuBar = app.menuBar;

Further more, if you want to open a view. The theia-quick-command-palette does not open for me when sending the shortcut, this is on linux. I have to modify the theia-view.ts file open method to use the View -> Open View menu.

Here is the modified code:

async open(): Promise<TheiaView> {
    if (!this.data.viewName) {
        throw new Error('View name must be specified to open via command palette');
    }

    // workaround [start] - control+shift+p doesn't open the command palette, use menu instead.
    const mainMenu = await this.app.menuBar.openMenu('View');
    const viewMenu = await mainMenu.menuItemByNamePath('Open View');
    await viewMenu?.click();
    await this.app.quickCommandPalette.trigger(this.data.viewName);

    // await this.app.quickCommandPalette.type('View: Open View');
    // await this.app.quickCommandPalette.trigger('View: Open View...', this.data.viewName);
    // workaround [end]

    await this.waitForVisible();
    return this;
}

Any update on this ? i’m not able to launch electron or run tests on it