Semantic Highlighting not working on Theia

Hi,

I’ve developed a VS Code language extension that allows to write code based on a custom DSL. The extension communicates with an Xtext LSP and is capable of syntax highlighting as well as semantic highlighting.

When a install the extension in a Theia instance, it works fine for syntax highlighting and all the other LSP communication (prediction, renaming, etc…) however the semantic tokens are not highlighted.

By taking a look to the client/server communication between the extension and the LSP in Theia, I cannot see requests of type textDocument/semanticTokens/. This does not happen in VS Code and the tokens are correctly highlighted with the corresponding theme color.

Am I missing something to be added configured in my Theia instance? Is perhaps the semantic highlighting not yet implemented in Theia?

Many thanks in advance.

@agonzalezgaliana thank you for the discussion :+1:
I believe semantic highlighting support was added as part of https://github.com/eclipse-theia/theia/pull/8593.
I’ll give a try with https://github.com/microsoft/vscode-extension-samples/tree/main/semantic-tokens-sample and report back.

Update: I tried the sample and it worked for me (although the custom theming did not as it is not yet supported):

editor.semanticHighlighting.enabled = false

editor.semanticHighlighting.enabled = true

vscode with the extra theming options:

"editor.semanticTokenColorCustomizations": {
  "enabled": true, // enable for all themes
  "rules": {
    "*.static": {
      "foreground": "#ff0000",
      "fontStyle": "bold"
    },
    "type": {
      "foreground": "#00aa00"
    }
  }
}

Thanks Vince for your reply. The VS Code example actually uses a VS Code semantic provider (vscode.DocumentSemanticTokensProvider) which returns the semantic tokens. My example does not use them and delegates this task to the LSP by sending the request textDocument/semanticTokens/. The LSP then responds returning the semantic tokens. In Theia, the request textDocument/semanticTokens/ is not even triggered. I think that the VS Code language client is the one responsible to send the request to the LSP only when no token provider is provided.

@agonzalezgaliana I see, I believe it should work but it is possible the initial pull-request was tested using DocumentSemanticTokensProvider examples. There is the following issue which provides additional information: https://github.com/eclipse-theia/theia/issues/9058#issuecomment-797808664.

I’m not sure if the problem is in the editor (unless is the responsible of triggering a semantic token request to the LSP server).

The following are snippets of the Xtext language server log comparison:

Here you can see how the server receives a request from the VS Code client whereas not from the Theia client.

Do I maybe miss to add an extension to my Theia instance? Currently the package.json looks like the following:

    {
    "private": true,
    "dependencies": {
      "@theia/callhierarchy": "1.18.0",
      "@theia/file-search": "1.18.0",
      "@theia/git": "1.18.0",
      "@theia/markers": "1.18.0",
      "@theia/messages": "1.18.0",
      "@theia/navigator": "1.18.0",
      "@theia/outline-view": "1.18.0",
      "@theia/plugin-ext": "1.18.0",
      "@theia/plugin-ext-vscode": "1.18.0",
      "@theia/preferences": "1.18.0",
      "@theia/preview": "1.18.0",
      "@theia/search-in-workspace": "1.18.0",
      "@theia/terminal": "1.18.0",
      "@theia/vsx-registry": "1.18.0",
	  "@theia/monaco": "1.18.0"
    },
    "devDependencies": {
      "@theia/cli": "1.18.0"
    },
    "scripts": {
      "prepare": "yarn run clean && yarn build && yarn run download:plugins",
      "clean": "theia clean",
      "build": "theia build --mode development",
      "start": "theia start --plugins=local-dir:plugins",
      "download:plugins": "theia download:plugins"
    },
    "theiaPluginsDir": "plugins",
    "theiaPlugins": {
      "vscode-builtin-extensions-pack": "https://open-vsx.org/api/eclipse-theia/builtin-extension-pack/1.61.0/file/eclipse-theia.builtin-extension-pack-1.61.0.vsix"
    },
    "theiaPluginsExcludeIds": [
      "vscode.extension-editing",
      "vscode.git",
      "vscode.git-ui",
      "vscode.github",
      "vscode.markdown-language-features",
      "vscode.microsoft-authentication"
    ]
  }