Download:plugins fails with 429

Hi I am getting the following error on download:plugins
$ curl --compressed -o- -L https://yarnpkg.com/install.sh | bash
$ yarn -s run clean && yarn -s --cwd packages/java-service-editor build && yarn -s run download:plugins && yarn -s --cwd java-service-editor-app/browser prepareBuild
warning package.json: No license field
— downloading plugins —

  • vscode-builtin-extensions-pack: downloaded successfully
  • vscode.markdown-language-features: downloaded successfully
  • vscjava.vscode-java-dependency: downloaded successfully
  • vscjava.vscode-java-debug: downloaded successfully
  • redhat.java: downloaded successfully
    — collecting extension-packs —
    ‘vscode.clojure’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.coffeescript’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.cpp’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.csharp’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.docker’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.emmet’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.extension-editing’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.fsharp’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.git’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.git-ui’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.github’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.github-authentication’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.go’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.groovy’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.grunt’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.gulp’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.handlebars’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.hlsl’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.image-preview’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.jake’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.less’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.lua’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.make’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.markdown-language-features’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.microsoft-authentication’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.objective-c’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.perl’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.php’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.php-language-features’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.pug’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.r’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.razor’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.ruby’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.rust’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.sql’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.swift’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘vscode.typescript’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    ‘ms-vscode.js-debug-companion’ referenced by ‘builtin-extension-pack’ (ext pack) is excluded because of ‘theiaPluginsExcludeIds’
    — resolving 1 extension-packs —
    StatusError
    at ClientRequest. (C:\git\sag\java-service-editor\applications\java-service-editor-theia-extension\node_modules\bent\src\nodejs.js:133:23)
    at Object.onceWrapper (node:events:510:26)
    at ClientRequest.emit (node:events:390:28)
    at HTTPParser.parserOnIncomingClient (node:_http_client:623:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
    at TLSSocket.socketOnData (node:_http_client:487:22)
    at TLSSocket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:199:23) {
    statusCode: 429,
    json: [AsyncFunction (anonymous)],
    text: [Function (anonymous)],
    arrayBuffer: [Function (anonymous)],
    headers: {
    server: ‘nginx’,
    date: ‘Fri, 28 Oct 2022 15:24:50 GMT’,
    ‘content-type’: ‘application/json;charset=ISO-8859-1’,
    ‘content-length’: ‘35’,
    connection: ‘close’,
    ‘x-rate-limit-retry-after-seconds’: ‘0’,
    ‘access-control-expose-headers’: ‘X-Rate-Limit-Retry-After-Seconds, X-Rate-Limit-Remaining’,
    ‘access-control-allow-origin’: ‘*’
    }
    }
  • ms-vscode.node-debug2@1.42.7: downloaded successfully
  • vscode.shellscript@1.53.2: downloaded successfully
  • vscode.java@1.53.2: downloaded successfully
  • vscode.javascript@1.53.2: downloaded successfully
  • vscode.json@1.53.2: downloaded successfully
  • vscode.ini@1.53.2: downloaded successfully
  • vscode.theme-defaults@1.53.2: downloaded successfully
  • vscode.markdown@1.53.2: downloaded successfully
  • vscode.vscode-theme-seti@1.53.2: downloaded successfully
  • vscode.scss@1.53.2: downloaded successfully
  • vscode.theme-abyss@1.53.2: downloaded successfully
  • vscode.theme-quietlight@1.53.2: downloaded successfully
  • vscode.theme-kimbie-dark@1.53.2: downloaded successfully
  • vscode.debug-auto-launch@1.53.2: downloaded successfully
  • vscode.configuration-editing@1.53.2: downloaded successfully
  • vscode.yaml@1.53.2: downloaded successfully
  • vscode.python@1.53.2: downloaded successfully
  • vscode.css@1.53.2: downloaded successfully
  • vscode.npm@1.53.2: downloaded successfully
  • vscode.bat@1.53.2: downloaded successfully
  • vscode.debug-server-ready@1.53.2: downloaded successfully
  • vscode.powershell@1.53.2: downloaded successfully
  • vscode.search-result@1.53.2: downloaded successfully
  • vscode.log@1.53.2: downloaded successfully
  • vscode.theme-monokai@1.53.2: downloaded successfully
  • vscode.shaderlab@1.53.2: downloaded successfully
  • vscode.theme-solarized-dark@1.53.2: downloaded successfully
  • vscode.css-language-features@1.53.2: downloaded successfully
  • vscode.merge-conflict@1.53.2: downloaded successfully
  • vscode.json-language-features@1.53.2: downloaded successfully
  • vscode.theme-monokai-dimmed@1.53.2: downloaded successfully
  • vscode.theme-red@1.53.2: downloaded successfully
  • vscode.html@1.53.2: downloaded successfully
  • vscode.html-language-features@1.53.2: downloaded successfully
    error Command failed with exit code 1.

I am on theia 1.25. Here is package.json:
{

“dependencies”: {
@builtioflow/origin-shared-components”: “^10.15.110”,
@delite/dlt-components”: “1.5.1”,
@theia/bulk-edit”: “1.25.0”,
@theia/callhierarchy”: “1.25.0”,
@theia/console”: “1.25.0”,
@theia/core”: “1.25.0”,
@theia/debug”: “1.25.0”,
@theia/editor”: “1.25.0”,
@theia/editor-preview”: “1.25.0”,
@theia/external-terminal”: “1.25.0”,
@theia/file-search”: “1.25.0”,
@theia/filesystem”: “1.25.0”,
@theia/keymaps”: “1.25.0”,
@theia/markers”: “1.25.0”,
@theia/messages”: “1.25.0”,
@theia/metrics”: “1.25.0”,
@theia/monaco”: “1.25.0”,
@theia/navigator”: “1.25.0”,
@theia/outline-view”: “1.25.0”,
@theia/output”: “1.25.0”,
@theia/plugin-dev”: “1.25.0”,
@theia/plugin-ext”: “1.25.0”,
@theia/plugin-ext-vscode”: “1.25.0”,
@theia/preferences”: “1.25.0”,
@theia/process”: “1.25.0”,
@theia/property-view”: “1.25.0”,
@theia/search-in-workspace”: “1.25.0”,
@theia/task”: “1.25.0”,
@theia/terminal”: “1.25.0”,
@theia/timeline”: “1.25.0”,
@theia/toolbar”: “1.25.0”,
@theia/typehierarchy”: “1.25.0”,
@theia/userstorage”: “1.25.0”,
@theia/variable-resolver”: “1.25.0”,
@theia/vsx-registry”: “1.25.0”,
@theia/workspace”: “1.25.0”,
“buffer”: “^6.0.3”,
“compression-webpack-plugin”: “^9.0.0”,
“copy-webpack-plugin”: “^8.1.1”,
“fs-extra”: “^9.0.1”,
“https-agent”: “^1.0.0”,
“webpack”: “^5.48.0”,
“webpack-cli”: “^4.7.0”
},
“devDependencies”: {
@theia/cli”: “1.25.0”,
@types/node”: “14”,
@types/bent”: “^7.3.2”,
“bent”: “^7.3.12”,
@types/node-fetch”: “^2.6.2”,
@typescript-eslint/eslint-plugin”: “^4.25.0”,
@typescript-eslint/eslint-plugin-tslint”: “^4.25.0”,
@typescript-eslint/parser”: “^4.25.0”,
@types/jquery”: “^3.3.33”,
@types/yargs”: “17.0.7”,
“concurrently”: “^3.5.0”,
“electron”: “^15.5.7”,
“node-gyp”: “^8.4.1”,
“lerna”: “^4.0.0”,
“rimraf”: “^2.7.1”,
“typescript”: “^4.5.5”,
“yargs”: “^17.2.1”,
“ts-node”: “^10.0.0”,
“wdio-chromedriver-service”: “^6.0.4”,
“webdriverio”: “^6.10.2”
},
“scripts”: {
“prepare”: “yarn -s run clean && yarn -s --cwd packages/java-service-editor build && yarn -s run download:plugins && yarn -s --cwd java-service-editor-app/browser prepareBuild”,
“clean”: “rimraf lib src-gen”,
“build”: “theia build”,
“download:plugins”: “theia download:plugins”,
“docs”: “–tsconfig tsconfig.json --options configs/typedoc.json .”,
“watch”: “tsc -w”,
“browser”: “yarn --cwd java-service-editor-app/browser”,
“compile:tsc”: “tsc -b”,
“watch:bundle”: “theia build --watch --mode development”,
“start:browser”: “yarn --cwd java-service-editor-app/browser start”,
“rebuild:browser”: “theia rebuild:browser”
},
“theiaPluginsDir”: “plugins”,
“theiaPlugins”: {
“vscode.markdown-language-features”: “https://open-vsx.org/api/vscode/markdown-language-features/1.39.2/file/vscode.markdown-language-features-1.39.2.vsix”,
“vscode-builtin-extensions-pack”: “https://open-vsx.org/api/eclipse-theia/builtin-extension-pack/1.50.1/file/eclipse-theia.builtin-extension-pack-1.50.1.vsix”,
“redhat.java”: “https://open-vsx.org/api/redhat/java/0.73.0/file/redhat.java-0.73.0.vsix”,
“vscjava.vscode-java-debug”: “https://open-vsx.org/api/vscjava/vscode-java-debug/0.30.0/file/vscjava.vscode-java-debug-0.30.0.vsix”,
“vscjava.vscode-java-dependency”: “https://open-vsx.org/api/vscjava/vscode-java-dependency/0.16.0/file/vscjava.vscode-java-dependency-0.16.0.vsix
},
“theiaPluginsExcludeIds”: [
“vscode.extension-editing”,
“vscode.git”,
“vscode.git-ui”,
“vscode.github”,
“vscode.github-authentication”,
“vscode.markdown-language-features”,
“vscode.microsoft-authentication”,
“ms-vscode.js-debug-companion”,
“vscode.less”,
“vscode.pug”,
“vscode.handlebars”,
“vscode.emmet”,
“vscode.image-preview”,
“vscode.lua”,
“vscode.coffeescript”,
“vscode.perl”,
“vscode.razor”,
“vscode.python@”,
“vscode.fsharp”,
“scode.shaderlab”,
“vscode.groovy”,
“vscode.cpp”,
“vscode.typescript”,
“vscode.docker”,
“vscode.make”,
“vscode.ruby”,
“vscode.clojure”,
“vscode.csharp”,
“vscode.jake”,
“vscode.go”,
“vscode.rust”,
“vscode.r”,
“vscode.objective-c”,
“vscode.php”,
“vscode.php-language-features”,
“vscode.swift”,
“vscode.sql”,
“vscode.hlsl”,
“vscode.gulp”,
“vscode.grunt”
],
“workspaces”: [
“packages/",
"java-service-editor-app/

],
“repository”: “Initiating SAML single sign-on”,
“license”: “see license in ‘LICENSE’ file”,
“engines”: {
“yarn”: “>=1.7.0 <2.x.x”,
“node”: “>=12.14.1”
},

}

  • List item

@abc thank you for the discussion, it’s an issue with open-vsx itself when it experiences a high load.

Thankyou! How can I work around this? My build is failing? ignore-errors does’t seem to work in @theia/cli/1.25.0

How can I work around this? My build is failing? ignore-errors does’t seem to work in @theia/cli/1.25.0

@abc if the service is not back yet you can:

  • execute yarn download:plugins --ignore-errors (the flag was added in 1.9.0)
  • not call yarn download:plugins at all

Hi,

the theia-trace-extension github verification builds for PRs have currently the same issue and it doesn’t seem to clear up by itself. Multiple PRs fail to build because of that consistently for roughly a day, e.g. Add trace name to opening trace dialog box · eclipse-cdt-cloud/theia-trace-extension@e697f0a · GitHub. It works locally on my laptop.

The suggestion to add --ignore-errors might work, but the example app won’t be build correctly (i.e. missing vscode plugins). I don’t think it’s a viable solution for the PR verification build.

Could you please look into this issue?

Thanks

@bhufmann if the theia-trace-extension does not test against the plugins it downloads then during CI it might be fine to include --ignore-errors.

The 429 errors are due to a recent change in open-vsx where they now added rate limiting for requests, we may need to improve the download:script to perform requests in batches and/or with a delay as to not hit the limit. These changes are still undocumented for the most part.

@vince-fugnitto thanks for the answers. --ingore-errors would be ok during CI, but some users are using the example app and for them it would not be ok.

Builds are working again after multiple tries. I’ll follow the bug tracker and hope the rate limit problem is address before we run into the issue again.

It was working for a short while, but now it’s failing again. I’m sure our project is not the only that has that issue. @vince-fugnitto any idea when this issue can be fixed?

Hi Bernd,

We are not sure yet of the details, but it’s likely that the new openvsx limits are shared between multiple users/builds when behind a proxy, since all requests will seem to come from the same IP address. In such a case, there may not be much to be done from the client-side.

We’ve had a similar issue with hitting GitHub API rate limits, downloading the ripgrep binary during Theia builds. The solution in that case was to authenticate to GitHub using a Personal Access Token during the build - when done that way, the limits were accounted for the user rather than the IP address.

Openvsx has authentication and tokens but currently they are for publishing and require signing the publishers agreement. It’s also not clear whether using a token would give one its own rate limits.

I asked a couples of questions on the related openvsx issue that Vince linked above:

–ingore-errors did not work with the version of theia we were using. To work around, we have split our build task. We down plugins as a separate gradle task with ignoreExitValue true flag

Vince is experimenting with the plugin download script and so far has success when making the downloads sequential, through a CLI option (that was already present). You can probably replicate easily in your projects, tweaking the way you invoke the script:

In short:
yarn -s download:plugins --parallel=false

Thanks for this suggestion. I’ll try it out:

PR for that seems to build successfully the first time:

1 Like