Removing UI tests from the build cycle?

They are very fragile (not only tests, but underlying frameworks as well) and slow. What if we run them nightly against docker images insted in theia-ide/theia-apps? It would stabilize and speed up our build, force to decouple logic from UI, write more unit tests, detect that new next version broke final apps (docker images). We can also write more UI tests without being afraid that it harms build turnarounds.

[original thread by Anton Kosyakov]

IMHO we should still have smoke tests for UI at build time.

maybe we can rewrite them with more stable test infrastructur then, not wdio

this week too many build failures because of it

I know that Paul and I have been attempting to stabilise the tests especially AppVeyor but sadly to no avail. @florentbenoit I remember in one of our dev meetings you mentioned people on your side thinking of moving towards Cypress? Do you know if anyone is looking into it?

i’m not even talking about appveyor, travis fails as well.

i am not againts UI tests, i would be glad to have more

but maybe decouple it from dev build cycle, like test separtely against nightly next and before releasing

so we can iterate fast between releases and still catch bugs from one release to another

Also testing within Theia repo is not real end-to-end testing. Real production apps are installed differently, if something starts up in Theia repo, it won’t necessary run ouside.

Hi Anton. We noticed that we are using quite old versions of wdio and related dev-dependencies. There is a fair chance that updating them to later version(s) would stabilize the UI tests on Travis.

When they pass, the UI tests only take maybe a minute; however when they fail, we re-run the whole suite, including unit tests, up-to 3 times, and that indeed is slow.

Are you ok giving a try to updated wdio-related deps, and if it doesn’t stabilize the UI tests, then we look into more drastic alternatives? I think Vince has a WIP PR for this.

sure, let’s give it a try

@vince-fugnitto: yes they’re just starting to work on it (Use cypress)

look at failures closely in the morning, they seem to be a follow-up of: https://github.com/theia-ide/theia/issues/4035

I would try https://github.com/theia-ide/theia/issues/4035#issuecomment-455987695