Commit graph

40 commits

Author SHA1 Message Date
Christof Thalhofer
2d2c545576 gb.test: Assert documentation
[GB.TEST]
* New: Assert is an instruction which is extended by gb.test. It's original functionality is now documented in the Assert.module.
2020-05-11 22:31:08 +02:00
Christof Thalhofer
77436fd06d gb.test Setup and Teardown methods hidden 2020-05-11 22:26:08 +02:00
gambas
492dccc28f Make the component visible again.
[GB.TEST]
* NEW: Make the component visible again.
* BUG: Fix two typos in comments.
2020-05-11 21:02:39 +02:00
Tobias Boege
e3a9ac58e8 gb.test: Exit with appropriate status code
[GB.TEST]
* BUG: Test.Main() now `Quit`s with 0 or 1 indicating whether tests passed or failed.
2020-05-07 13:15:54 +02:00
Tobias Boege
a965afa0e5 gb.test: Hide more classes and symbols
[GB.TEST]
* OPT: Hide more classes and symbols.

TapPrinter, TapParser and TapContext are not exported anymore. The Tap
constants are moved into other classes. Some public methods/properties
that expose printer, parser or context are hidden.
2020-05-07 12:57:55 +02:00
Christof Thalhofer
08d798c186 hide Test.Finish and documentation 2020-05-02 18:35:48 +02:00
Christof Thalhofer
2c111a14c3 Assert documentation – not fully done
Left descrition of Approximate and RelativeApproximate to a mathematician.
;-)
2020-05-02 17:20:42 +02:00
Christof Thalhofer
91c4ce2638 started documentation and task hide symbols 2020-05-02 11:53:30 +02:00
Christof Thalhofer
3cf670ef85 Test.PrintSummary: Result in the last line 2020-05-01 18:59:20 +02:00
Tobias Boege
4c2a4f47f5 Show TODOs and SKIPs in summary
[GB.TEST]
* BUG: Do not trim whitespace from diagnostic lines.
* NEW: Show TODOs (failed and bonus) and SKIPs in summary.

Previously, TapPrinter.Diagnostic would Trim$ its argument before
splitting it by line. This would remove any indentation that may
be useful for formatted diagnostics output, so it has to go.
Additionally I special-case Null and gb.Lf to print exactly "#\n",
which is a common use case as a visual separator.
2020-05-01 15:57:53 +02:00
Christof Thalhofer
c306e9a5fc Fix: Free Assert from non-assertions
Assert free from non-assertions. Also Assert and Test now are
modules.
2020-04-27 12:01:47 +02:00
Christof Thalhofer
d7a478cc81 Fix: Test bails out if testmodule doesn't exist 2020-04-26 11:23:01 +02:00
Christof Thalhofer
7d55499305 Better readability of TAP output for humans
... I mean elder humans, like me
2020-04-26 10:44:59 +02:00
Christof Thalhofer
e5c9b3c1c2 gb.test and gbt3 some refactoring and future tasks 2020-04-25 21:28:51 +02:00
Tobias Boege
35cac9053f Refactor TapPrinter and Assert
This is a big refactor of the TapPrinter and Assert interaction with
the goal of enabling gb.test to print a self-summary after a TAP session
(without storing the TAP we printed and parsing it afterwards). The main
feature addition is a new class representing an assertion (together with
its subtests). Objects of this type are now produced by TapPrinter when
it prints an assertion and by TapParser alike when it parses TAP.
This allows more uniform and more detailed reporting of failures, too.

For organizational reasons, subtest support was moved into the printer.

[GB.TEST]
* NEW: Expose the TapContext class which holds the current TAP session's state.
* NEW: Represent assertions (and entire subtests) by TestAssertion objects in both TapPrinter and TapParser.
* NEW: Coalesce the four TapParser events for assertions into one.
* OPT: Move subtest support into TapPrinter, removing the need to create multiple TapPrinter objects.
* OPT: Remove the need for buffering the TAP to print a self-summary.
* BUG: Fix the buffering issue that would lose TAP when the process crashes.
2020-04-25 05:45:42 +02:00
Christof Thalhofer
294194557b gbtest.Test FIXME comment 2020-04-10 13:52:56 +02:00
Christof Thalhofer
254251b2f7 refact 2020-04-10 11:15:17 +02:00
Christof Thalhofer
72c2c1e158 TestFailures report ok now, new TestCrashes
Tests in TestCrashes fail and must be reported as failed
2020-04-09 08:13:09 +02:00
Christof Thalhofer
815a1ddcf2 TestCommand parsing done, but system still buggy
Without "Debug oTest.Name" in TestSuite.class:39 it would silently crash anywhere
after TestNoMessage is done without printing any single character. Reason is
"Assert.BailOut" in TestSuite:51 which miraculously does nothing.

If the testsystem itself crashes, ist must print "BailOut ..."

But here also can be seen, why the system has to print every assertion immediately.
Any bufferíng is bad because if the testsystem crashes the buffered results are gone.

So we must get rid of the buffered stream thing and go back to Print scattered all over ;-)
2020-04-08 12:39:55 +02:00
Christof Thalhofer
ce56c1ddb4 New signature of Test.Main(), not really functional
I'm goin to bed, just to backup my work to Gitlab
2020-04-08 00:46:09 +02:00
Christof Thalhofer
33641cac1b TestCommand Parser
Parses the string that containes testmodules and -methods. The parser is tested in TestInternals.
2020-04-08 00:40:14 +02:00
Christof Thalhofer
728194b3da Rename Unittest to Test 2020-04-07 16:48:46 +02:00
Tobias Boege
84e9792043 gb.test: remove self-test concept
[GB.TEST]
* OPT: Remove the concept of self-tests. Since ".../.test" exists there is no need to distinguish their tests and ours anymore.
2020-04-07 15:08:54 +02:00
Tobias Boege
6144a78829 gb.test: fix Assert.ErrorCode and Teardown spelling
[GB.TEST]
* BUG: Forward Description in Assert.ErrorCode.
* BUG: Fix spelling of TearDown -> Teardown special test method.
* OPT: Use Assert.Note() instead of Print in tests.
2020-04-07 15:08:54 +02:00
Tobias Boege
2489001ca4 gb.test: restore failing self-tests
[GB.TEST]
* NEW: Errors in tests are caught again.
* OPT: Shorten StringEquals diagnostics.
2020-04-07 15:08:46 +02:00
Tobias Boege
548d7da1cb gb.test: Remove ITest, ITestCase leftovers
[GB.TEST]
* OPT: Remove the inheritance hierarchy of ITest and ITestCase.

They are no longer needed to identify tests.
2020-02-27 20:58:27 +01:00
Tobias Boege
350d61cd85 gb.test: Restore BailOut on uncaught errors
[GB.TEST.TAP]
* NEW: Assert.SubtestLevel is a new property that returns the number of nested subtests.
* NEW: Assert.BailOut always unwinds the subtest stack and bails out of the entire TAP session.
* BUG: Tests can bail out even if Finished was called.

[GB.TEST]
* NEW: Bail out again if errors from test methods are caught.
* BUG: Don't call Finish on a subtest when it was already finished.

After the Bail out message, we Quit with non-zero exit status to signal
an abnormal termination to the harness.
2020-02-27 20:38:55 +01:00
Tobias Boege
c9c99c0050 gb.test: Use gb.test.tap
[GB.TEST]
* NEW: Use gb.test.

This removes the Assert and Track classes and uses gb.test.tap instead.
Test methods are now executed inside subtests, plan lines are auto-
matically emitted by the harness.

Differences between gb.tests's and gb.test.tap's Assert module:

  - Instead of Assert.True and Assert.False, use Assert.Ok.
  - Instead of Assert.Equals* for every type, there is a generic
    Assert.Equals for Variants and a specific Assert.StringEquals
    for strings which prints diagnostics when the test fails.
  - Assert.BailOut does not terminate the entire program, just
    the TAP stream.
  - Assert.ErrorWithCode is called Assert.ErrorCode.

All TAP printing is done through Assert now. Errors happening in the
test code are not caught and spoofed in the TAP stream anymore, they
cause a test failure. The system would be more robust if the testee
would be in a different process from the tester.
2020-02-27 20:38:55 +01:00
Christof Thalhofer
3752088d37 gbt3: Fix cherrypicking
[GBT3]
* BUG: Fix selection of testmodule and testmethod
2020-02-24 14:08:40 +01:00
Christof Thalhofer
06b405f22d gb.test: BailOut if a testmodule is empty
[GB.TEST]
* NEW: BailOut if a testmodule is empty
2020-02-23 22:31:28 +01:00
Christof Thalhofer
e4f06f6674 gb.test refactoring for use of test modules
[GB.TEST]
* NEW: Add support for test modules.
2020-02-23 12:38:53 +01:00
Christof Thalhofer
d9fff7c380 gb.test Assert.Error clears error 2020-02-10 11:18:29 +01:00
Christof Thalhofer
313ef71b0f gb.test Assert.Note complains about Null value
[GB.TEST]
    * NEW: Assert.Note complains about Null value
2020-02-01 23:54:01 +01:00
Christof Thalhofer
466aec07bb gb.test Assert returns result as boolean
[GB.TEST]
    * NEW: Assertions return test results as boolean. This can be used
    to print out an Assert.Note on failure.
2020-01-28 15:58:13 +01:00
Christof Thalhofer
550c56b596 gb.test: Fix plan if just one testmethod called
[GB.TEST]
    * BUG: Fix plan that counted all testmethods in a container even if just one was called
2020-01-24 10:10:39 +01:00
Christof Thalhofer
bae7204720 gb.test: Track errors with Error.Code 0 2020-01-22 09:13:41 +01:00
Christof Thalhofer
386f76e9fe gb.test UnitTest now scans .../.gambas for testclasses 2020-01-04 14:48:51 +01:00
Christof Thalhofer
c0fcfe0e5b gb.test: Always report Track.Plan <> Track.Counter
[GB.TEST]
*FIX: Always report Track.Plan <> Track.Counter even if no success
2019-12-31 01:51:24 +01:00
Christof Thalhofer
f9495607cf gb.test: Refactoring and improvements
[GB.TEST]
* NEW: Assert some reworking, skip, todo and body removed
* NEW: in a testcontainer in _new the count of tests can be defined by Me.Plan and differences between assertions counted and planned lead to No success
* NEW: Unittest.Bailout
2019-12-30 22:10:37 +01:00
Christof Thalhofer
6beaefbe1c new component gb.test
[GB.TEST]
* NEW: component for unittesting
2019-11-15 21:33:54 +00:00