If `VtEngine` gets removed from conhost, we need to be able to run
without any renderer present whatsoever. To make this possible,
I've turned all `Renderer&` into `Renderer*`.
Part of #14000
Subjectively speaking, this commit makes 3 improvements:
* Most importantly, it now would work with arbitrary Unicode text.
(No more `IsGlyphFullWidth` or DBCS handling during reflow.)
* Due to the simpler implementation it hopefully makes review of
future changes and maintenance simpler. (~3x less LOC.)
* It improves perf. by 1-2 orders of magnitude.
(At 120x9001 with a full buffer I get 60ms -> 2ms.)
Unfortunately, I'm not confident that the new code replicates the old
code exactly, because I failed to understand it. During development
I simply tried to match its behavior with what I think reflow should do.
Closes#797Closes#3088Closes#4968Closes#6546Closes#6901Closes#15964
Closes MSFT:19446208
Related to #5800 and #8000
## Validation Steps Performed
* Unit tests ✅
* Feature tests ✅
* Reflow with a scrollback ✅
* Reflowing the cursor cell causes a forced line-wrap ✅
(Even at the end of the buffer. ✅)
* `color 8f` and reflowing retains the background color ✅
* Enter alt buffer, Resize window, Exit alt buffer ✅
The ultimate goal of this PR was to use ICU for text search to
* Improve Unicode support
Previously we used `towlower` and only supported BMP glphs.
* Improve search performance (10-100x)
This allows us to search for all results in the entire text buffer
at once without having to do so asynchronously.
Unfortunately, this required some significant changes too:
* ICU's search facilities operate on text positions which we need to be
mapped back to buffer coordinates. This required the introduction of
`CharToColumnMapper` to implement sort of a reverse-`_charOffsets`
mapping. It turns text (character) positions back into coordinates.
* Previously search restarted every time you clicked the search button.
It used the current selection as the starting position for the new
search. But since ICU's `uregex` cannot search backwards we're
required to accumulate all results in a vector first and so we
need to cache that vector in between searches.
* We need to know when the cached vector became invalid and so we have
to track any changes made to `TextBuffer`. The way this commit solves
it is by splitting `GetRowByOffset` into `GetRowByOffset` for
`const ROW` access and `GetMutableRowByOffset` which increments a
mutation counter on each call. The `Search` instance can then compare
its cached mutation count against the previous mutation count.
Finally, this commit makes 2 semi-unrelated changes:
* URL search now also uses ICU, since it's closely related to regular
text search anyways. This significantly improves performance at
large window sizes.
* A few minor issues in `UiaTracing` were fixed. In particular
2 functions which passed strings as `wstring` by copy are now
using `wstring_view` and `TraceLoggingCountedWideString`.
Related to #6319 and #8000
## Validation Steps Performed
* Search upward/downward in conhost ✅
* Search upward/downward in WT ✅
* Searching for any of ß, ẞ, ss or SS matches any of the other ✅
* Searching for any of Σ, σ, or ς matches any of the other ✅
This is a follow-up of #13025 to make the members of `til::point/size/rect`
uniform and consistent without the use of `unions`. The only file that has
any changes is `src/host/getset.cpp` where an if condition was simplified.
## Validation Steps Performed
* Host unit tests ✅
* Host feature tests ✅
* ControlCore feature tests ✅
This commit replaces `Utf16Parser` with `<til/unicode.h>` which includes:
* `til::utf16_iterator` as a replacement for `Utf16Parser::Parse`
* `til::utf16_next` as a replacement for `Utf16Parser::ParseNext`
This fixes 2 bugs with `Utf16Parser`:
* Swallowing invalid surrogate pairs instead of turning them into U+FFFD.
* `std::vector<std::vector<wchar_t>>`. It's now >12000% faster.
## Validation Steps Performed
* New unit tests pass ✅
* Searching for narrow/wide characters in conhost works ✅
This commit is a from-scratch rewrite of `ROW` with the primary goal to get
rid of the rather bodgy `UnicodeStorage` class and improve Unicode support.
Previously a 120x9001 terminal buffer would store a vector of 9001 `ROW`s
where each `ROW` stored exactly 120 `wchar_t`. Glyphs exceeding their
allocated space would be stored in the `UnicodeStorage` which was basically
a `hashmap<Coordinate, String>`. Iterating over the text in a `ROW` would
require us to check each glyph and fetch it from the map conditionally.
On newlines we'd have to invalidate all map entries that are now gone,
so for every invalidated `ROW` we'd iterate through all glyphs again and if
a single one was stored in `UnicodeStorage`, we'd then iterate through the
entire hashmap to remove all coordinates that were residing on that `ROW`.
All in all, this wasn't the most robust nor performant code.
The new implementation is simple (from a design perspective):
Store all text in a `ROW` in a regular string. Grow the string if needed.
The association between columns and text works by storing character offsets
in a column-wide array. This algorithm is <100 LOC and removes ~1000.
As an aside this PR does a few more things that go hand in hand:
* Remove most of `ROW` helper classes, which aren't needed anymore.
* Allocate backing memory in a single `VirtualAlloc` call.
* Rewrite `IsCursorDoubleWidth` to use `DbcsAttrAt` directly.
Improves overall performance by 10-20% and makes this implementation
faster than the previous NxM storage, despite the added complexity.
Part of #8000
## Validation Steps Performed
* Existing and new unit and feature tests complete ✅
* Printing Unicode completes without crashing ✅
* Resizing works without crashing ✅
Previously this project used a great variety of types to present text buffer
coordinates: `short`, `unsigned short`, `int`, `unsigned int`, `size_t`,
`ptrdiff_t`, `COORD`/`SMALL_RECT` (aka `short`), and more.
This massive commit migrates almost all use of those types over to the
centralized types `til::point`/`size`/`rect`/`inclusive_rect` and their
underlying type `til::CoordType` (aka `int32_t`).
Due to the size of the changeset and statistics I expect it to contain bugs.
The biggest risk I see is that some code potentially, maybe implicitly, expected
arithmetic to be mod 2^16 and that this code now allows it to be mod 2^32.
Any narrowing into `short` later on would then throw exceptions.
## PR Checklist
* [x] Closes#4015
* [x] I work here
* [x] Tests added/passed
## Validation Steps Performed
Casual usage of OpenConsole and Windows Terminal. ✅
## Summary of the Pull Request
The purpose of the `IRenderTarget` interface was to support the concept
of multiple buffers in conhost. When a text buffer needed to trigger a
redraw, the render target implementation would be responsible for
deciding whether to forward that redraw to the renderer, depending on
whether the buffer was active or not.
This PR instead introduces a flag in the `TextBuffer` class to track
whether it is active or not, and it can then simply check the flag to
decide whether it needs to trigger a redraw or not. That way it can work
with the `Renderer` directly, and we can avoid a bunch of virtual calls
for the various redraw methods.
## PR Checklist
* [x] Closes#12551
* [x] CLA signed.
* [ ] Tests added/passed
* [ ] Documentation updated.
* [ ] Schema updated.
* [x] I've discussed this with core contributors already. Issue number
where discussion took place: #12551
## Detailed Description of the Pull Request / Additional comments
Anywhere that had previously been getting an `IRenderTarget` from the
`TextBuffer` class in order to trigger a redraw, will now just call the
`TriggerRedraw` method on the `TextBuffer` class itself, since that will
handle the active check which used to be the responsibility of the
render target. All the redraw methods that were in `IRenderTarget` are
now exposed in `TextBuffer` instead.
For this to work, though, the `Renderer` needed to be available before
the `TextBuffer` could be constructed, which required a change in the
conhost initialization order. In the `ConsoleInitializeConnectInfo`
function, I had to move the `Renderer` initialization up so it could be
constructed before the call to `SetUpConsole` (which initializes the
buffer). Once both are ready, the `Renderer::EnablePainting` method is
called to start the render thread.
The other catch is that the renderer used to setup its initial viewport
in the constructor, but with the new initialization order, the viewport
size would not be known at that point. I've addressed that problem by
moving the viewport initialization into the `EnablePainting` method,
since that will only be called after the buffer is setup.
## Validation Steps Performed
The changes in architecture required a number of tweaks to the unit
tests. Some were using a dummy `IRenderTarget` implementation which now
needed to be replaced with a full `Renderer` (albeit with mostly null
parameters). In the case of the scroll test, a mock `IRenderTarget` was
used to track the scroll delta, which now had to be replaced with a mock
`RenderEngine` instead.
Some tests previously relied on having just a `TextBuffer` but without a
`Renderer`, and now they require both. And tests that were constructing
the `TextBuffer` and `Renderer` themselves had to be updated to use the
new initialization order, i.e. with the renderer constructed first.
Semantically, though, the tests still essentially work the same way as
they did before, and they all still pass.
Moving things out of CharRow into ROW helps us hide it as an implementation detail.
This is part one of many.
### CharRow: Hide ClearCell, use ROW::ClearColumn
### CharRow: Hide GetText, use ROW::GetText
### CharRowBaseTests: remove dead file (never used!)
### CharRow: Move DoubleBytePadded into ROW
### CharRow: Move WrapForced into ROW
### Char/AttrRow: Hide Reset, use ROW::Reset
### Remove RowCellIterator (dead code)
RCI was unused; it was replaced by TextBufferCellIterator shortly after its creation
### Move AttrRowTests to ut_textbuffer from ut_host
It had no reliance on the host.
This is by no means comprehensive. It will be unmarked as draft when it
is more comprehensive.
This pull request adds some tests for resizing a TextBuffer and
reflowing its contents. Each test takes the form of an initial state and
a number of buffers of different sizes. The initial state is used to
seed the first TextBuffer, and the subsequent buffers are only used to
compare.
I manually reimplemented some of the DBCS logic to ensure that the
buffers contain _exactly_ what they're supposed to. I know this is
non-ideal. After some of the CharRow changes in #8446 land, this will
need to be updated.
There's a cool bit of TAEF gore in here: the IDataSource. An IDataSource
allows us to programmatically return test cases. It's a code-only
version of its support for parameterized tests of the form `Data:x = {0,
1, 2}` .
The only downsides are...
1. It looks like COM (it is not using COM under the hood, just the COM
ABI)
2. Property values must be returned as strings.
To best support rich test types, I used IDataSource to produce _a lit of
array indices_ and nothing more. The test is run once for array member,
and it is the test's responsibility to look up the object to which that
index refers.
Works great though! Each reflow test is its own unit, and a failure in
an earlier reflow test will not tank a later one.
