Home - Waterfall Grid T-Grid Console Builders Recent Builds Buildslaves Changesources - JSON API - About

Console View

Legend:   Passed Failed Warnings Failed Again Running Exception Offline No data

Anton Shestakov
copystore: provide unit to ui.makeprogress()
Martin von Zweigbergk
requires: use atomictemp=True when writing .hg/requires

We use an unusual file system at Google that allows writes (and
renames) but not deletions (for certain paths). That causes problems
when writing the requires files without atomictemp=True. There doesn't
appear to be any real drawbacks to using atomictemp, so I'm hoping we
can just change it in core.

Differential Revision: https://phab.mercurial-scm.org/D5274
Anton Shestakov
verify: provide unit to ui.makeprogress()
Yuya Nishihara
blackbox: inline temporary variables which are referenced only once
Yuya Nishihara
blackbox: simply update global lastui variable at once
Yuya Nishihara
blackbox: consolidate conditions for early return

Just pick the lastui only if it is usable.
Yuya Nishihara
blackbox: remove redundant check for unassigned repo

Since ui._bbvfs is looked through ui._bbrepo, the repo instance should
exist if ui._bbvfs isn't None.
Yuya Nishihara
blackbox: extract _log() function which is called after lastui is resolved

This makes sure that self is the solo ui instance used in _log().
Augie Fackler
tests: fix bytes/str issue I introduced when adding this test

# skip-blame just b prefixes for py3

Differential Revision: https://phab.mercurial-scm.org/D5271
Kyle Lippincott
shelve: use matcher to restrict prefetch to just the modified files

Shelve currently operates by:
- make a temp commit
- identify all the bases necessary to shelve, put them in the bundle
- use exportfile to export the temp commit to the bundle ('file' here means
  "export to this fd", not "export this file")
- remove the temp commit

exportfile calls prefetchfiles, and prefetchfiles uses a matcher to restrict
what files it's going to prefetch; if it's not provided, it's alwaysmatcher.
This means that `hg shelve` in a remotefilelog repo can possibly download the
file contents of everything in the repository, even when it doesn't need to. It
luckily is restricted to the narrowspec (if there is one), but this is still a
lot of downloading that's just unnecessary, especially if there's a "smart"
VCS-aware filesystem involved.

exportfile is called with exactly one revision to emit, so we're just
restricting it to prefetching the files from that revision. The base revisions
having separate files should not be a concern since they're handled already;
example:

commit 10 is draft and modifies foo/a.txt and foo/b.txt
commit 11 is draft and modifies foo/a.txt
my working directory that I'm shelving modifies foo/b.txt

By the time we get to exportfile, commit 10 and 11 are already handled, so the
matcher only specifying foo/b.txt does not cause any problems. I verified this
by doing an `hg unbundle` on the bundle that shelve produces, and getting the
full contents of those commits back out, instead of just the files that were
modified in the shelve.

Differential Revision: https://phab.mercurial-scm.org/D5268
Pulkit Goyal
store: raise ProgrammingError if unable to decode a storage path

Right now, the function magically return False which is dangerous, so let's
raise ProgrammingError.

Suggested by Augie in D5139.

Differential Revision: https://phab.mercurial-scm.org/D5264
Matt Harbison
tests: document a known failing interaction between narrow and lfs

This is one of the two remaining aborts I found looking into issue5794.  I've
got no idea what's wrong with the hook, since the changes there fixed the other
two problems noted in that bug report.  It seems like it might go away when the
narrow issue is fixed, but let's make sure this doesn't get lost.

The stacktrace for the hook seems to indicate that the missing file *is* in ctx:

  remote: Traceback (most recent call last):
  remote:  File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs
  remote:    if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
  remote:  File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr>
  remote:    if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
  remote:  File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs
  remote:    return _islfs(self.filelog(), self.filenode())
  remote:  File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode
  remote:    return self._filenode
  remote:  File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__
  remote:    result = self.func(obj)
  remote:  File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode
  remote:    return self._filelog.lookup(self._fileid)
  remote:  File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup
  remote:    self._revlog.indexfile)
  remote:  File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup
  remote:    raise error.LookupError(fileid, identifier, _('no match found'))
  remote: LookupError: data/inside2/f.i@f59b4e021835: no match found
Gregory Szorc
revlog: use single file handle when de-inlining revlog

_getsegmentforrevs() will eventually call into _datareadfp() to
resolve a file handle to read revision data. If no file handle
is passed into _getsegmentforrevs(), it opens a new one.

Explicit is better than implicit.

This commit changes _enforceinlinesize() to open a file handle
explicitly when converting inline revlogs to split revlogs and
to pass this file handle into _getsegmentforrevs().

I haven't measured, but this change should improve performance,
as we no longer reopen the revlog for reading for every revision
in the revlog when it is converted from inline to split. Instead,
we open it at most once and use it for the duration of the
operation. That being said, I /think/ the chunk cache may mitigate
the number of file opens required.

Differential Revision: https://phab.mercurial-scm.org/D5265
Gregory Szorc
revlog: automatically read from opened file handles

The revlog reading code commonly opens a new file handle for
reading on demand. There is support for passing a file handle
to revlog.revision(). But it is marked as an internal argument.

When revlogs are written, we write() data as it is available. But
we don't flush() data until all revisions are written.

Putting these two traits together, it is possible for an in-process
revlog reader during active writes to trigger the opening of a new
file handle on a file with unflushed writes. The reader won't have
access to all "available" revlog data (as it hasn't been flushed).
And with the introduction of the previous patch, this can lead to
the revlog raising an error due to a partial read.

I witnessed this behavior when applying changegroup data (via
`hg pull`) before issue6006 was fixed via different means. Having
this and the previous patch in play would have helped cause errors
earlier rather than manifesting as hash verification failures.

While this has been a long-standing issue, I believe the relatively
new delta computation code has tickled it into being more common.
This is because the new delta computation code will compute deltas
in more scenarios. This can lead to revlog reading. While the delta
computation code is probably supposed to reuse file handles, it
appears it isn't doing so in all circumstances.

But the issue runs deeper than that. Theoretically, any code can
access revision data during revlog writes. It appears we were just
getting lucky that it wasn't. (The "add revision callback" passed to
addgroup() provides an avenue to do this.)

If I changed the revlog's behavior to not cache the full revision
text or to clear caches after revision insertion during addgroup(),
I was able to produce crashes 100% of the time when writing changelog
revisions. This is because changelog's add revision callback attempts
to resolve the revision data to access the changed files list. And
without the revision's fulltext being cached, we performed a revlog
read, which required opening a new file handle. This attempted to read
unflushed data, leading to a partial read and a crash.

This commit teaches the revlog to store the file handles used for
writing multiple revisions during addgroup(). It also teaches the
code for resolving a file handle when reading to use these handles,
if available. This ensures that *any* reads (regardless of their
source) use the active writing file handles, if available. These
file handles have access to the unflushed data because they wrote it.
This allows reads to complete without issue.

Differential Revision: https://phab.mercurial-scm.org/D5267
Gregory Szorc
revlog: detect incomplete revlog reads

_readsegment() is supposed to return N bytes of revlog revision
data starting at a file offset. Surprisingly, its behavior before
this patch never verified that it actually read and returned N
bytes! Instead, it would perform the read(), then return whatever
data was available. And even more surprisingly, nothing in the
call chain appears to have been validating that it received all
the data it was expecting.

This behavior could lead to partial or incomplete revision chunks
being operated on. This could result in e.g. cached deltas being
applied against incomplete base revisions. The delta application
process would happily perform this operation. Only hash
verification would detect the corruption and save us.

This commit changes the behavior of raw revlog reading to validate
that we actually read() the number of bytes that were requested.
We will raise a more specific error faster, rather than possibly
have it go undetected or manifest later in the call stack, at
delta application or hash verification.

Differential Revision: https://phab.mercurial-scm.org/D5266
Yuya Nishihara
logtoprocess: drop support for ui.log() call with invalid msg arguments (BC)

Before, the logtoprocess extension put a formatted message into $MSG1, and
its arguments to $MSG2... If the specified arguments couldn't be formatted
because of a caller bug, an unformatted message was passed in to $MSG1
instead of exploding. This behavior doesn't make sense.

Since I'm planning to formalize the ui.log() interface such that we'll no
longer have to extend the ui class, I want to remove any features not
conforming to the ui.log() API. So this patch removes the support for
ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems
useless as long as the message can be formatted. The $MSG1 variable isn't
renamed for the maximum compatibility.

In future patches, a formatted msg will be passed to a processlogger object,
instead of overriding the ui.log() function.

.. bc::

  The logtoprocess extension no longer supports invalid ``ui.log()``
  arguments. A log message is always formatted and passed in to the
  ``$MSG1`` environment variable.
Yuya Nishihara
templater: check invalid use of list expression properly (issue5920)

The error message is still cryptic, but it should be better.
Yuya Nishihara
logtoprocess: rewrite dict building in py3-compatible way
Yuya Nishihara
logtoprocess: leverage procutil.shellenviron() to stringify variables (BC)

This should make the extension more Py3 friendly. The environment variables
of the main process are copied to the dict by shellenviron().

.. bc::

  Boolean options passed to the logtoprocess extension are now formatted
  as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.
Yuya Nishihara
py3: byte-stringify inline extension in test-logtoprocess.t
Augie Fackler
tests: fix wireproto redirection test on systems without tls1.2

Our automated package builder has some ancient configuration that
lacks modern TLS, which is how we noticed this.

Tested: the test now passes on both macOS High Sierra (has tls1.2) and
Ubuntu Trusty (which does not).
  • FreeBSD hg tests (stable): run-tests.py (python 2.7.13) failed -  stdiowarnings (1)
  • Win7 x86_64 hg tests (stable): run-tests.py (python 2.7.13) failed -  stdiowarnings (1)
Pulkit Goyal
py3: use node.hex(h.digest()) instead of h.hexdigest()

hashlib.sha1.hexdigest() returns str on Python 3.

Differential Revision: https://phab.mercurial-scm.org/D5261
Pulkit Goyal
py3: add b suffix to make sure file is opened in bytes mode

Differential Revision: https://phab.mercurial-scm.org/D5263
Pulkit Goyal
py3: make second argument of fdopen() a str

# skip-blame as just r'' prefixes

Differential Revision: https://phab.mercurial-scm.org/D5260
Pulkit Goyal
py3: don't use dict.iterkeys()

dict.iterkeys() is not present on Python 3.

Differential Revision: https://phab.mercurial-scm.org/D5262
Pulkit Goyal
py3: fix keyword arguments handling in hgext/remotefilelog/

Keys of kwargs on Python 3 should be strings. This patch fixes them by appending
r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs().

Differential Revision: https://phab.mercurial-scm.org/D5259
Pulkit Goyal
py3: add __pycache__ to .hgignore

Something changed with Python 3 just like million on things. Maybe they should
named Python 3 as anaconda or cobra instead of just increasing the version
number.

This makes test-custom-filters.t pass on Python 3.

Differential Revision: https://phab.mercurial-scm.org/D5039
Boris Feld
sparse-revlog: fast-path before computing payload size

In this fast-path case, we do not need to compute the full delta chain
payload. Since it comes with a significant cost, we avoid doing so if
possible.
Boris Feld
sparse-revlog: stop using a heap to track selected gap

Same logic as for 'gapsheap', we don't actually need a heap.
Boris Feld
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity

We no longer need to deal with slicing logic including uncommitted revision. We
drop the associated code.
Boris Feld
sparse-revlog: drop unused deltainfo parameter from segmentspan

We no longer need to deal with slicing logic including uncommitted revision.
We drop the associated code.
Boris Feld
sparse-revlog: skip the span check in the sparse-revlog case

This significantly improves the performance on unbundling on smaller
repositories.

Mercurial: unbundling 1K revisions

    no-sparse-revlog:    500 ms
    sparse-revlog-before: 689 ms
    sparse-revlog-after:  484 ms

Pypy: unbundling 1K revisions

    no-sparse-revlog:    1.242 s
    sparse-revlog-before: 1.135 s
    sparse-revlog-after:  0.860 s

NetBeans: unbundling 1K revisions

    no-sparse-revlog:    1.386 s
    sparse-revlog-before: 2.368 s
    sparse-revlog-after:  1.191 s

Mozilla: unbundling 1K revisions

    no-sparse-revlog:    3.103 s
    sparse-revlog-before: 3.367 s
    sparse-revlog-after:  3.093 s
Boris Feld
sparse-revlog: stop using a heap to track gaps

The heap doesn't bring any performance advantage as we can simply sort the
final list.

Moreover, the lesser complexity helps a lot when we later implement it in C.
Augie Fackler
histedit: import chistedit curses UI from hg-experimental

I don't tend to like curses interfaces, but this gets enough use at
work that it seems like it's worth bringing into core. This is a
minimal import from hg-experimental revision 4c7f33bf5f00, in that
I've done the smallest amount of code movement and editing in order to
import the functionality.

.. feature::

    `hg histedit` will now present a curses UI if curses is available
    and `ui.interface` or `ui.interface.histedit` is set to `curses`.

Differential Revision: https://phab.mercurial-scm.org/D5146
Augie Fackler
manifest: also reject obviously-too-short lines when parsing lines

Differential Revision: https://phab.mercurial-scm.org/D5257
Augie Fackler
revlog: replace PyInt_AS_LONG with a more portable helper function

PyInt_AS_LONG disappears on Python, and our previous #define was
producing some problems on Python 3. Let's give up and make an inline
helper function that makes this more sane.

Differential Revision: https://phab.mercurial-scm.org/D5235
Augie Fackler
manifest: perform cheap checks before potentially allocating memory

Differential Revision: https://phab.mercurial-scm.org/D5258
Augie Fackler
manifest: make sure there's a filename before bothering to look for newline

There's no valid manifest that would have no characters before the NUL byte on
a line, and this fixes some erratic timeouts in the fuzzer.

Differential Revision: https://phab.mercurial-scm.org/D5256
Yuya Nishihara
commandserver: add experimental option to use separate message channel

This is loosely based on the idea of the TortoiseHg's pipeui extension,
which attaches ui.label to message text so the command-server client can
capture prompt text, for example.

https://bitbucket.org/tortoisehg/thg/src/4.7.2/tortoisehg/util/pipeui.py

I was thinking that this functionality could be generalized to templating,
but changed mind as doing template stuff would be unnecessarily complex.
It's merely a status message, a simple serialization option should suffice.

Since this slightly changes the command-server protocol, it's gated by a
config knob. If the config is enabled, and if it's supported by the server,
"message-encoding: <name>" is advertised so the client can stop parsing
'o'/'e' channel data and read encoded messages from the 'm' channel. As we
might add new message encodings in future releases, client can specify a list
of encoding names in preferred order.

This patch includes 'cbor' encoding as example. Perhaps, 'json' should be
supported as well.
  • Win7 x86_64 hg tests: make local failed -  stdio
Yuya Nishihara
help: document weird behavior of uisetup()

While refactoring ui.log() API, it was really annoying that uisetup() is
actually an extsetup() the phase 1. Let's document that. I'm planning to
add another uisetup() which is called per ui instance, though I can't think
of a good name for it.
  • Win7 x86_64 hg tests: make local failed -  stdio