[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[libreoffice-website] [TDF infra announce/request] reduce downstream traffic by 50× on git clones with `git config protocol.version 2`


TL;DR: run `git config protocol.version 2` in your local clones of the
core and online repositories. That should reduce downstream traffic by
over 50x (from 9MiB to 150kiB) in no-op `git fetch` commands.
Client-side this only applies to git ≥2.18.0.

Dear developers,

We just upgraded https://gerrit.libreoffice.org to Gerrit v3.1. Thanks
to upstream work by David Ostrovsky and others this version adds support
for version 2 of the git wire protocol [0], which significantly improves
performance of `git fetch` commands on repositories with many references
such as ours. That's something we had our eyes on for a while [1].

core currently has over 300k references, the overwhelming majority of
which from changesets (each changeset creates n+1 refs, where n is the
number of its revisions). In version 1 of the git wire protocol the
entire reference list, along with the commit IDs they point to, is sent
to client in the initial server advertisement (as can be seen by adding
‘GIT_TRACE_PACKET=1’ to the environment of a git-fetch(1) command).
This causes significant downstream traffic, especially because the
commit IDs makes compression moot. On our large repositories this is
even a noticeable bottleneck. In contrast, with version 2 only requested
refs are being sent, thereby significantly reducing downstream traffic
when the repository is already up to date. Here are some metrics for a
fresh core clone:

* `git fetch` yields an advertisement of size 8666kiB, and takes over
3s to complete (in the same datacenter); while
* `git -c protocol.version=2 fetch` yields a response of size 150k and
takes under 0.5s to complete.

That's over 50× smaller! Of course on an “active” clone the initial
advertisement won't be empty as it'll likely contain refs for the recent
changesets you're working on, but certainly not the 300k (and counting)
references. So most likely a two digit factor improvement :-)

The performance gain is visible for other transports as well, so this
applies to ssh://gerrit.libreoffice.org:9418 too (and even to the now
deprecated git://git.libreoffice.org).

Please run

git config protocol.version 2

in your local clones to force the client to use version 2 of the wire
protocol. Or even globally (it will fallback to the original wire
protocol if the server doesn't support it) with

git config --global protocol.version 2

To check if that worked, `GIT_TRACE_PACKET=1 git fetch 2>&1 | head`
should produce the following output

pkt-line.c:80 packet: git< version 2
pkt-line.c:80 packet: git< ls-refs
[…]
pkt-line.c:80 packet: git< 0000
pkt-line.c:80 packet: fetch< version 2
pkt-line.c:80 packet: fetch< ls-refs
[…]

This is a wire protocol, so the change is only relevant for transport.
No need to re-create existing clones.

I suppose git upstream will change the default protocol.version at some
point, but given the above I see no reason to wait.

Thanks!
Cheers.
--
Guilhem, for The Document Foundation's infrastructure team.

PS: Please preserve the recipient list in replying.

[0] https://git-scm.com/docs/protocol-v2
[1] https://lists.freedesktop.org/archives/libreoffice/2018-October/081249.html

--
To unsubscribe e-mail to: website+unsubscribe@global.libreoffice.org
Problems? https://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: https://wiki.documentfoundation.org/Netiquette
List archive: https://listarchives.libreoffice.org/global/website/
Privacy Policy: https://www.documentfoundation.org/privacy

Privacy Policy | Impressum (Legal Info) | Copyright information: Unless otherwise specified, all text and images on this website are licensed under the Creative Commons Attribution-Share Alike 3.0 License. This does not include the source code of LibreOffice, which is licensed under the Mozilla Public License (MPLv2). "LibreOffice" and "The Document Foundation" are registered trademarks of their corresponding registered owners or are in actual use as trademarks in one or more countries. Their respective logos and icons are also subject to international copyright laws. Use thereof is explained in our trademark policy.