[V3 Downloader] Revision tracking (#2571)

* feat(downloader): Install cog from specific commit in repo (initial commit) - Repo and Installable have commit property now - New class inheriting from Installable - InstalledCog (old one from converters.py removed) - New Repo.checkout() method, which is also async ctx manager ref #2527 * fix(downloader): Keep information about repo's branch in config - This is needed to make sure that repo can go back from detached state in some rare unexpected cases - current branch is determined by `git symbolic-ref` now as this command errors for detached HEAD * feat(downloader): Update repo without cogs, update single cog The most important part of issue #2527 has been added here - `[p]repo update` command added - new conf format - nested dictionary repo_name->cog_name->cog_json installed libraries are now kept in conf too - `InstalledCog` renamed to `InstalledModule` - installed libraries use this class - `Downloader.installed_libraries()` and `Downloader.installed_modules()` added - `Downloader._add_to_installed()` and `Downloader._remove_from_installed()` now accept list of modules, of both cogs and libraries - `[p]cog install` tells about fails of copying cog and installing shared libraries - `[p]cog update` will truly update only chosen cogs (if provided) or cogs that need update - pinned cogs aren't checked - before update, repos are updated - to determine if update is needed `Repo.get_modified_modules()` is used - `[p]cog pin` and `[p]cog unpin` commands for pinning/unpinning cogs added - `Repo.checkout()` allows to choose ctx manager exit's checkout revision - `Repo.install_cog()` returns `InstalledModule` now and raises CopyingError (maybe breaking?) - `Repo.install_libraries()` returns 2-tuple of installed and failed libraries (maybe breaking?) - `RepoManager.get_all_cogs()` added, which returns cogs from all repos - `RepoManager.repos` property added, which contains tuple of `Repo` * test(downloader): Repo.current_branch() throws an exception, when branch can't be determined * style(downloader): rename _add_to_installed to _save_to_installed This method is used for both adding and updating existing modules in Config * refactor(downloader): add ctx.typing() for few commands `[p]cog install` is nested hell, can't wait for moving install logic to separate method * fix(downloader): refactor and fix `set` usage * perf(downloader): update commits for ALL checked modules to omit diffs next time This will also disable running git diff for cogs that have the same commit as the latest one * style(downloader): few style improvements - use of mutable object in method definition - make Repo._get_full_sha1() public method - too long line - don't use len to check if sequence is empty * feat(downloader): add `[p]cog updateallfromrepos` and `[p]cog updatetoversion` commands - moved cog update logic into `Downloader._cog_update_logic()` (lack of better name) - splitted whole cog update process into smaller methods - might still need some improvements - added new methods to `Repo` class: - `is_on_branch()` to check if repo is currently checked out to branch - `is_ancestor()` to check if one commit is ancestor of the other - fix for `Downloader._available_updates()` behaviour broken by commit 5755ab08ba67556b3863e907c6f44d80f4f13d88 * feat(downloader): try to find last commit where module is still present Enhancements: - `Installable` now has `repo` attribute containing repo object or `None` if repo is missing - `Downloader._install_cogs()` and `Downloader._reinstall_libraries()` are able to install modules from different commits of repo - `Repo.checkout()` as ctx manager will now exit to commit which was active before checking out - unification of `rev` and `hash` terms: All function parameters are explicitly called `hash`, if it can only be commit's full sha1 hash or `rev` if it can be anything that names a commit object, see [link](https://git-scm.com/docs/git-rev-parse#_specifying_revisions) - new `Repo.get_last_module_occurence()` method, which gets module's Installable from last commit in which it still occurs * docs(downloader): Add basic description for `InstalledModule` * fix(downloader): cog ignored during updates if its commit was missing After config format update, commit string is empty until update and when such cog was checked and it wasn't available in repo anymore, it was ignored * refactor(downloader): Installing cogs from specific rev will pin them * perf(downloader): Don't checkout when current commit equals target hash - changes to `Repo.checkout()`: - `exit_to_rev` is now keyword only argument - added `force_checkout` to force checkout even if `Repo.commit` value is the same as target hash * refactor(downloader): Repo._run() stderr is redirected to debug log now - added two keyword arguments: - `valid_exit_codes` which specifies valid exit codes, used to determine if stderr should be sent as debug or error level in logging - `debug_only` which specifies if stderr can be sent only as debug level in logging * style(downloader): stop using `set` as arg name in `_load_repos()` * feat(downloader): pass multiple cogs to `[p]cog (un)pin` * refactor(downloader): accept module name instead of instance, fix spelling * style(downloader): few small style changes * fix(downloader): add type annotations + fixes based on them - fix wrong type annotations and add a lot of new ones - add checks for `Installable.repo` being `None` - fix wrong return type in `Downloader._install_requirements` - show repo names correctly when updating all repos - fix error when some requirement fails to install BREAKING CHANGE: - type of `Repo.available_modules` is now consistent (always `tuple`) * tests: use same event loop policy as in Red's code * enhance(downloader): fully handle ambiguous revisions * build(deps): add pytest-mock dependency to tests extra * fix(downloader): minor fixes * feat(downloader): add tool for editing Downloader's test repo This script aims to help update the human-readable version of repo used for git integration tests in ``redbot/tests/downloader_testrepo.export`` by exporting/importing it in/from provided directory. Note ---- Editing `downloader_git_test_repo.export` file manually is strongly discouraged, especially editing any part of commit directives as that causes a change in the commit's hash. Another problem devs could encounter when trying to manually edit that file are editors that will use CRLF instead of LF for new line character(s) and therefore break it. I also used `.gitattributes` to prevent autocrlf from breaking testrepo. Also, if Git ever changes currently used SHA-1 to SHA-256 we will have to update old hashes with new ones. But it's a small drawback, when we can have human-readable version of repo. Known limitations ----------------- ``git fast-export`` exports commits without GPG signs so this script disables it in repo's config. This also means devs shouldn't use ``--gpg-sign`` flag in ``git commit`` within the test repo. * tests(downloader): add git tests and test repo for them Also added Markdown file that is even more clear than export file on what the test repo contains. This is manually created but can be automated on later date. * test(downloader): add more tests related to RepoManager These tests use expected output that is already guaranteed by git tests. * chore(CODEOWNERS): add jack1142 to Downloader's folders I know this doesn't actually give any benefit to people that don't have write permission to the repo but I saw other big fella devs doing this, so I think this might be advisable. * enhance(downloader): allow easy schema updates in future * enhance(downloader): more typing fixes, add comments for clarity * feat(downloader): add python and bot version check to update process follow-up on #2605, this commit fully fixes #1866 * chore(changelog): add towncrier entries * fix(downloader): use `*args` instead of `commands.Greedy` * fix(downloader): hot-reload issue - `InstallableType` now inherits from `IntEnum` There's desync of `InstallableType` class types due to hot-reload and `IntEnum` allows for equality check between different types * enhance(downloader): ensure there's no cog with same name installed should fix #2927 * fix(downloader): last few changes before marking as ready for review
2025-11-20 09:56:05 -05:00 · 2019-11-08 02:36:16 +01:00
parent d85fb260e7
commit e2c8b11008
40 changed files with 2922 additions and 452 deletions
--- a/redbot/pytest/.gitattributes
+++ b/redbot/pytest/.gitattributes
@@ -0,0 +1 @@
+downloader_testrepo.export -text
--- a/redbot/pytest/downloader.py
+++ b/redbot/pytest/downloader.py
@@ -1,39 +1,43 @@
 from collections import namedtuple
 from pathlib import Path
 import json
+import subprocess as sp
+import shutil

 import pytest

-from redbot.cogs.downloader.repo_manager import RepoManager, Repo
-from redbot.cogs.downloader.installable import Installable
+from redbot.cogs.downloader.repo_manager import RepoManager, Repo, ProcessFormatter
+from redbot.cogs.downloader.installable import Installable, InstalledModule

 __all__ = [
    "patch_relative_to",
    "repo_manager",
    "repo",
-    "repo_norun",
    "bot_repo",
    "INFO_JSON",
    "LIBRARY_INFO_JSON",
    "installable",
+    "installed_cog",
    "library_installable",
    "fake_run_noprint",
+    "fake_current_commit",
+    "_session_git_repo",
+    "git_repo",
+    "cloned_git_repo",
+    "git_repo_with_remote",
 ]


-async def fake_run(*args, **kwargs):
-    fake_result_tuple = namedtuple("fake_result", "returncode result")
-    res = fake_result_tuple(0, (args, kwargs))
-    print(args[0])
-    return res
-
-
 async def fake_run_noprint(*args, **kwargs):
    fake_result_tuple = namedtuple("fake_result", "returncode result")
    res = fake_result_tuple(0, (args, kwargs))
    return res


+async def fake_current_commit(*args, **kwargs):
+    return "fake_result"
+
+
@pytest.fixture(scope="module", autouse=True)
 def patch_relative_to(monkeysession):
    def fake_relative_to(self, some_path: Path):
@@ -50,30 +54,26 @@ def repo_manager(tmpdir_factory):


@pytest.fixture
-def repo(tmpdir):
-    repo_folder = Path(str(tmpdir)) / "repos" / "squid"
+def repo(tmp_path):
+    repo_folder = tmp_path / "repos" / "squid"
    repo_folder.mkdir(parents=True, exist_ok=True)

    return Repo(
        url="https://github.com/tekulvw/Squid-Plugins",
        name="squid",
        branch="rewrite_cogs",
+        commit="6acb5decbb717932e5dc0cda7fca0eff452c47dd",
        folder_path=repo_folder,
    )


-@pytest.fixture
-def repo_norun(repo):
-    repo._run = fake_run
-    return repo
-
-
@pytest.fixture
 def bot_repo(event_loop):
    cwd = Path.cwd()
    return Repo(
        name="Red-DiscordBot",
        branch="WRONG",
+        commit="",
        url="https://empty.com/something.git",
        folder_path=cwd,
        loop=event_loop,
@@ -120,6 +120,16 @@ def installable(tmpdir):
    return cog_info


+@pytest.fixture
+def installed_cog(tmpdir):
+    cog_path = tmpdir.mkdir("test_repo").mkdir("test_installed_cog")
+    info_path = cog_path.join("info.json")
+    info_path.write_text(json.dumps(INFO_JSON), "utf-8")
+
+    cog_info = InstalledModule(Path(str(cog_path)))
+    return cog_info
+
+
@pytest.fixture
 def library_installable(tmpdir):
    lib_path = tmpdir.mkdir("test_repo").mkdir("test_lib")
@@ -128,3 +138,93 @@ def library_installable(tmpdir):

    cog_info = Installable(Path(str(lib_path)))
    return cog_info
+
+
+# Git
+TEST_REPO_EXPORT_PTH: Path = Path(__file__).parent / "downloader_testrepo.export"
+
+
+def _init_test_repo(destination: Path):
+    # copied from tools/edit_testrepo.py
+    git_dirparams = ("git", "-C", str(destination))
+    init_commands = (
+        (*git_dirparams, "init"),
+        (*git_dirparams, "config", "--local", "user.name", "Cog-Creators"),
+        (*git_dirparams, "config", "--local", "user.email", "cog-creators@example.org"),
+        (*git_dirparams, "config", "--local", "commit.gpgSign", "false"),
+    )
+
+    for args in init_commands:
+        sp.run(args, check=True)
+    return git_dirparams
+
+
+@pytest.fixture(scope="session")
+async def _session_git_repo(tmp_path_factory, event_loop):
+    # we will import repo only once once per session and duplicate the repo folder
+    repo_path = tmp_path_factory.mktemp("session_git_repo")
+    repo = Repo(
+        name="redbot-testrepo",
+        url="",
+        branch="master",
+        commit="",
+        folder_path=repo_path,
+        loop=event_loop,
+    )
+    git_dirparams = _init_test_repo(repo_path)
+    fast_import = sp.Popen((*git_dirparams, "fast-import", "--quiet"), stdin=sp.PIPE)
+    with TEST_REPO_EXPORT_PTH.open(mode="rb") as f:
+        fast_import.communicate(f.read())
+    return_code = fast_import.wait()
+    if return_code:
+        raise Exception(f"git fast-import failed with code {return_code}")
+    sp.run((*git_dirparams, "reset", "--hard"))
+    return repo
+
+
+@pytest.fixture
+async def git_repo(_session_git_repo, tmp_path, event_loop):
+    # fixture only copies repo that was imported in _session_git_repo
+    repo_path = tmp_path / "redbot-testrepo"
+    shutil.copytree(_session_git_repo.folder_path, repo_path)
+    repo = Repo(
+        name="redbot-testrepo",
+        url=_session_git_repo.url,
+        branch=_session_git_repo.branch,
+        commit=_session_git_repo.commit,
+        folder_path=repo_path,
+        loop=event_loop,
+    )
+    return repo
+
+
+@pytest.fixture
+async def cloned_git_repo(_session_git_repo, tmp_path, event_loop):
+    # don't use this if you want to edit origin repo
+    repo_path = tmp_path / "redbot-cloned_testrepo"
+    repo = Repo(
+        name="redbot-testrepo",
+        url=str(_session_git_repo.folder_path),
+        branch=_session_git_repo.branch,
+        commit=_session_git_repo.commit,
+        folder_path=repo_path,
+        loop=event_loop,
+    )
+    sp.run(("git", "clone", str(_session_git_repo.folder_path), str(repo_path)), check=True)
+    return repo
+
+
+@pytest.fixture
+async def git_repo_with_remote(git_repo, tmp_path, event_loop):
+    # this can safely be used when you want to do changes to origin repo
+    repo_path = tmp_path / "redbot-testrepo_with_remote"
+    repo = Repo(
+        name="redbot-testrepo",
+        url=str(git_repo.folder_path),
+        branch=git_repo.branch,
+        commit=git_repo.commit,
+        folder_path=repo_path,
+        loop=event_loop,
+    )
+    sp.run(("git", "clone", str(git_repo.folder_path), str(repo_path)), check=True)
+    return repo
--- a/redbot/pytest/downloader_testrepo.export
+++ b/redbot/pytest/downloader_testrepo.export
@@ -0,0 +1,134 @@
+# THIS FILE SHOULDN'T BE EDITED MANUALLY. USE `edit_testrepo.py` TOOL TO UPDATE THE REPO.
+blob
+mark :1
+original-oid cfd75093008a560c1f2a09e5068e0dd1517eaa1c
+data 14
+Sample file 1.
+reset refs/heads/ambiguous_with_tag
+commit refs/heads/ambiguous_with_tag
+mark :2
+original-oid c6f0e5ec04d99bdf8c6c78ff20d66d286eecb3ea
+author Cog-Creators <cog-creators@example.org> 1571921830 +0200
+committer Cog-Creators <cog-creators@example.org> 1571919491 +0200
+data 27
+Commit ambiguous with tag.
+M 100644 :1 sample_file1.txt
+
+reset refs/heads/ambiguous_1
+commit refs/heads/ambiguous_1
+mark :3
+original-oid 95da0b576271cb5bee5f3e075074c03ee05fed05
+author Cog-Creators <cog-creators@example.org> 1571777704 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
+data 23
+Ambiguous commit 16955
+M 100644 :1 sample_file1.txt
+
+reset refs/heads/ambiguous_2
+commit refs/heads/ambiguous_2
+mark :4
+original-oid 95da0b57a416d9c8ce950554228d1fc195c30b43
+author Cog-Creators <cog-creators@example.org> 1571777704 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
+data 23
+Ambiguous commit 44414
+M 100644 :1 sample_file1.txt
+
+blob
+mark :5
+original-oid f1a18139c84a82addbded8a7b5738c36fb02fce1
+data 22
+print("Hello world!")
+
+blob
+mark :6
+original-oid 1abb7a2470722faee2175980ee202717b4158057
+data 14
+Sample file 2.
+reset refs/tags/lightweight
+commit refs/tags/lightweight
+mark :7
+original-oid c950fc05a540dd76b944719c2a3302da2e2f3090
+author Cog-Creators <cog-creators@example.org> 1571776887 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777047 +0200
+data 31
+Initial commit, prepare files.
+M 100644 :5 mycog/__init__.py
+M 100644 :1 sample_file1.txt
+M 100644 :6 sample_file2.txt
+
+blob
+mark :8
+original-oid 10ec5813415b6d7c902eee95cc13dc38c6f50917
+data 11
+Added file.
+blob
+mark :9
+original-oid 5ed17bf7914989db85f2e66045e62b35eed10f3b
+data 42
+def setup(bot):
+    print("Hello world!")
+
+commit refs/tags/lightweight
+mark :10
+original-oid fb99eb7d2d5bed514efc98fe6686b368f8425745
+author Cog-Creators <cog-creators@example.org> 1571777140 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777140 +0200
+data 39
+Add, modify, rename and remove a file.
+from :7
+M 100644 :8 added_file.txt
+M 100644 :9 mycog/__init__.py
+D sample_file1.txt
+D sample_file2.txt
+M 100644 :6 sample_file3.txt
+
+commit refs/tags/annotated
+mark :11
+original-oid a7120330cc179396914e0d6af80cfa282adc124b
+author Cog-Creators <cog-creators@example.org> 1571777209 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777209 +0200
+data 14
+Remove mycog.
+from :10
+D mycog/__init__.py
+
+blob
+mark :12
+original-oid 1ba9a868ae2f65571c75681ec47d40595bea4882
+data 14
+Sample file 4.
+commit refs/heads/master
+mark :13
+original-oid 2db662c1d341b1db7d225ccc1af4019ba5228c70
+author Cog-Creators <cog-creators@example.org> 1571777704 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
+data 32
+One commit after mycog removal.
+from :11
+M 100644 :12 sample_file4.txt
+
+reset refs/heads/dont_add_commits
+commit refs/heads/dont_add_commits
+mark :14
+original-oid a0ccc2390883c85a361f5a90c72e1b07958939fa
+author Cog-Creators <cog-creators@example.org> 1571777548 +0200
+committer Cog-Creators <cog-creators@example.org> 1571777548 +0200
+data 103
+Don't edit this, this is used for tests for current commit, latest commit, full sha1 from branch name.
+M 100644 :1 sample_file1.txt
+
+tag annotated
+from :11
+original-oid 41f6cf3b58e774d2b3414ced3ee9f2541f1c682f
+tagger Cog-Creators <cog-creators@example.org> 1571777367 +0200
+data 15
+Annotated tag.
+
+tag ambiguous_tag_66387
+from :2
+original-oid c6f028f843389c850e2c20d8dd1f5fa498252764
+tagger Cog-Creators <cog-creators@example.org> 1571919491 +0200
+data 37
+Annotated tag ambiguous with commit.
+
--- a/redbot/pytest/downloader_testrepo.md
+++ b/redbot/pytest/downloader_testrepo.md
@@ -0,0 +1,102 @@
+# Downloader's test repo reference
+
+This file can be used as a reference on what repo contains
+if some dev will want to add more test in future.
+
+Branch master
+---
+
+**Commit:** c950fc05a540dd76b944719c2a3302da2e2f3090
+**Commit message:** Initial commit, prepare files.
+**Tree status:**
+```
+downloader_testrepo/
+    ├── mycog
+A   │   ├── __init__.py
+A   ├── sample_file1.txt
+A   └── sample_file2.txt
+```
+---
+**Commit:** fb99eb7d2d5bed514efc98fe6686b368f8425745
+**Tag:** lightweight
+**Commit message:** Add, modify, rename and remove a file.
+**Tree status:**
+```
+downloader_testrepo/
+    ├── mycog/
+M   │   ├── __init__.py
+A   ├── added_file.txt
+D   ├── sample_file1.txt
+R   └── sample_file2.txt -> sample_file3.txt
+```
+---
+**Commit:** a7120330cc179396914e0d6af80cfa282adc124b
+**Tag:** annotated (sha1: 41f6cf3b58e774d2b3414ced3ee9f2541f1c682f)
+**Commit message:** Remove mycog.
+**Tree status:**
+```
+downloader_testrepo/
+D   ├── mycog/
+D   │   ├── __init__.py
+    ├── added_file.txt
+    └── sample_file3.txt
+```
+---
+**Commit:** 2db662c1d341b1db7d225ccc1af4019ba5228c70
+**Commit message:** One commit after mycog removal.
+**Tree status:**
+```
+downloader_testrepo/
+    ├── added_file.txt
+    ├── sample_file3.txt
+A   └── sample_file4.txt
+```
+
+Branch with persistent HEAD
+---
+
+**Commit:** a0ccc2390883c85a361f5a90c72e1b07958939fa
+**Branch:** dont_add_commits
+**Commit message:** Don't edit this, this is used for tests for current commit, latest commit, full sha1 from branch name.
+**Tree status:**
+```
+downloader_testrepo/
+A   └── sample_file1.txt
+```
+
+Branches with ambiguous commits (95da0b57)
+---
+
+**Commit:** 95da0b576271cb5bee5f3e075074c03ee05fed05
+**Branch:** ambiguous_1
+**Commit message:** Ambiguous commit 16955
+**Tree status:**
+```
+downloader_testrepo/
+A   └── sample_file1.txt
+```
+
+
+**Commit:** 95da0b57a416d9c8ce950554228d1fc195c30b43
+**Branch:** ambiguous_2
+**Commit message:** Ambiguous commit 44414
+**Tree status:**
+```
+downloader_testrepo/
+A   └── sample_file1.txt
+```
+
+
+Branch with ambiguous tag (c6f0)
+---
+
+**Commit:** c6f0e5ec04d99bdf8c6c78ff20d66d286eecb3ea
+**Branch:** ambiguous_with_tag
+**Tag:** ambiguous_tag_66387 (sha1: c6f028f843389c850e2c20d8dd1f5fa498252764)
+**Commit message:** Commit ambiguous with tag.
+**Tree status:**
+
+```
+downloader_testrepo/
+A   └── sample_file1.txt
+```