Update deps, allow Python 3.9, drop Fedora 32 (#5121)

* Update deps + allow Python 3.9

* Add in updates from Jack's branch

* Fix multiple target for cross-reference warnings

* Update a few more Python 3.8 occurrences

* Bump further

* Don't install tox in CodeQL environment

* Bump Python version in docs to 3.9.7

* more bumps

* Add missing pin

* Stop using the deprecated distro.linux_distribution()

* Suppress deprecation warning

* Fix OpenSUSE Leap instructions

* Drop Fedora 32

* Add Python 3.10-dev to CI

* meh

Co-authored-by: jack1142 <6032823+jack1142@users.noreply.github.com>

.github/workflows/codeql-analysis.yml

@@ -27,7 +27,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install -U pip setuptools wheel
-        python -m pip install -r ./tools/dev-requirements.txt
+        python -m pip install .[all]
         # Set the `CODEQL-PYTHON` environment variable to the Python executable
         # that includes the dependencies
         echo "CODEQL_PYTHON=$(which python)" >> $GITHUB_ENV

.github/workflows/tests.yml

@@ -17,12 +17,18 @@ jobs:
           python_version:
             - "3.8"
           tox_env:
-            - py
             - style
             - docs
           include:
-            - tox_env: py
+            - tox_env: py38
-              friendly_name: Tests
+              python_version: "3.8"
+              friendly_name: Python 3.8 - Tests
+            - tox_env: py39
+              python_version: "3.9"
+              friendly_name: Python 3.9 - Tests
+            - tox_env: py310
+              python_version: "3.10-dev"
+              friendly_name: Python 3.10-dev - Tests
             - tox_env: style
               friendly_name: Style
             - tox_env: docs
@@ -52,6 +58,8 @@ jobs:
         matrix:
           python_version:
             - "3.8"
+            - "3.9"
+            - "3.10-dev"
         fail-fast: false
       name: Tox - Postgres
       services:

docs/changelog_3_4_0.rst

@@ -1053,7 +1053,7 @@ Core Bot
 
 - Added data request API (:issue:`4045`,  :issue:`4169`)
 
-    - New special methods added to `commands.Cog`: `red_get_data_for_user()` (documented provisionally), `red_delete_data_for_user()`
+    - New special methods added to `redbot.core.commands.Cog`: `red_get_data_for_user()` (documented provisionally), `red_delete_data_for_user()`
     - New special module level variable added: ``__red_end_user_data_statement__``
     - These methods and variables should be added by all cogs according to their documentation; see `recommendations-for-cog-creators` for more information
     - New ``info.json`` key added: ``end_user_data_statement``; see `Info.json format documentation <info-json-format>` for more information

docs/install_linux_mac.rst

@@ -17,7 +17,7 @@ Installing the pre-requirements
 Please install the pre-requirements using the commands listed for your operating system.
 
 The pre-requirements are:
- - Python 3.8.1 or greater; **Python 3.9 is currently not supported!**
+ - Python 3.8.1 or greater
  - Pip 18.1 or greater
  - Git 2.11+
  - Java Runtime Environment 11 (for audio support)
@@ -40,11 +40,6 @@ Operating systems
 Arch Linux
 ~~~~~~~~~~
 
-.. warning::
-
-    Latest Python packages for Arch Linux provide Python 3.9 which Red does not currently support.
-    To use Red on Arch Linux, you will need to install latest version of Python 3.8 on your own.
-
 .. prompt:: bash
 
     sudo pacman -Syu python python-pip git jre11-openjdk-headless base-devel nano
@@ -76,7 +71,7 @@ In order to install Git 2.11 or greater, we recommend adding the IUS repository:
     sudo yum -y install https://repo.ius.io/ius-release-el7.rpm
     sudo yum -y swap git git224
 
-Complete the rest of the installation by `installing Python 3.8 with pyenv <install-python-pyenv>`.
+Complete the rest of the installation by `installing Python 3.9 with pyenv <install-python-pyenv>`.
 
 ----
 
@@ -94,7 +89,7 @@ CentOS and RHEL 8
     sudo yum -y groupinstall development
     sudo yum -y install git zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel xz xz-devel tk-devel libffi-devel findutils java-11-openjdk-headless nano
 
-Complete the rest of the installation by `installing Python 3.8 with pyenv <install-python-pyenv>`.
+Complete the rest of the installation by `installing Python 3.9 with pyenv <install-python-pyenv>`.
 
 ----
 
@@ -114,7 +109,7 @@ Debian/Raspbian Buster. This guide will tell you how. First, run the following c
     sudo apt -y install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev libgdbm-dev uuid-dev python3-openssl git openjdk-11-jre-headless nano
     CXX=/usr/bin/g++
 
-Complete the rest of the installation by `installing Python 3.8 with pyenv <install-python-pyenv>`.
+Complete the rest of the installation by `installing Python 3.9 with pyenv <install-python-pyenv>`.
 
 ----
 
@@ -124,12 +119,12 @@ Complete the rest of the installation by `installing Python 3.8 with pyenv <inst
 Fedora Linux
 ~~~~~~~~~~~~
 
-Fedora Linux 32 and above has all required packages available in official repositories. Install
+Fedora Linux 33 and above has all required packages available in official repositories. Install
 them with dnf:
 
 .. prompt:: bash
 
-    sudo dnf -y install python38 git java-11-openjdk-headless @development-tools nano
+    sudo dnf -y install python39 git java-11-openjdk-headless @development-tools nano
 
 Continue by `creating-venv-linux`.
 
@@ -153,8 +148,8 @@ one-by-one:
 
 .. prompt:: bash
 
-    brew install python@3.8
+    brew install python@3.9
-    echo 'export PATH="$(brew --prefix)/opt/python@3.8/bin:$PATH"' >> ~/.profile
+    echo 'export PATH="$(brew --prefix)/opt/python@3.9/bin:$PATH"' >> ~/.profile
     source ~/.profile
     brew install git
     brew install --cask adoptopenjdk/openjdk/adoptopenjdk11
@@ -172,7 +167,7 @@ openSUSE
 openSUSE Leap 15.2+
 *******************
 
-We recommend installing a community package to get Python 3.8 on openSUSE Leap 15.2+. This package will
+We recommend installing a community package to get Python 3.9 on openSUSE Leap 15.2+. This package will
 be installed to the ``/opt`` directory.
 
 First, add the Opt-Python community repository:
@@ -187,7 +182,7 @@ Now install the pre-requirements with zypper:
 
 .. prompt:: bash
 
-    sudo zypper -n install opt-python38 opt-python38-setuptools git-core java-11-openjdk-headless nano
+    sudo zypper -n install opt-python39 opt-python39-setuptools git-core java-11-openjdk-headless nano
     sudo zypper -n install -t pattern devel_basis
 
 Since Python is now installed to ``/opt/python``, we should add it to PATH. You can add a file in
@@ -198,11 +193,16 @@ Since Python is now installed to ``/opt/python``, we should add it to PATH. You
     echo 'export PATH="/opt/python/bin:$PATH"' | sudo tee /etc/profile.d/opt-python.sh
     source /etc/profile.d/opt-python.sh
 
-Now, install pip with easy_install:
+Now, bootstrap pip with ensurepip:
 
 .. prompt:: bash
 
-    sudo /opt/python/bin/easy_install-3.8 pip
+    sudo /opt/python/bin/python3.9 -m ensurepip --altinstall
+
+.. note::
+
+    After this command, a warning about running pip as root might be printed.
+    For this specific command, this warning can be safely ignored.
 
 Continue by `creating-venv-linux`.
 
@@ -214,7 +214,7 @@ with zypper:
 
 .. prompt:: bash
 
-    sudo zypper -n install python38-base python38-pip git-core java-11-openjdk-headless nano
+    sudo zypper -n install python39-base python39-pip git-core java-11-openjdk-headless nano
     sudo zypper -n install -t pattern devel_basis
 
 Continue by `creating-venv-linux`.
@@ -235,7 +235,7 @@ We recommend adding the ``git-core`` ppa to install Git 2.11 or greater:
     sudo apt -y install software-properties-common
     sudo add-apt-repository -y ppa:git-core/ppa
 
-We recommend adding the ``deadsnakes`` ppa to install Python 3.8.1 or greater:
+We recommend adding the ``deadsnakes`` ppa to install Python 3.9:
 
 .. prompt:: bash
 
@@ -245,7 +245,7 @@ Now install the pre-requirements with apt:
 
 .. prompt:: bash
 
-    sudo apt -y install python3.8 python3.8-dev python3.8-venv python3-pip git openjdk-11-jre-headless build-essential nano
+    sudo apt -y install python3.9 python3.9-dev python3.9-venv python3-pip git openjdk-11-jre-headless build-essential nano
 
 Continue by `creating-venv-linux`.
 
@@ -269,7 +269,7 @@ Now install the pre-requirements with apt:
 
 .. prompt:: bash
 
-    sudo apt -y install python3.8 python3.8-dev python3.8-venv python3-pip git openjdk-11-jre-headless build-essential nano
+    sudo apt -y install python3.9 python3.9-dev python3.9-venv python3-pip git openjdk-11-jre-headless build-essential nano
 
 Continue by `creating-venv-linux`.
 
@@ -297,7 +297,7 @@ installing pyenv. To do this, first run the following commands:
     sudo apt -y install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev libgdbm-dev uuid-dev python3-openssl git openjdk-11-jre-headless nano
     CXX=/usr/bin/g++
 
-And then complete the rest of the installation by `installing Python 3.8 with pyenv <install-python-pyenv>`.
+And then complete the rest of the installation by `installing Python 3.9 with pyenv <install-python-pyenv>`.
 
 ----
 
@@ -312,7 +312,7 @@ Installing Python with pyenv
     If you followed one of the sections above, and weren't linked here afterwards, you should skip
     this section.
 
-On distributions where Python 3.8 needs to be compiled from source, we recommend the use of pyenv.
+On distributions where Python 3.9 needs to be compiled from source, we recommend the use of pyenv.
 This simplifies the compilation process and has the added bonus of simplifying setting up Red in a
 virtual environment.
 
@@ -327,7 +327,7 @@ Then run the following command:
 
 .. prompt:: bash
 
-    CONFIGURE_OPTS=--enable-optimizations pyenv install 3.8.10 -v
+    CONFIGURE_OPTS=--enable-optimizations pyenv install 3.9.7 -v
 
 This may take a long time to complete, depending on your hardware. For some machines (such as
 Raspberry Pis and micro-tier VPSes), it may take over an hour; in this case, you may wish to remove
@@ -339,9 +339,9 @@ After that is finished, run:
 
 .. prompt:: bash
 
-    pyenv global 3.8.10
+    pyenv global 3.9.7
 
-Pyenv is now installed and your system should be configured to run Python 3.8.
+Pyenv is now installed and your system should be configured to run Python 3.9.
 
 Continue by `creating-venv-linux`.
 
@@ -381,7 +381,7 @@ Create your virtual environment with the following command:
 
 .. prompt:: bash
 
-    python3.8 -m venv ~/redenv
+    python3.9 -m venv ~/redenv
 
 And activate it with the following command:
 

docs/install_windows.rst

@@ -11,7 +11,7 @@ Installing the pre-requirements
 Please install the pre-requirements by following instructions from one of the following subsections.
 
 The pre-requirements are:
- - Python 3.8.1 or greater; **Python 3.9 is currently not supported!**
+ - Python 3.8.1 or greater
  - Pip 18.1 or greater
  - Git 2.11+
  - Java Runtime Environment 11 (for audio support)
@@ -40,7 +40,7 @@ Then run each of the following commands:
     iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
     choco upgrade git --params "/GitOnlyOnPath /WindowsTerminal" -y
     choco upgrade visualstudio2019-workload-vctools -y
-    choco upgrade python3 -y --version 3.8.10
+    choco upgrade python3 -y --version 3.9.7
 
 For Audio support, you should also run the following command before exiting:
 
@@ -64,7 +64,7 @@ Manually installing dependencies
 
 * `MSVC Build tools <https://www.visualstudio.com/downloads/#build-tools-for-visual-studio-2019>`_
 
-* `Python 3.8.1 or greater <https://www.python.org/downloads/>`_; **Python 3.9 is currently not supported!**
+* `Python 3.8.1 or greater <https://www.python.org/downloads/>`_
 
 .. attention:: Please make sure that the box to add Python to PATH is CHECKED, otherwise
                you may run into issues when trying to run Red.
@@ -111,7 +111,7 @@ Then create your virtual environment with the following command
 
 .. prompt:: batch
 
-    py -3.8 -m venv "%userprofile%\redenv"
+    py -3.9 -m venv "%userprofile%\redenv"
 
 And activate it with the following command
 

redbot/__init__.py

@@ -227,3 +227,12 @@ if "--debug" not in _sys.argv:
     # Individual warnings - tracked in https://github.com/Cog-Creators/Red-DiscordBot/issues/3529
     # DeprecationWarning: an integer is required (got type float).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
     _warnings.filterwarnings("ignore", category=DeprecationWarning, module="importlib", lineno=219)
+    # DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10
+    #   stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr,
+    # this is a bug in CPython
+    _warnings.filterwarnings(
+        "ignore",
+        category=DeprecationWarning,
+        module="asyncio",
+        message="The loop argument is deprecated since Python 3.8",
+    )

redbot/__main__.py

@@ -85,8 +85,7 @@ def debug_info():
         os_info = platform.mac_ver()
         osver = "Mac OSX {} {}".format(os_info[0], os_info[2])
     else:
-        os_info = distro.linux_distribution()
+        osver = f"{distro.name()} {distro.version()}".strip()
-        osver = "{} {}".format(os_info[0], os_info[1]).strip()
     user_who_ran = getpass.getuser()
     info = (
         "Debug Info for Red\n\n"

redbot/core/bot.py

@@ -1230,7 +1230,7 @@ class RedBase(
             The channel to check embed settings for.
         user : `discord.abc.User`
             The user to check embed settings for.
-        command : `commands.Command`, optional
+        command : `redbot.core.commands.Command`, optional
             The command ran.
 
         Returns

redbot/core/core_commands.py

@@ -3599,8 +3599,7 @@ class Core(commands.commands._RuleDropper, commands.Cog, CoreLogic):
             os_info = platform.mac_ver()
             osver = f"Mac OSX {os_info[0]} {os_info[2]}"
         elif IS_LINUX:
-            os_info = distro.linux_distribution()
+            osver = f"{distro.name()} {distro.version()}".strip()
-            osver = f"{os_info[0]} {os_info[1]}".strip()
         else:
             osver = "Could not parse OS, report this on Github."
         user_who_ran = getpass.getuser()

setup.cfg

@@ -25,7 +25,9 @@ classifiers =
     Operating System :: MacOS :: MacOS X
     Operating System :: Microsoft :: Windows
     Operating System :: POSIX :: Linux
+    Programming Language :: Python :: 3 :: Only
     Programming Language :: Python :: 3.8
+    Programming Language :: Python :: 3.9
     Topic :: Communications :: Chat
 license_files =
     LICENSE
@@ -33,89 +35,93 @@ license_files =
 
 [options]
 packages = find_namespace:
-python_requires = >=3.8.1,<3.9
+python_requires = >=3.8.1,<3.10
 include_package_data = True
 install_requires =
-    aiohttp==3.7.3
+    aiohttp==3.7.4.post0
     aiohttp-json-rpc==0.13.3
-    aiosqlite==0.16.1
+    aiosqlite==0.17.0
     appdirs==1.4.4
-    apsw-wheels==3.34.1.post1
+    apsw-wheels==3.36.0.post1
     async-timeout==3.0.1
-    attrs==20.3.0
+    attrs==21.2.0
-    Babel==2.9.0
+    Babel==2.9.1
-    chardet==3.0.4
+    cffi==1.14.6
-    click==7.1.2
+    chardet==4.0.0
+    click==8.0.1
     colorama==0.4.4
     commonmark==0.9.1
-    contextlib2==0.6.0.post1
+    contextlib2==21.6.0
     discord.py==1.7.3
-    distro==1.5.0; sys_platform == "linux"
+    distro==1.6.0; sys_platform == "linux"
     fuzzywuzzy==0.18.0
-    idna==2.10
+    idna==3.2
-    Markdown==3.3.3
+    Markdown==3.3.4
     multidict==5.1.0
     psutil==5.8.0
-    PyNaCl==1.3.0
+    pycparser==2.20
-    Pygments==2.7.4
+    Pygments==2.10.0
-    python-dateutil==2.8.1
+    PyNaCl==1.4.0
+    python-dateutil==2.8.2
     python-Levenshtein-wheels==0.13.2
     pytz==2021.1
     PyYAML==5.4.1
     Red-Lavalink==0.8.1
-    rich==9.9.0
+    rich==10.9.0
     schema==0.7.4
-    six==1.15.0
+    six==1.16.0
-    typing-extensions==3.7.4.3
+    typing-extensions==3.10.0.2
-    uvloop==0.15.0; sys_platform != "win32" and platform_python_implementation == "CPython"
+    uvloop==0.16.0; sys_platform != "win32" and platform_python_implementation == "CPython"
     yarl==1.6.3
 
 [options.extras_require]
 docs =
     alabaster==0.7.12
-    certifi==2020.12.5
+    certifi==2021.5.30
+    charset-normalizer==2.0.4
     docutils==0.16
     imagesize==1.2.0
-    Jinja2==2.11.3
+    Jinja2==3.0.1
-    MarkupSafe==1.1.1
+    MarkupSafe==2.0.1
-    packaging==20.9
+    packaging==21.0
     pyparsing==2.4.7
-    requests==2.25.1
+    requests==2.26.0
     snowballstemmer==2.1.0
-    Sphinx==3.4.3
+    Sphinx==4.1.2
-    sphinx-prompt==1.4.0
+    sphinx-prompt==1.5.0
-    sphinx-rtd-theme==0.5.1
+    sphinx-rtd-theme==0.5.2
     sphinxcontrib-applehelp==1.0.2
     sphinxcontrib-devhelp==1.0.2
-    sphinxcontrib-htmlhelp==1.0.3
+    sphinxcontrib-htmlhelp==2.0.0
     sphinxcontrib-jsmath==1.0.1
     sphinxcontrib-qthelp==1.0.3
-    sphinxcontrib-serializinghtml==1.1.4
+    sphinxcontrib-serializinghtml==1.1.5
     sphinxcontrib-trio==1.1.2
-    urllib3==1.26.3
+    urllib3==1.26.6
 postgres =
-    asyncpg==0.22.0
+    asyncpg==0.24.0
 style =
     black==20.8b1
     mypy-extensions==0.4.3
-    pathspec==0.8.1
+    pathspec==0.9.0
-    regex==2020.11.13
+    regex==2021.8.28
     toml==0.10.2
-    typed-ast==1.4.2
+    typed-ast==1.4.3
 test =
-    astroid==2.4.2
+    astroid==2.7.3
     iniconfig==1.1.1
-    isort==5.7.0
+    isort==5.9.3
-    lazy-object-proxy==1.4.3
+    lazy-object-proxy==1.6.0
     mccabe==0.6.1
-    packaging==20.9
+    packaging==21.0
-    pluggy==0.13.1
+    platformdirs==2.3.0
+    pluggy==1.0.0
     py==1.10.0
-    pylint==2.6.0
+    pylint==2.10.2
     pyparsing==2.4.7
-    pytest==6.2.2
+    pytest==6.2.5
-    pytest-asyncio==0.14.0
+    pytest-asyncio==0.15.1
-    pytest-mock==3.5.1
+    pytest-mock==3.6.1
     toml==0.10.2
     wrapt==1.12.1
 all =

setup.py

@@ -1,4 +1,10 @@
+import os
+import sys
 from setuptools import setup
 
-# Metadata and options defined in setup.cfg
+if os.getenv("TOX_RED", False) and sys.version_info >= (3, 10):
-setup()
+    # We want to be able to test Python versions that we do not support yet.
+    setup(python_requires=">=3.8.1")
+else:
+    # Metadata and options defined in setup.cfg
+    setup()

tests/core/test_config.py

@@ -531,7 +531,7 @@ async def test_config_value_atomicity(config):
                 await asyncio.sleep(0.1)
                 await config.foo.set(foo)
 
-        tasks.append(func())
+        tasks.append(asyncio.create_task(func()))
 
     await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)
 
@@ -549,7 +549,7 @@ async def test_config_ctxmgr_atomicity(config):
                 foo.append(0)
                 await asyncio.sleep(0.1)
 
-        tasks.append(func())
+        tasks.append(asyncio.create_task(func()))
 
     await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)
 

tests/core/test_version.py

@@ -1,5 +1,9 @@
 import importlib.metadata
 import pkg_resources
+import os
+import sys
+
+import pytest
 
 from redbot import core
 from redbot.core import VersionInfo
@@ -39,7 +43,24 @@ def test_version_info_gt():
     assert VersionInfo.from_str(version_tests[1]) > VersionInfo.from_str(version_tests[0])
 
 
-def test_python_version_has_upper_and_lower_bound():
+def test_python_version_has_lower_bound():
+    """
+    Due to constant issues in support with Red being installed on a Python version that was not
+    supported by any Red version, it is important that we have both an upper and lower bound set.
+    """
+    requires_python = importlib.metadata.metadata("Red-DiscordBot")["Requires-Python"]
+    assert requires_python is not None
+
+    # `pkg_resources` needs a regular requirement string, so "x" serves as requirement's name here
+    req = pkg_resources.Requirement.parse(f"x{requires_python}")
+    assert any(op in (">", ">=") for op, version in req.specs)
+
+
+@pytest.mark.skipif(
+    os.getenv("TOX_RED", False) and sys.version_info >= (3, 10),
+    reason="Testing on yet to be supported Python version.",
+)
+def test_python_version_has_upper_bound():
     """
     Due to constant issues in support with Red being installed on a Python version that was not
     supported by any Red version, it is important that we have both an upper and lower bound set.
@@ -50,4 +71,3 @@ def test_python_version_has_upper_and_lower_bound():
     # `pkg_resources` needs a regular requirement string, so "x" serves as requirement's name here
     req = pkg_resources.Requirement.parse(f"x{requires_python}")
     assert any(op in ("<", "<=") for op, version in req.specs)
-    assert any(op in (">", ">=") for op, version in req.specs)

tox.ini

@@ -6,6 +6,7 @@
 [tox]
 envlist =
     py38
+    py39
     docs
     style
 skip_missing_interpreters = True
@@ -16,6 +17,8 @@ whitelist_externals =
     pytest
     pylint
 extras = voice, test
+setenv =
+    TOX_RED = 1
 commands =
     python -m compileall ./redbot/cogs
     pytest
@@ -27,6 +30,7 @@ whitelist_externals =
     pytest
 extras = voice, test, postgres
 setenv =
+    TOX_RED = 1
     RED_STORAGE_TYPE=postgres
 passenv =
     # Use the following env vars for connection options, or other default options described here: