37 Commits

Author SHA1 Message Date
jack1142
36e2cde04d Move [p]backup command to cli command - redbot-setup backup (#3235)
* refactor: replace backup command with cli command

* chore(changelog): add towncrier entries
2020-01-02 09:03:32 -05:00
jack1142
f3e7c2028c [Setup] Use instance name in default data path (#3171)
* enhance(setup): use instance name in default data path

* chore(changelog): add towncrier entries

* enhance(setup): tell user that instance name is case-sensitive
2020-01-02 08:59:22 -05:00
jack1142
ab747d2432 [Utils] Privatize internal utils (#3240)
* refactor(utils): privatize some utils

* chore(changelog): add towncrier entry

* refactor: update internal utils imports
2020-01-02 08:44:55 -05:00
Michael H
dd899c804a
Remove the mongo driver (#3099)
* kills mongo

* changelog

* more refeences to mongo needed to go
2019-11-08 14:48:04 -05:00
jack1142
1651de1305 [Core] Add redbot --edit cli flag (replacement for [p]set owner&token) (#3060)
* feat(core): add `redbot --edit` cli flag

* chore(changelog): add towncrier entries

* refactor(core): clean up `redbot --edit`, few fixes

* fix(core): prepare for review

* chore(changelog): update towncrier entry to use double ticks :p

* style(black): ugh, Sinbad's git hook isn't perfect (using worktrees)

* fix: Address Flame's first review
2019-11-08 12:07:32 -05:00
Toby Harradine
7e9b1b87e6 Allow keeping data in redbot-setup delete (#2965)
* Allow keeping data in `redbot-setup delete`

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add changelog entry

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-11-06 12:24:54 -05:00
jack1142
b490942bcd [Core] Various fixes to redbot-setup delete (#2958)
- ``redbot-setup delete`` no longer errors about "unexpected keyword argument" (fix #2955)
- ``redbot-setup delete`` no longer prompts about backup when user passes ``--no-prompt`` option (fix #2956)
- ``--[no-]backup``, ``--[no-]drop-db`` and ``--[no-]remove-datapath`` in ``redbot-setup delete`` command are now on/off flags. ``--no-prompt`` was changed to flag too, but it doesn't have ``--prompt`` equivalent as there's no much point in having it
- ``redbot-setup`` now uses `click.confirm` for confirmation prompts and thy now also have default values for user convenience
2019-08-30 08:23:31 +10:00
Toby Harradine
d1a46acc9a PostgreSQL driver, tests against DB backends, and general drivers cleanup (#2723)
* PostgreSQL driver and general drivers cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Make tests pass

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add black --target-version flag in make.bat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Rewrite postgres driver

Most of the logic is now in PL/pgSQL.

This completely avoids the use of Python f-strings to format identifiers into queries. Although an SQL-injection attack would have been impossible anyway (only the owner would have ever had the ability to do that), using PostgreSQL's format() is more reliable for unusual identifiers. Performance-wise, I'm not sure whether this is an improvement, but I highly doubt that it's worse.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Reformat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix PostgresDriver.delete_all_data()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Clean up PL/pgSQL code

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More PL/pgSQL cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* PL/pgSQL function optimisations

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Ensure compatibility with PostgreSQL 10 and below

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More/better docstrings for PG functions

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix typo in docstring

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Return correct value on toggle()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Use composite type for PG function parameters

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix JSON driver's Config.clear_all()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Correct description for Mongo tox recipe

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix linting errors

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update dep specification after merging bumpdeps

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add towncrier entries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update from merge

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Mention [postgres] extra in install docs

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Support more connection options and use better defaults

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Actually pass PG env vars in tox

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Replace event trigger with manual DELETE queries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-08-26 22:02:26 -04:00
jack1142
942dca43d3 [Downloader]: RepoManager: don't load repos in __init__ (#2829)
Loading repos is already done in initialize() method.
This could actually turn out badly
if both
of git processes would touch the same repo at the same time.

This also fixes create_backup in
setup.py - now it properly generates repos.json
2019-07-02 11:57:30 +10:00
Toby Harradine
bff7e214ab Kill JsonIO and fix JSON driver caching issues (#2796)
* Kill JsonIO and fix JSON driver caching issues

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Ensure lock covers critical region in set()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Make tests pass

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Use pickle over deepcopy in Config

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix temp instance creation

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Serialise value before doing anything in set()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-06-27 14:49:45 -04:00
Michael H
16443c8cc0 [CI] Improve automated checks (#2702)
* same stuff, but with some more spurious error supression

* fix issue in permissions found in this

* fix a few more spurious errors

* fix another issue

* semi-spurious error fixes

* .

* formatting

* move this to properly log

* distutils import + virtualenv

* more fixes
2019-06-02 19:42:57 +02:00
jack1142
cdcde26dfc [Setup] Fix: wrong var used for instance data in remove_instance (#2709) 2019-05-19 10:13:01 -04:00
Will
95d5ec5f0e
[Setup] Fix the mongo name to create instances using new driver (#2594)
* oops

* damn you
2019-04-22 18:33:52 -04:00
Will
6c296a9a17 [V3 Setup] Overhaul backend conversion process through setup scripts (#2579)
* swap to click for setup

* Initial changes

* expose some stuff to allow for per-driver optimizations

* overwrite base config

* add red log

* add one print juuuust in case

* fix this

* thanks kowlin

* damn

* oops

* fix thing

* partial commit

* Working mongo -> json conversion, it sucks tho

* remove unused line

* Wrote initial optimized json importer

* optimized json importer

* remove useless line

* update mongo to json converter

* lets try writing the correct entry

* oops

* style fix

* add some garbage data filters going from old mongo to json

* ignore garbage data in mongov2 conversions

* simplify code a bit and add a completion message

* missed one

* Update pipfile lock

* Lock click version
2019-04-20 20:10:44 -04:00
Will
c82ac5ae68
Add some errors for backend conversions and only allow MongoV2 creation (#2570)
* Add some errors for conversions and only allow mongoV2 creation

* Add another message

* Fixed message to be more clear
2019-04-10 20:42:28 -04:00
Will
1cd7e41f33
[V3 Config] Update Mongo document organization to bypass doc size restriction (#2536)
* modify config to use identifier data class and update json driver

* move identifier data attributes into read only properties

* Update mongo get and set methods

* Update get/set to use UUID separately, make clear work

* Remove not implemented and fix get_raw

* Update remaining untouched get/set/clear

* Fix get_raw

* Finally fix get_raw and set_raw

* style

* This is better

* Sorry guys

* Update get behavior to handle "all" calls as expected

* style again

* Why do you do this to me

* style once more

* Update mongo schema
2019-04-03 09:04:47 -04:00
Toby Harradine
ec4c325efd
Guard parsing of CLI args in launcher, setup scripts (#2432)
Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-02-08 18:08:10 +11:00
Toby Harradine
0d4e6a0865
Fix MongoDB to JSON migration and warn about Mongo driver (#2373)
Resolves #2372.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-01-18 11:26:33 +11:00
Toby Harradine
2bd05a5a04
Fix JSON to Mongo migration (#2346)
Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2018-12-29 10:51:31 +11:00
Tobotimus
622382f425 [V3] Clean up some ugly auto-formatted strings (#1753)
* [V3] Cleanup some ugly auto-formatted strings

* Reformat
2018-06-02 18:01:14 -04:00
palmtree5
7775b16199 [V3] Optimize the backup command (#1666)
* [V3 Core] Enhance [p]backup to exclude some files

* Backup the repo list too

* Lol Sinbad's pre-commit hook

* Add option of sending the backup to the owner via DM

* Drop an unnecessary config object in RepoManager

* Move the backup functionality in redbot-setup to the new stuff

* More work on implementation, including backing up the instance data
2018-05-28 06:56:28 +02:00
Tobotimus
3337a9cbab [V3 Launcher] Fix error when removing Mongo instance (#1710)
* [V3 Launcher] Fix error when removing Mongo instance

Fixes #1573

* Fix issue causing style check to fail

* Remove unneeded whitespace
2018-05-27 16:08:25 -08:00
Michael H
b88b5a2601 [V3] Update code standards (black code format pass) (#1650)
* ran black: code formatter against `redbot/` with `-l 99`

* badge
2018-05-14 15:33:24 -04:00
retke
95ef5d6348 [V3 Launcher] Reinstall Red option (#1536)
* [V3 Launcher] Reinstall Red option

* [V3 Setup] Divided remove_instance function

* Removing changes from another PR

* Indent fails fix

* use remove_instance_interaction for --delete

* Fix some issues with remove_instance

removed `index: int` because what's being passed there is a string
data -> instance_data

* bug fixes, working version
2018-05-04 08:01:37 +02:00
palmtree5
580aeea4e2 [V3 Instance setup] fix an issue getting credentials to convert Mongo to JSON (#1514)
* [V3 Instance setup] fix an issue getting credentials to convert Mongo to JSON

* Attempt a fix at data conversion issues
2018-04-16 10:03:21 +02:00
palmtree5
c2e7bfa641 [V3 Instance setup] Implement backup support for instances using Mongo (#1470)
* [V3 Instance setup] import backup support for instances using Mongo

* remove_instance is now a coroutine + remove NIE

* move Mongo import down

* move another import
2018-03-28 20:12:26 -04:00
palmtree5
9ecea9e1d5 [V3 instance setup] Warn and request confirmation if instance name already in use (#1469) 2018-03-28 19:13:34 -04:00
palmtree5
59281afe90 [V3 Instance setup] cog identifiers are strings, not ints (#1448) 2018-03-22 19:11:27 -04:00
palmtree5
eb3b6346bb [V3] Set r/w access before deleting files (#1412)
* [V3] set access before deleting

* [V3] move+rename do_delete and use in repo removal in downloader
2018-03-20 18:46:15 -04:00
palmtree5
a8f4659552
[V3 Instance Setup] Storage swapping (#1421)
* [V3 Instance Setup] start work on storage swapping

* This should do the trick for Mongo -> JSON

* Fix typo

* Fix a few more typos

* resolve the data path

* Upsert the imported data

* need a list of the documents

* to_list is a coro
2018-03-16 13:37:12 -08:00
palmtree5
5bdb455bc0 [V3 Instance setup] Change backup time separator for Windows (#1419) 2018-03-12 18:48:13 -04:00
palmtree5
01e8579215 [V3 instance setup] allow removing instances without taking a backup + error handling for missing data path (#1277) 2018-02-15 02:39:29 +01:00
palmtree5
0c85e0fe22 [V3] Add support for removing instances (#1270)
* [V3 Instance Setup] add a way of removing instances

* [V3 Launcher] add instance removal support to the launcher
2018-01-29 04:56:28 +01:00
Michael H
de3a057e6e [V3] Allow root to own data (#1264) 2018-01-28 18:41:01 -09:00
Michael H
b80c327e35 For Linux service users, (#1236)
* For Linux service users,
use `appdirs.AppDirs().site_data_dir`
instead of `appdirs.AppDirs().user_config_dir

* somewhat important for this to be changed consistently

* need to import os to use it

* travis doesn't like how I was checking in config_dir had been set.
2018-01-15 05:35:33 +01:00
Will
5d48e6f3b1 [V3 Config] Add MongoDB driver (#972)
* Basic Mongo Driver

* Update docstrings

* WIP motor support

* Use motor

* Update docs, add selective importer

* Make use of selective importer

* Fix docs

* Fix config storage location for JSON

* Add delimiters in the drivers doc section

* Make async things async

* Add basic config information for mongo driver

* get info from basic setup into config

* IT WORKS

* Add dependency for RJM's PR.
2017-10-20 19:12:45 -04:00
Will
d69fd63da7 [V3 Everything] Package bot and write setup scripts (#964)
Ya'll are gonna hate me.

* Initial modifications

* Add initial setup.py

* working setup py help

* Modify setup file to package stuff

* Move a bunch of shit and fix imports

* Fix or skip tests

* Must add init files for find_packages to work

* Move main to scripts folder and rename

* Add shebangs

* Copy over translation files

* WORKING PIP INSTALL

* add dependency information

* Hardcoded version for now, will need to figure out a better way to do this

* OKAY ITS FINALLY FUCKING WORKING

* Add this guy

* Fix stuff

* Change readme to rst

* Remove double sentry opt in

* Oopsie

* Fix this thing

* Aaaand fix test

* Aaaand fix test

* Fix core cog importing and default cog install path

* Adjust readme

* change instance name from optional to required

* Ayyy let's do more dependency injection
2017-09-08 23:14:32 -04:00