38 Commits

Author SHA1 Message Date
Michael H
dd899c804a
Remove the mongo driver (#3099)
* kills mongo

* changelog

* more refeences to mongo needed to go
2019-11-08 14:48:04 -05:00
jack1142
8267ad9aab [Docs] Add missing descriptions for function returns (#3054)
* docs: add missing descriptions for function returns

* chore(changelog): add towncrier entry
2019-10-21 22:41:35 -04:00
Toby Harradine
326c53d6c4
Quick fix for postgres config details prompt (#2951)
Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-08-27 14:25:05 +10:00
Toby Harradine
d1a46acc9a PostgreSQL driver, tests against DB backends, and general drivers cleanup (#2723)
* PostgreSQL driver and general drivers cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Make tests pass

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add black --target-version flag in make.bat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Rewrite postgres driver

Most of the logic is now in PL/pgSQL.

This completely avoids the use of Python f-strings to format identifiers into queries. Although an SQL-injection attack would have been impossible anyway (only the owner would have ever had the ability to do that), using PostgreSQL's format() is more reliable for unusual identifiers. Performance-wise, I'm not sure whether this is an improvement, but I highly doubt that it's worse.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Reformat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix PostgresDriver.delete_all_data()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Clean up PL/pgSQL code

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More PL/pgSQL cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* PL/pgSQL function optimisations

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Ensure compatibility with PostgreSQL 10 and below

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More/better docstrings for PG functions

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix typo in docstring

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Return correct value on toggle()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Use composite type for PG function parameters

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix JSON driver's Config.clear_all()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Correct description for Mongo tox recipe

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix linting errors

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update dep specification after merging bumpdeps

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add towncrier entries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update from merge

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Mention [postgres] extra in install docs

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Support more connection options and use better defaults

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Actually pass PG env vars in tox

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Replace event trigger with manual DELETE queries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-08-26 22:02:26 -04:00
Draper
556af32bb5 Stop saving JSON files with indents (#2921)
Stop saving json files with 4 indents, this will significantly reduce file size and improve `.set()` performance for config saves when using the JSON driver.

Signed-off-by: Draper <guyreis96@gmail.com>
2019-08-07 10:13:31 +10:00
Toby Harradine
af096bc1cc Config locks (#2654)
* Config locks

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add locks for all_XXX

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Remove a word

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add acquire_lock kwarg for value context manager

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add towncrier entry

Signed-off-by: Toby <tobyharradine@gmail.com>

* Fix issues with `get_custom_lock` and `get_members_lock`

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-07-23 16:50:07 -04:00
Toby Harradine
f2039300c2 [Mongo] Use escaped identifiers to extract inner value (#2832)
This was causing a KeyError to be raised whenever a key containing $ or . was part of the identifiers path, even if the value was actually in the dict.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-07-05 18:02:18 -04:00
Toby Harradine
bff7e214ab Kill JsonIO and fix JSON driver caching issues (#2796)
* Kill JsonIO and fix JSON driver caching issues

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Ensure lock covers critical region in set()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Make tests pass

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Use pickle over deepcopy in Config

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix temp instance creation

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Serialise value before doing anything in set()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-06-27 14:49:45 -04:00
Toby Harradine
71d0bd0d07
Various Config and Mongo Driver fixes (#2795)
- Fixes defaults being mixed into custom groups above the document level when doing `Group.all()`
- Fixes `Config.clear_all()` with Mongo driver
- Fixes `Group.set()` with Mongo driver on custom groups above the document level
- Fixes `IdentifierData.custom_group_data` being set to the wrong thing in `BaseDriver.import/export_data` (although this was an inconsequential bug)

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2019-06-24 12:55:49 +10:00
Will
165e40c0db [Mongo] Unescape dict keys when rebuild (#2671)
* Unescape dict keys when rebuilding

* update pipfile

* Really update pipfile now
2019-05-14 23:50:51 -04:00
Will
05e2851c67
[Config] Migrate UUID for JSON backend (#2604)
* Migrate UUID for JSON backend

* Interesting...

* black

* Simplify UUID creation
2019-04-24 16:03:27 -04:00
Will
f20a174038 Fix the check to raise the correct message (#2603) 2019-04-24 21:24:31 +02:00
zephyrkul
874204bf18 [mongo setup] utilize getpass (#2593) 2019-04-21 07:04:08 -08:00
Will
6c296a9a17 [V3 Setup] Overhaul backend conversion process through setup scripts (#2579)
* swap to click for setup

* Initial changes

* expose some stuff to allow for per-driver optimizations

* overwrite base config

* add red log

* add one print juuuust in case

* fix this

* thanks kowlin

* damn

* oops

* fix thing

* partial commit

* Working mongo -> json conversion, it sucks tho

* remove unused line

* Wrote initial optimized json importer

* optimized json importer

* remove useless line

* update mongo to json converter

* lets try writing the correct entry

* oops

* style fix

* add some garbage data filters going from old mongo to json

* ignore garbage data in mongov2 conversions

* simplify code a bit and add a completion message

* missed one

* Update pipfile lock

* Lock click version
2019-04-20 20:10:44 -04:00
Will
0652dd344b
[V3 Mongo] Correct dictionary rebuilding process for global all case (#2581) 2019-04-13 15:51:49 -04:00
Will
8b3c3e89e9
[V3 Mongo] Fix all behavior (#2580) 2019-04-13 15:24:50 -04:00
Will
c82ac5ae68
Add some errors for backend conversions and only allow MongoV2 creation (#2570)
* Add some errors for conversions and only allow mongoV2 creation

* Add another message

* Fixed message to be more clear
2019-04-10 20:42:28 -04:00
Will
0852d1be9f
[V3 Config] Require custom group initialization before usage (#2545)
* Require custom group initialization before usage and write that data to disk

* Style

* add tests

* remove custom info update method from drivers

* clean up remnant

* Turn config objects into a singleton to deal with custom group identifiers

* Fix dumbassery

* Stupid stupid stupid
2019-04-04 21:47:08 -04:00
Will
1cd7e41f33
[V3 Config] Update Mongo document organization to bypass doc size restriction (#2536)
* modify config to use identifier data class and update json driver

* move identifier data attributes into read only properties

* Update mongo get and set methods

* Update get/set to use UUID separately, make clear work

* Remove not implemented and fix get_raw

* Update remaining untouched get/set/clear

* Fix get_raw

* Finally fix get_raw and set_raw

* style

* This is better

* Sorry guys

* Update get behavior to handle "all" calls as expected

* style again

* Why do you do this to me

* style once more

* Update mongo schema
2019-04-03 09:04:47 -04:00
Toby Harradine
f7b1f9f0dc
[MongoDB] Escape special characters in keys (#2212)
Essentially resolves #2038, although this is escaping and not rejecting keys as that issue implies.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
2018-10-11 11:20:42 +11:00
zephyrkul
de4b42a11e [mongo] fix missing URI variable assignment (#2186) 2018-10-06 14:48:03 +10:00
Ryan
b6c8be5f43 [MongoDB] Support mongodb+srv protocol (#2159) 2018-10-01 16:49:29 +10:00
Kowlin
7971c02dc5 Escape user and passwords for mongo (#2111)
* Escape user and passwords for mongo.

* Quote -> Quote_Plus

* formatting
2018-09-09 23:07:40 +10:00
Tobotimus
0298b53803 [V3 JSON] Drivers deepcopy input/output data (#1855)
* [V3 JSON] Return deepcopy in JSON driver

* Add a test

* foo not bar

* Add a test for setting and then mutating

* Resolve issue for setting and mutating as well

* Reformat
2018-06-11 12:31:01 -04:00
Eslyium
60a72b2ba4 [V3] Cleanup quotes in cogs (#1782)
* Cleanup quotes in cogs

* More quote cleanup that I missed

fixed a little bit of grammar here and there as well.

* [V3 Warnings] Change allowcustomreasons docstring

To help not confuse users who would believe that the command would use allow or disallow.

* Run black reformat
2018-06-07 00:42:59 -04:00
Tobotimus
14cc701b25 [V3] Update black version and reformat (#1745)
* Update black version and reformat

* Pin black in extras_require
2018-05-29 18:37:00 -08:00
Michael H
b88b5a2601 [V3] Update code standards (black code format pass) (#1650)
* ran black: code formatter against `redbot/` with `-l 99`

* badge
2018-05-14 15:33:24 -04:00
Will
29ce2401ca [V3 Config] Fix unloading and implement singleton driver (#1458)
* Add the identifier as an initialization parameter

* Remove config object singleton and opt for a shared JSON datastore

* Fix bot unloading to deal with memory leaks

* Fix tests

* Fix clear all bug
2018-04-02 16:47:27 -08:00
Will
c9281f734b [V3 Config] Fix clear throwing errors (#1374) 2018-03-06 10:28:37 +11:00
Thomas Mercurio
b10b746d9e [V3 Config] Correct Mongo connection URI without credentials (#1362)
* Correct connection string without credentials

* Remove extra whitespace.
2018-03-02 18:39:45 +01:00
Will
004f42bf6e
[V3 Config] Limit connections to Mongo DB (#1356) 2018-02-26 15:34:30 -05:00
Will
41e0d105f6
[V3 Config] Make Mongo work with any db (#1355) 2018-02-26 15:28:20 -05:00
Will
e13de0950c [V3 Config] Implement a true clear method (#1344)
* Add clear methods to drivers

* Update config clear methods, add to Value

* Add test
2018-02-25 21:07:28 -09:00
Will
f9d846a704 [V3 Config] Driver code initial cleanup (#1315)
* Remove get_driver

* Rename self.driver to self._driver

* Do not unnecessarily pass the cog identifier

* Remove unused import

* Fix type annotation

* Missed a keyword rename

* Modify signature of get/set methods in drivers
2018-02-18 18:30:32 -09:00
Tobotimus
9b1018fa96 [V3 JSON Driver] Don't save json if it exists (#1133) 2017-12-03 17:54:24 -09:00
Will
5d48e6f3b1 [V3 Config] Add MongoDB driver (#972)
* Basic Mongo Driver

* Update docstrings

* WIP motor support

* Use motor

* Update docs, add selective importer

* Make use of selective importer

* Fix docs

* Fix config storage location for JSON

* Add delimiters in the drivers doc section

* Make async things async

* Add basic config information for mongo driver

* get info from basic setup into config

* IT WORKS

* Add dependency for RJM's PR.
2017-10-20 19:12:45 -04:00
Tobotimus
c80684a129 [V3] NumPy Docstrings (#1032)
* ALL THE DOCSTRINGS

* Remove imports in drivers package

* Fixed build warnings
2017-10-17 22:01:59 -04:00
Will
d69fd63da7 [V3 Everything] Package bot and write setup scripts (#964)
Ya'll are gonna hate me.

* Initial modifications

* Add initial setup.py

* working setup py help

* Modify setup file to package stuff

* Move a bunch of shit and fix imports

* Fix or skip tests

* Must add init files for find_packages to work

* Move main to scripts folder and rename

* Add shebangs

* Copy over translation files

* WORKING PIP INSTALL

* add dependency information

* Hardcoded version for now, will need to figure out a better way to do this

* OKAY ITS FINALLY FUCKING WORKING

* Add this guy

* Fix stuff

* Change readme to rst

* Remove double sentry opt in

* Oopsie

* Fix this thing

* Aaaand fix test

* Aaaand fix test

* Fix core cog importing and default cog install path

* Adjust readme

* change instance name from optional to required

* Ayyy let's do more dependency injection
2017-09-08 23:14:32 -04:00