Merge branch 'praw4'

item-del-count
Scott 8 years ago
commit 1467a8e75a

@ -10,11 +10,11 @@ prior to deletion. In fact you can actually turn off deletion all together and j
about Shreddit) but this will increase how long it takes the script to run as it will be going over all of your messages
every run.
## User Login deprecation
## Important New Changes (as of Dec 2016)
Reddit intends to disable username-password based authentication to access its APIs in the near future. You can specify
your username and password in the `shreddit.yml` or the `praw.ini` to make it work **FOR NOW**. But consider looking at
the [OAuth2 instructions](#oauth2-instructions) if you intend to use this program in the future.
Due to deprecation of the PRAW 3.x library, Shreddit is using PRAW 4. This requires that OAuth be used to authenticate.
Thankfully, however, it is much easier than in previous versions. If you are upgrading, [please review the usage section
to ensure that you have set up credentials correctly.](#configuring-credentials)
## Pip Installation
@ -31,10 +31,55 @@ installation.
## Usage
After installing the `shreddit` command line utility, the first step is setting up the tool's configuration file. Simply
typing `shreddit` will print a message with an example config. Copy the message from `---` onwards and save it as
`shreddit.yml`. Now, the tool may be used by simply typing `shreddit` from this directory. Alternatively, if you named
the configuration file something different such as `config.yml`, you may use it with `shreddit -c config.yml`.
After installing the `shreddit` command line utility, the first step is setting up the tool's configuration files.
Simply typing `shreddit -g` will generate configs. After configuring credentials, running the tool with the `shreddit`
command will begin the tool's operation.
### Configuring Credentials
Running `shreddit -g` will generate a blank praw.ini file that looks like this:
```
# Credentials go here. Fill out default, or provide one or more names and call shreddit with the -u option to specify
# which set to use.
[default]
client_id=
client_secret=
username=
password=
```
**You must provide values for each of these.** As strange as it may seem to provide both a username/password pair *and*
a client id/secret pair, that is how the Reddit API does "OAuth" script applications.
Username and password are simply your Reddit login credentials for the account that will be used. However, to obtain the
client ID and secret, follow these steps (taken from
[PRAW documentation](http://praw.readthedocs.io/en/latest/getting_started/authentication.html#script-application)):
1. Open your Reddit application preferences by clicking [here](https://www.reddit.com/prefs/apps/).
2. Add a new application. It doesn't matter what it's named, but calling it "shreddit" makes it easier to remember.
3. Select "script".
4. Redirect URL does not matter for script applications, so enter something like http://127.0.0.1:8080
5. Once created, you should see the name of your application followed by 14 character string. Enter this 14 character
string as your `client_id`.
6. Copy the 27 character "secret" string into the `client_secret` field.
Finally, your praw.ini should look like this (with fake data provided here):
```
[default]
client_id=f3FaKeD4t40PsJ
client_secret=dfK3pfMoReFAkEDaTa123456789
username=testuser
password=123passwordgoeshere123
```
Keep your praw.ini either in the current directory when running `shreddit`, or in one of the config folders
[described here](http://praw.readthedocs.io/en/latest/getting_started/configuration/prawini.html) such as
`~/.config` in Linux or `%APPDATA%` in Windows.
To use more than one account, you can add multiple profiles instead of just `[default]` and use the `-u` option to
`shreddit` to choose which one each time.
### Automating
@ -43,6 +88,9 @@ user's crontab settings.
**Examples:**
The following examples require that the PRAW configuration file is located in the config directory. See [this PRAW
documentation](http://praw.readthedocs.io/en/latest/getting_started/configuration/prawini.html) for more information.
- Run every hour on the hour
`0 * * * * shreddit -c <full path to shreddit.yml>`
@ -60,7 +108,7 @@ If virtualenv was used, be sure to add `source /full/path/to/venv/bin/activate &
```
$ shreddit --help
usage: shreddit [-h] [-c CONFIG] [-p PRAW] [-t]
usage: app.py [-h] [-c CONFIG] [-g] [-u USER]
Command-line frontend to the shreddit library.
@ -68,8 +116,10 @@ optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Config file to use instead of the default shreddit.yml
-p PRAW, --praw PRAW PRAW config (if not ./praw.ini)
-t, --test-oauth Perform OAuth test and exit
-g, --generate-configs
Write shreddit and praw config files to current
directory.
-u USER, --user USER User section from praw.ini if not default
```
## For Windows users
@ -81,23 +131,6 @@ optional arguments:
3. Open a new command prompt and verify that the `shreddit` command works before moving on to the [usage](#usage)
section.
## OAuth2 Instructions
1. Visit: https://www.reddit.com/prefs/apps
2. Click on 'Create app'.
- Fill in the name and select the 'script' option
- Under "redirect uri" put http://127.0.0.1:65010
3. Copy from or rename `praw.ini.example` to `praw.ini` and open it. Enter the values from the Reddit page.
- oauth\_client\_id = { The ID displayed next to the icon thingy (under
"personal use script") }
- oauth\_client\_secret = { The secret }
- oauth\_redirect\_uri = http://127.0.0.1:65010
- Save the file.
4. Run `python get_secret.py` in the command prompt.
5. Your browser will open to a page on Reddit listing requested permissions.
6. Click 'Allow'.
## Caveats
- Certain limitations in the Reddit API and the PRAW library make it difficult to delete more than 1,000 comments.
@ -105,3 +138,4 @@ optional arguments:
- We are relying on Reddit admin words that they do not store edits, deleted posts are still stored in the database
they are merely inaccessible to the public.

@ -1,68 +0,0 @@
#!/usr/bin/env python
'''
So I heard you want to use OAuth2? This is a helper tool that gets the
authenticaton code for you and fires it into praw.ini.
How to use:
- Visit: https://www.reddit.com/prefs/apps
- Create new "script", under "redirect uri" put http://127.0.0.1:65010
- Open praw.ini
- oauth_client_id = { The ID displayed under the icon thingy }
- oauth_client_secret = { The secret }
- oauth_redirect_uri = http://127.0.0.1:65010
- Run this script
- Your browser will open to a page on Reddit listing requested perms
- Click permit
'''
import praw
import webbrowser
from warnings import warn
from praw.errors import HTTPException, OAuthAppRequired
from tornado import gen, web
from tornado.ioloop import IOLoop
from tornado.httpserver import HTTPServer
r = praw.Reddit('Shreddit refresh token grabber')
class Page(web.RequestHandler):
def get(self):
code = self.get_argument("code", default=None, strip=False)
self.write("Success! Your code: %s<br> \
It will now be appended to praw.ini and you \
should be able to enjoy Shreddit without storing \
your user / pass anywhere." % code)
IOLoop.current().stop()
self.login(code)
def login(self, code):
deets = r.get_access_information(code)
print("oauth_refresh_token: %s" % deets['refresh_token'])
r.set_access_credentials(**deets)
with open('praw.ini', mode='a') as fh:
fh.write('oauth_refresh_token = %s' % deets['refresh_token'])
print("Refresh token written to praw.ini")
application = web.Application([(r"/", Page)])
try:
r.refresh_access_information()
except HTTPException:
url = r.get_authorize_url('uniqueKey', ['identity', 'read', 'vote', 'edit', 'history'], True)
try:
print("Opening url: %s" % url)
webbrowser.open(url, new=2)
except NameError:
warn('''Couldn't open URL: %s\n please do so manually''' % url)
server = HTTPServer(application)
server.listen(65010)
IOLoop.current().start()
if r.user == None:
print("Failed to log in. Something went wrong!")
else:
print("Logged in as %s." % r.user)
print()

@ -1,18 +0,0 @@
---
- hosts: ro
vars:
user: napalm
install_dir: "/home/{{ user }}/Shreddit"
tasks:
- name: Update the Shreddit repository
git: repo=https://github.com/x89/Shreddit.git dest={{ install_dir }}
- name: Ensure shreddit.cfg is correct
copy: src=shreddit.cfg dest={{ install_dir }}/shreddit.cfg owner={{ user }} group={{ user }} mode=0600
- name: Ensure praw.ini is correct
copy: src=praw.ini dest={{ install_dir }}/praw.ini owner={{ user }} group={{ user }} mode=0600
- name: Run get_secret.py to check that we can login with oauth
shell: bash -c "source .venv/bin/activate && python get_secret.py" chdir={{ install_dir }}
- name: Ensure Shreddit cron job
cron: user={{ user }} name="Shreddit" minute="0" job="cd {{ install_dir }} && bash -c 'source .venv/bin/activate && python shreddit.py' 2>/dev/null"
- name: Run a test Shreddit run
shell: bash -c "source .venv/bin/activate && python shreddit.py" chdir={{ install_dir }}

@ -1,17 +0,0 @@
#!/usr/bin/env python
'''
Simple script to check if your oauth is working.
'''
import praw
import sys
r = praw.Reddit('Shreddit oauth test')
try:
r.refresh_access_information()
if r.is_oauth_session():
sys.exit(0)
else:
sys.exit(2)
except:
sys.exit(1)

@ -1,12 +0,0 @@
"""This module contains the handler function called by AWS.
"""
from shreddit.shredder import shred
import yaml
def lambda_handler(event, context):
with open("shreddit.yml") as fh:
config = yaml.safe_load(fh)
if not config:
raise Exception("No config options passed!")
shred(config)

@ -1,13 +1,7 @@
[DEFAULT]
# Will be deprecated sometime in 2015 (probably)
user =
pswd =
## OAuth2 settings:
# Client / Secret from your own app
oauth_client_id =
oauth_client_secret =
# Corresponds with the callback URL in the Reddit app
oauth_redirect_uri = http://127.0.0.1:65010
log_requests = 0
# After running get_secret.py you should find your auth ID below
# Credentials go here. Fill out default, or provide one or more names and call shreddit with the -u option to specify
# which set to use.
[default]
client_id=
client_secret=
username=
password=

@ -1,6 +1,6 @@
arrow==0.9.0
decorator==4.0.10
praw<4
praw==4.0.0
PyYAML==3.12
requests==2.12.1
six==1.10.0

@ -4,7 +4,7 @@ from setuptools import setup
from codecs import open
from os import path
VERSION = "5.0.2"
VERSION = "6.0.0"
DESCRIPTION = " Remove your comment history on Reddit as deleting an account does not do so."
here = path.abspath(path.dirname(__file__))
@ -22,10 +22,12 @@ setup(
author_email="david@vaunt.eu",
classifiers=["Development Status :: 4 - Beta",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: BSD License",
"Programming Language :: Python"],
license="FreeBSD License",
packages=["shreddit"],
install_requires=["arrow", "backports-abc", "praw<4", "PyYAML", "requests", "six", "tornado"],
package_data={"shreddit": ["shreddit.yml.example"]},
install_requires=["arrow", "backports-abc", "praw>=4", "PyYAML", "requests", "six", "tornado"],
package_data={"shreddit": ["*.example"]},
entry_points={
"console_scripts": [
"shreddit=shreddit.app:main"

@ -1,11 +1,4 @@
---
# Login details for Reddit. Fill out if you don't wish
# to be prompted for a login every time you run Shreddit.
# If your username or password has characters that could cause problems, surround them in quotes.
### NOTE: This may be deprecated as you can specify in praw.ini instead
username:
password:
# How many hours of comments you want to keep
# 24 hours in a day,
# 168 hours in a week,
@ -52,10 +45,6 @@ multi_whitelist: []
# but the output from the program will be shown as an example
trial_run: False
# Don't delete but *do* edit, could prove... interesting to see a comment
# with 5000 upvotes and it's just a lorem ipsum!
edit_only: False
# Ignore distinguished comments.
whitelist_distinguished: True

@ -6,26 +6,31 @@ import logging
import os
import pkg_resources
from shreddit import default_config
from shreddit.oauth import oauth_test
from shreddit.shredder import Shredder
def main():
parser = argparse.ArgumentParser(description="Command-line frontend to the shreddit library.")
parser.add_argument("-c", "--config", help="Config file to use instead of the default shreddit.yml")
parser.add_argument("-p", "--praw", help="PRAW config (if not ./praw.ini)")
parser.add_argument("-t", "--test-oauth", help="Perform OAuth test and exit", action="store_true")
parser.add_argument("-g", "--generate-configs", help="Write shreddit and praw config files to current directory.",
action="store_true")
parser.add_argument("-u", "--user", help="User section from praw.ini if not default", default="default")
args = parser.parse_args()
if args.test_oauth:
oauth_test(args.praw)
if args.generate_configs:
if not os.path.isfile("shreddit.yml"):
print("Writing shreddit.yml file...")
with open("shreddit.yml", "wb") as fout:
fout.write(pkg_resources.resource_string("shreddit", "shreddit.yml.example"))
if not os.path.isfile("praw.ini"):
print("Writing praw.ini file...")
with open("praw.ini", "wb") as fout:
fout.write(pkg_resources.resource_string("shreddit", "praw.ini.example"))
return
config_filename = args.config or "shreddit.yml"
if not os.path.isfile(config_filename):
print("No configuration file could be found. Paste the following into a file called \"shreddit.yml\" and " \
"try running shreddit again:\n\n")
print(pkg_resources.resource_string("shreddit", "shreddit.yml.example"))
print("No shreddit configuration file was found or provided. Run this script with -g to generate one.")
return
with open(config_filename) as fh:
@ -36,8 +41,7 @@ def main():
if option in user_config:
default_config[option] = user_config[option]
# TODO: Validate config options
shredder = Shredder(default_config, args.praw)
shredder = Shredder(default_config, args.user)
shredder.shred()

@ -1,22 +0,0 @@
"""This module contains a function that tests OAuth session validity.
"""
import os
import praw
def oauth_test(praw_ini):
if praw_ini:
# PRAW won't panic if the file is invalid, so check first
if not os.path.exists(praw_ini):
print("PRAW configuration file \"{}\" not found.".format(praw_ini))
return
praw.settings.CONFIG.read(praw_ini)
r = praw.Reddit("Shreddit oauth test")
try:
r.refresh_access_information()
if r.is_oauth_session():
print("Session is valid.")
else:
print("Session is not a valid OAuth session.")
except Exception as e:
print("Error encountered while checking credentials:\n{}".format(e))

@ -0,0 +1,7 @@
# Credentials go here. Fill out default, or provide one or more names and call shreddit with the -u option to specify
# which set to use.
[default]
client_id=
client_secret=
username=
password=

@ -1,32 +1,30 @@
import os
import sys
import logging
import arrow
import argparse
import json
import arrow
import yaml
import logging
import os
import praw
import sys
import time
from re import sub
import yaml
from datetime import datetime, timedelta
from praw.errors import (InvalidUser, InvalidUserPass, RateLimitExceeded,
HTTPException, OAuthAppRequired)
from praw.objects import Comment, Submission
from shreddit.util import get_sentence
from praw.models import Comment, Submission
from prawcore.exceptions import ResponseException, OAuthException
from re import sub
from shreddit.util import get_sentence, ShredditError
class Shredder(object):
"""This class stores state for configuration, API objects, logging, etc. It exposes a shred() method that
application code can call to start it.
"""
def __init__(self, config, praw_ini=None):
def __init__(self, config, user):
logging.basicConfig()
self._logger = logging.getLogger("shreddit")
self._logger.setLevel(level=logging.DEBUG if config.get("verbose", True) else logging.INFO)
self.__dict__.update({"_{}".format(k): config[k] for k in config})
self._praw_ini = praw_ini
self._connect(praw_ini, self._username, self._password)
self._connect(user)
if self._save_directory:
self._r.config.store_json_result = True
@ -36,8 +34,6 @@ class Shredder(object):
if self._save_directory:
if not os.path.exists(self._save_directory):
os.makedirs(self._save_directory)
self._limit = None
self._api_calls = []
# Add any multireddit subreddits to the whitelist
self._whitelist = set([s.lower() for s in self._whitelist])
@ -68,7 +64,7 @@ class Shredder(object):
self._logger.info("Trial run - no deletion will be performed")
def shred(self):
deleted = self._remove_things(self._get_things())
deleted = self._remove_things(self._build_iterator())
self._logger.info("Finished deleting {} items. ".format(deleted))
if deleted >= 1000:
# This user has more than 1000 items to handle, which angers the gods of the Reddit API. So chill for a
@ -78,33 +74,14 @@ class Shredder(object):
self._connect(None, self._username, self._password)
self.shred()
def _connect(self, praw_ini, username, password):
self._r = praw.Reddit(user_agent="shreddit/5.0")
if praw_ini:
# PRAW won't panic if the file is invalid, so check first
if not os.path.exists(praw_ini):
print("PRAW configuration file \"{}\" not found.".format(praw_ini))
return
praw.settings.CONFIG.read(praw_ini)
def _connect(self, user):
try:
# Try to login with OAuth2
self._r.refresh_access_information()
self._logger.debug("Logged in with OAuth.")
except (HTTPException, OAuthAppRequired) as e:
self._logger.warning("You should migrate to OAuth2 using get_secret.py before Reddit disables this login "
"method.")
try:
try:
self._r.login(username, password)
except InvalidUserPass:
self._r.login() # Supply details on the command line
except InvalidUser as e:
raise InvalidUser("User does not exist.", e)
except InvalidUserPass as e:
raise InvalidUserPass("Specified an incorrect password.", e)
except RateLimitExceeded as e:
raise RateLimitExceeded("You're doing that too much.", e)
self._logger.info("Logged in as {user}.".format(user=self._r.user))
self._r = praw.Reddit(user, user_agent="python:shreddit:v6.0.0")
self._logger.info("Logged in as {user}.".format(user=self._r.user.me()))
except ResponseException:
raise ShredditError("Bad OAuth credentials")
except OAuthException:
raise ShredditError("Bad username or password")
def _check_whitelist(self, item):
"""Returns True if the item is whitelisted, False otherwise.
@ -137,13 +114,9 @@ class Shredder(object):
short_text = sub(b"\n\r\t", " ", comment.body[:35].encode("utf-8"))
msg = "/r/{}/ #{} ({}) with: {}".format(comment.subreddit, comment.id, short_text, replacement_text)
if self._edit_only:
self._logger.debug("Editing (not removing) {msg}".format(msg=msg))
else:
self._logger.debug("Editing and deleting {msg}".format(msg=msg))
self._logger.debug("Editing and deleting {msg}".format(msg=msg))
if not self._trial_run:
comment.edit(replacement_text)
self._api_calls.append(int(time.time()))
def _remove(self, item):
if self._keep_a_copy and self._save_directory:
@ -151,28 +124,23 @@ class Shredder(object):
if self._clear_vote:
try:
item.clear_vote()
self._api_calls.append(int(time.time()))
except HTTPException:
self._logger.debug("Couldn't clear vote on {item}".format(item=item))
if isinstance(item, Submission):
self._remove_submission(item)
elif isinstance(item, Comment):
self._remove_comment(item)
if not self._edit_only and not self._trial_run:
if not self._trial_run:
item.delete()
self._api_calls.append(int(time.time()))
def _remove_things(self, items):
self._logger.info("Loading items to delete...")
to_delete = [item for item in items]
self._logger.info("Done. Starting on batch of {} items...".format(len(to_delete)))
for idx, item in enumerate(to_delete):
minute_ago = arrow.now().replace(minutes=-1).timestamp
self._api_calls = [api_call for api_call in self._api_calls if api_call >= minute_ago]
if len(self._api_calls) >= 60:
self._logger.info("Sleeping 10 seconds to wait out API cooldown...")
time.sleep(10)
self._logger.debug("Examining item {}: {}".format(idx + 1, item))
count = 0
for item in to_delete:
count += 1
self._logger.debug("Examining item {}: {}".format(count, item))
created = arrow.get(item.created_utc)
if str(item.subreddit).lower() in self._blacklist:
self._logger.debug("Deleting due to blacklist")
@ -188,14 +156,22 @@ class Shredder(object):
continue
else:
self._remove(item)
return idx + 1
return count
def _get_things(self):
def _build_iterator(self):
item = self._r.user.me()
if self._item == "comments":
return self._r.user.get_comments(limit=self._limit, sort=self._sort)
item = item.comments
elif self._item == "submitted":
return self._r.user.get_submitted(limit=self._limit, sort=self._sort)
elif self._item == "overview":
return self._r.user.get_overview(limit=self._limit, sort=self._sort)
item = item.submissions
if self._sort == "new":
return item.new(limit=None)
elif self._sort == "top":
return item.top(limit=None)
elif self._sort == "hot":
return item.hot(limit=None)
elif self._sort == "controversial":
return item.controversial(limit=None)
else:
raise Exception("Your deletion section is wrong")
raise ShredditError("Sorting \"{}\" not recognized.".format(self._sort))

@ -1,11 +1,4 @@
---
# Login details for Reddit. Fill out if you don't wish
# to be prompted for a login every time you run Shreddit.
# If your username or password has characters that could cause problems, surround them in quotes.
### NOTE: This may be deprecated as you can specify in praw.ini instead
username:
password:
# How many hours of comments you want to keep
# 24 hours in a day,
# 168 hours in a week,
@ -52,10 +45,6 @@ multi_whitelist: []
# but the output from the program will be shown as an example
trial_run: False
# Don't delete but *do* edit, could prove... interesting to see a comment
# with 5000 upvotes and it's just a lorem ipsum!
edit_only: False
# Ignore distinguished comments.
whitelist_distinguished: True

@ -21,3 +21,11 @@ except ImportError:
words = LOREM.split()
random.shuffle(words)
return " ".join(words)
class ShredditError(Exception):
def __init__(self, value=None):
self.value = value if value else "No information provided"
def __str__(self):
return repr(self.value)

Loading…
Cancel
Save