Merge pull request #20 from mitchellklijs/develop

Add ability to fetch from watchlists and other private lists

README.md

@@ -1,12 +1,27 @@
 # Traktarr
 Script to add new shows & movies to Sonarr/Radarr based on Trakt lists.
 
-Trakt lists currently supported:
+Types of Trakt lists supported:
-- anticipated
+
-- boxoffice
+- Official Trakt lists
-- interval
+
-- popular
+  - Trending
-- trending
+  
+  - Popular
+  
+  - Anticipated
+  
+  - boxoffice
+  
+- Public lists
+
+- Private lists*
+
+  - Watchlist
+  
+  - Custom list(s)
+
+\* Support for multiple (authenticated) users.
 
 # Requirements
 
@@ -28,7 +43,47 @@ Install Traktarr to be run with `traktarr` command.
 7. `traktarr` - run once to generate a default a config.json file.
 8. `nano config.json` - edit preferences.
 
-## 2. Setup Schedule
+## 2. Create app authentication
+
+1. Create a Trakt application by going [here](https://trakt.tv/oauth/applications/new)
+2. Enter a name for your application; for example `Traktarr`
+3. Enter `urn:ietf:wg:oauth:2.0:oob` in the `Redirect uri` field.
+4. Click "SAVE APP".
+5. Open the Traktarr configuration file `config.json` and insert the Client ID in the `api_key` and the Client Secret in the `api_secret`, like this:
+
+   ```
+       {
+           "trakt": {
+               "api_key": "my_client_id",
+               "api_secret": "my_client_secret_key"
+           }
+       }
+   ```
+
+## 3. Authenticate User(s) (optional)
+
+For each user you want to access the private lists for (i.e. watchlist and/or custom lists), you will need to to authenticate that user.
+
+
+Repeat the following steps for every user you want to authenticate:
+1. Run `traktarr trakt_authentication`
+2. You wil get the following prompt:
+
+   ```
+   INFO       - media.trakt                         - __oauth_request_device_code         - We're talking to Trakt to get your verification code. Please wait a moment...
+   INFO       - media.trakt                         - __oauth_request_device_code         - Go to: https://trakt.tv/activate on any device and enter A0XXXXXX. We'll be polling Trakt every 5 seconds for a reply
+   ```
+3. Go to https://trakt.tv/activate.
+4. Enter the code you see in your terminal.
+5. Click continue.
+6. If you are not logged in to Trakt, login now.
+7. Click "Accept".
+8. You will get the message: "Woohoo! Your device is now connected and will automatically refresh in a few seconds.".
+
+You've now authenticated the user.
+You can repeat this process for as many users as you like.
+
+## 4. Setup Schedule
 
 To have Traktarr get Movies and Shows for you automatically, on set interval.
 
@@ -41,6 +96,8 @@ To have Traktarr get Movies and Shows for you automatically, on set interval.
 
 # Configuration
 
+## Sample Configuration
+
 ```json
 {
   "core": {
@@ -149,7 +206,8 @@ To have Traktarr get Movies and Shows for you automatically, on set interval.
     "url": "http://localhost:8989/"
   },
   "trakt": {
-    "api_key": ""
+    "api_key": "",
+    "api_secret": ""
   }
 }
 ```
@@ -179,13 +237,17 @@ Movies can be run on a separate schedule from Shows.
     "boxoffice": 10,
     "interval": 24,
     "popular": 3,
-    "trending": 2
+    "trending": 2,
+    "watchlist": {},
+    "lists": {}
   },
   "shows": {
     "anticipated": 10,
     "interval": 48,
     "popular": 1,
-    "trending": 2
+    "trending": 2,
+    "watchlist": {},
+    "lists": {}
   }
 },
 ```
@@ -194,6 +256,108 @@ Movies can be run on a separate schedule from Shows.
 
 `anticipated`, `popular`, `trending`, `boxoffice` (movies only) - specify how many items from each Trakt list to find.
 
+`watchlist` - specify which watchlists to fetch (see explanation below)
+
+`lists` - specify which custom lists to fetch (see explanation below)
+
+### Personal Watchlists
+
+The watchlist task can be scheduled with a differtent item limit for every (authenticated) user.*
+
+So for every user, you will add: `"username": limit` to the watchlist key. For example:
+
+```json
+"automatic": {
+  "movies": {
+    "watchlist": {
+        "user1": 10,
+        "user2": 5
+    }
+  },
+  "shows": {
+    "watchlist": {
+        "user1": 2,
+        "user3": 1
+    }
+  }
+},
+```
+
+Of course you can combine this with running the other list types as well.
+
+### Custom Lists
+
+You can also schedule any number of public or private custom lists.
+
+For both public and private lists you'll need the url to that list. When viewing the list on Trakt, simply copy the url from the address bar of the your browser. 
+
+#### Public Lists
+
+Public lists can be added by specifying the url and the item limit like this:
+
+```json
+"automatic": {
+  "movies": {
+    "lists": {
+        "https://trakt.tv/users/rkerwin/lists/top-100-movies": 10
+    }
+  },
+  "shows": {
+    "lists": {
+        "https://trakt.tv/users/claireaa/lists/top-100-tv-shows-of-all-time-ign": 10
+    }
+  }
+},
+```
+
+
+#### Private Lists
+
+Private lists can be added in two ways:
+
+1. If there is only one authenticated user, you can add the private list just like any other public list:
+
+```json
+"automatic": {
+  "movies": {
+    "lists": {
+        "https://trakt.tv/users/user/lists/my-private-movies-list": 10
+    }
+  },
+  "shows": {
+    "lists": {
+        "https://trakt.tv/users/user/lists/my-private-shows-list": 10
+    }
+  }
+},
+```
+
+2. If there are multiple authenticated users you want to fetch the lists from, you'll need to specify the username under `authenticate_as`. 
+
+_Note: The user should have access to the list (either own the list or a list that was shared to them by a friend)._
+
+```json
+"automatic": {
+  "movies": {
+    "lists": {
+        "https://trakt.tv/users/user/lists/my-private-movies-list": {
+            "authenticate_as": "user2",
+            "limit": 10
+        }
+    }
+  },
+  "shows": {
+    "lists": {
+        "https://trakt.tv/users/user/lists/my-private-shows-list": {
+            "authenticate_as": "user2",
+            "limit": 10
+        }
+    }
+  }
+},
+```
+
+
 ## Filters
 
 Use filters to specify the movie/shows's country of origin or blacklist (i.e. filter-out) certain keywords, genres, years, runtime, or specific movies/shows.
@@ -387,6 +551,8 @@ Sonarr configuration.
 
 To show how tags work, we will create a sample tag `AMZN` and assign it to certain networks.
 
+_Note: These are optional._
+
 ### Sonarr
 
 First, we will create a tag in Sonarr (Settings > Indexers > Restrictions).
@@ -399,7 +565,7 @@ Tags: AMZN
 
 ### Traktarr
 
-Finally, we will edit the Traktarr config and assign the `AMZN` tag to certain networks.
+Finally, we will edit the Traktarr config and assign the `AMZN` tag to certain networks. 
 
 ```json
 "tags": {
@@ -429,22 +595,19 @@ Finally, we will edit the Traktarr config and assign the `AMZN` tag to certain n
 
 ## Trakt
 
+Trakt Authentication info: 
+
 ```json
 "trakt": {
-  "api_key": ""
+  "api_key": "",
+  "api_scret": ""
 }
 ```
 
 `api_key` - Fill in your Trakt API key (_Client ID_).
 
+`api_secret` - Fill in your Trakt Secret key (_Client Scret_)
 
-How to get a Trakt API Key:
-  - Go to https://trakt.tv/oauth/applications/new
-  - Fill in:
-    - Name: `Traktarr`
-    - Redirect uri: `https://google.com`
-  - Click `Save App`
-  - Retrieve the _Client ID_.
 
 # Usage
 
@@ -467,9 +630,10 @@ Options:
   --help          Show this message and exit.
 
 Commands:
-  movies  Add new movies to Radarr.
+  movies                Add new movies to Radarr.
-  run     Run in automatic mode.
+  run                   Run in automatic mode.
-  shows   Add new shows to Sonarr.
+  shows                 Add new shows to Sonarr.
+  trakt_authentication  Authenticate Traktrarr to index your personal...
   ```
 
 
@@ -486,8 +650,9 @@ Usage: traktarr movies [OPTIONS]
   Add new movies to Radarr.
 
 Options:
-  -t, --list-type [anticipated|trending|popular|boxoffice]
+  -t, --list-type TEXT            Trakt list to process. For example, anticipated,
-                                  Trakt list to process.  [required]
+                                  trending, popular, boxoffice, watchlist or any
+                                  URL to a list  [required]
   -l, --add-limit INTEGER         Limit number of movies added to Radarr.
                                   [default: 0]
   -d, --add-delay FLOAT           Seconds between each add request to Radarr.
@@ -496,7 +661,9 @@ Options:
   -f, --folder TEXT               Add movies with this root folder to Radarr.
   --no-search                     Disable search when adding movies to Radarr.
   --notifications                 Send notifications.
-  --help                          Show this message and exit.
+  --authencate-user TEXT          Specify which user to authenticate with to
+                                  retrieve Trakt lists. Default: first user in the
+                                  config
 ```
 
 
@@ -508,8 +675,9 @@ Usage: traktarr shows [OPTIONS]
   Add new shows to Sonarr.
 
 Options:
-  -t, --list-type [anticipated|trending|popular]
+  -t, --list-type TEXT            Trakt list to process. For example, anticipated,
-                                  Trakt list to process.  [required]
+                                  trending, popular, watchlist or any URL to a
+                                  list  [required]
   -l, --add-limit INTEGER         Limit number of shows added to Sonarr.
                                   [default: 0]
   -d, --add-delay FLOAT           Seconds between each add request to Sonarr.
@@ -518,17 +686,40 @@ Options:
   -f, --folder TEXT               Add shows with this root folder to Sonarr.
   --no-search                     Disable search when adding shows to Sonarr.
   --notifications                 Send notifications.
+  --authencate-user TEXT          Specify which user to authenticate with to
+                                  retrieve Trakt lists. Default: first user in the
+                                  config
   --help                          Show this message and exit.
 ```
 
 ## Examples
 
+- Fetch boxoffice movies, labeled with the comedy genre, limited to 10 items, and send notifications:
 
-```
+  ```
-traktarr movies -t boxoffice -g comedy -l 10 --notifications
+  traktarr movies -t boxoffice -g comedy -l 10 --notifications
+  ```
 
-```
+- Fetch popular shows, limited to 2 items, and don't start the search in Sonarr:
 
-```
+  ```
-traktarr shows -t popular -l 2 --no-search
+  traktarr shows -t popular -l 2 --no-search
-```
+  ```
+
+- Fetch all shows from the watchlist of `user1`:
+
+  ```
+  traktarr shows -t watchlist --authenticate-user user1
+  ```
+  
+- Fetch all movies from the public list `https://trakt.tv/users/rkerwin/lists/top-100-movies`:
+
+  ```
+  traktarr movies -t https://trakt.tv/users/rkerwin/lists/top-100-movies
+  ```
+
+- Fetch all movies from the private list `https://trakt.tv/users/user1/lists/private-movies-list` of `user1`:
+
+  ```
+  traktarr movies -t https://trakt.tv/users/user1/lists/private-movies-list --authenticate-user=user1
+  ```

media/trakt.py

@@ -15,8 +15,12 @@ def backoff_handler(details):
 
 
 class Trakt:
-    def __init__(self, api_key):
+    non_user_lists = ['anticipated', 'trending', 'popular', 'boxoffice']
-        self.api_key = api_key
+
+    def __init__(self, cfg):
+        self.cfg = cfg
+        self.api_key = cfg.trakt.api_key
+        self.api_secret = cfg.trakt.api_secret
         self.headers = {
             'Content-Type': 'application/json',
             'trakt-api-version': '2',
@@ -44,6 +48,161 @@ class Trakt:
             log.exception("Exception validating api_key: ")
         return False
 
+    ############################################################
+    # OAuth Authentication Initialisation
+    ############################################################
+
+    def __oauth_request_device_code(self):
+        log.info("We're talking to Trakt to get your verification code. Please wait a moment...")
+
+        payload = {'client_id': self.api_key}
+
+        # Request device code
+        req = requests.post('https://api.trakt.tv/oauth/device/code', params=payload, headers=self.headers)
+        device_code_response = req.json()
+
+        # Display needed information to the user
+        log.info('Go to: %s on any device and enter %s. We\'ll be polling Trakt every %s seconds for a reply',
+                 device_code_response['verification_url'], device_code_response['user_code'],
+                 device_code_response['interval'])
+
+        return device_code_response
+
+    def __oauth_process_token_request(self, req):
+        success = False
+
+        if req.status_code == 200:
+            # Success; saving the access token
+            access_token_response = req.json()
+            access_token = access_token_response['access_token']
+
+            # But first we need to find out what user this token belongs to
+            temp_headers = self.headers
+            temp_headers['Authorization'] = 'Bearer ' + access_token
+
+            req = requests.get('https://api.trakt.tv/users/me', headers=temp_headers)
+
+            from misc.config import Config
+            new_config = Config()
+
+            new_config.merge_settings({
+                "trakt": {
+                    req.json()['username']: access_token_response
+                }
+            })
+
+            success = True
+        elif req.status_code == 404:
+            log.debug('The device code was wrong')
+            log.error('Whoops, something went wrong; aborting the authentication process')
+        elif req.status_code == 409:
+            log.error('You\'ve already authenticated this application; aborting the authentication process')
+        elif req.status_code == 410:
+            log.error('The authentication process has expired; please start again')
+        elif req.status_code == 418:
+            log.error('You\'ve denied the authentication; are you sure? Please try again')
+        elif req.status_code == 429:
+            log.debug('We\'re polling too quickly.')
+
+        return success, req.status_code
+
+    def __oauth_poll_for_access_token(self, device_code, polling_interval=5, polling_expire=600):
+        polling_start = time.time()
+        time.sleep(polling_interval)
+        tries = 0
+
+        while time.time() - polling_start < polling_expire:
+            tries += 1
+
+            log.debug('Polling Trakt for the %sth time; %s seconds left', tries,
+                      polling_expire - round(time.time() - polling_start))
+
+            payload = {'code': device_code, 'client_id': self.api_key, 'client_secret': self.api_secret,
+                       'grant_type': 'authorization_code'}
+
+            # Poll Trakt for access token
+            req = requests.post('https://api.trakt.tv/oauth/device/token', params=payload, headers=self.headers)
+
+            success, status_code = self.__oauth_process_token_request(req)
+
+            if success:
+                break
+            elif status_code == 426:
+                log.debug('Increasing the interval by one second')
+                polling_interval += 1
+
+            time.sleep(polling_interval)
+        return False
+
+    def __oauth_refresh_access_token(self, refresh_token):
+        # TODO Doesn't work
+
+        payload = {'refresh_token': refresh_token, 'client_id': self.api_key, 'client_secret': self.api_secret,
+                   'grant_type': 'refresh_token'}
+
+        req = requests.post('https://api.trakt.tv/oauth/token', params=payload, headers=self.headers)
+
+        success, status_code = self.__oauth_process_token_request(req)
+
+        return success
+
+    def oauth_authentication(self):
+        try:
+            device_code_response = self.__oauth_request_device_code()
+
+            if self.__oauth_poll_for_access_token(device_code_response['device_code'],
+                                                  device_code_response['interval'],
+                                                  device_code_response['expires_in']):
+                return True
+        except Exception:
+            log.exception("Exception occurred when authenticating user")
+        return False
+
+    def oauth_headers(self, user):
+        headers = self.headers
+
+        if user is None:
+            users = self.cfg['trakt']
+            
+            if 'api_key' in users.keys():
+                users.pop('api_key')
+
+            if 'api_secret' in users.keys():
+                users.pop('api_secret')
+
+            if len(users) > 0:
+                user = list(users.keys())[0]
+
+                log.debug('No user provided, so default to the first user in the config (%s)', user)
+        elif user not in self.cfg['trakt'].keys():
+            log.error(
+                'The user %s you specified to use for authentication is not authenticated yet. Authenticate the user first, before you use it to retrieve lists.',
+                user)
+
+            exit()
+
+        # If there is no default user, try without authentication
+        if user is None:
+            log.info('Using no authentication')
+
+            return headers, user
+
+        token_information = self.cfg['trakt'][user]
+        # Check if the acces_token for the user is expired
+        expires_at = token_information['created_at'] + token_information['expires_in']
+
+        if expires_at < round(time.time()):
+            log.info("The access token for the user %s has expired. We're requesting a new one; please wait a moment.",
+                     user)
+
+            if self.__oauth_refresh_access_token(token_information["refresh_token"]):
+                log.info("The access token for the user %s has been refreshed. Please restart the application.",
+                         user)
+
+        headers['Authorization'] = 'Bearer ' + token_information['access_token']
+
+        return headers, user
+
     ############################################################
     # Shows
     ############################################################
@@ -108,6 +267,149 @@ class Trakt:
             log.exception("Exception retrieving anticipated shows: ")
         return None
 
+    @backoff.on_predicate(backoff.expo, lambda x: x is None, max_tries=4, on_backoff=backoff_handler)
+    def get_watchlist_shows(self, authenticate_user=None, limit=1000, languages=None):
+        try:
+            processed_shows = []
+
+            if languages is None:
+                languages = ['en']
+
+            # generate payload
+            payload = {'extended': 'full', 'limit': limit, 'page': 1}
+            if languages:
+                payload['languages'] = ','.join(languages)
+
+            # make request
+            while True:
+                headers, authenticate_user = self.oauth_headers(authenticate_user)
+
+                req = requests.get('https://api.trakt.tv/users/' + authenticate_user + '/watchlist/movies',
+                                   params=payload,
+                                   headers=headers,
+                                   timeout=30)
+                log.debug("Request User: %s", authenticate_user)
+                log.debug("Request URL: %s", req.url)
+                log.debug("Request Payload: %s", payload)
+                log.debug("Response Code: %d", req.status_code)
+                log.debug("Response Page: %d of %d", payload['page'],
+                          0 if 'X-Pagination-Page-Count' not in req.headers else int(
+                              req.headers['X-Pagination-Page-Count']))
+
+                if req.status_code == 200:
+                    resp_json = req.json()
+
+                    for show in resp_json:
+                        if show not in processed_shows:
+                            processed_shows.append(show)
+
+                    # check if we have fetched the last page, break if so
+                    if 'X-Pagination-Page-Count' not in req.headers or not int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There was no more pages to retrieve")
+                        break
+                    elif payload['page'] >= int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There are no more pages to retrieve results from")
+                        break
+                    else:
+                        log.info("There are %d pages left to retrieve results from",
+                                 int(req.headers['X-Pagination-Page-Count']) - payload['page'])
+                        payload['page'] += 1
+                        time.sleep(5)
+                elif req.status_code == 401:
+                    log.error("The authentication to Trakt is revoked. Please re-authenticate.")
+
+                    exit()
+                else:
+                    log.error("Failed to retrieve shows on watchlist from %s, request response: %d", authenticate_user,
+                              req.status_code)
+                    break
+
+            if len(processed_shows):
+                log.debug("Found %d shows on watchlist from %s", len(processed_shows), authenticate_user)
+
+                return processed_shows
+            return None
+        except Exception:
+            log.exception("Exception retrieving shows on watchlist")
+        return None
+
+    @backoff.on_predicate(backoff.expo, lambda x: x is None, max_tries=4, on_backoff=backoff_handler)
+    def get_user_list_shows(self, list_url, authenticate_user=None, limit=1000, languages=None):
+        try:
+            processed_shows = []
+
+            if languages is None:
+                languages = ['en']
+
+            # generate payload
+            payload = {'extended': 'full', 'limit': limit, 'page': 1}
+            if languages:
+                payload['languages'] = ','.join(languages)
+
+            try:
+                import re
+                list_user = re.search('\/users\/([^/]*)', list_url).group(1)
+                list_key = re.search('\/lists\/([^/]*)', list_url).group(1)
+            except:
+                log.error('The URL "%s" is not in the correct format', list_url)
+
+                exit()
+
+            log.debug('Fetching %s from %s', list_key, list_user)
+
+            # make request
+            while True:
+                headers, authenticate_user = self.oauth_headers(authenticate_user)
+
+                req = requests.get('https://api.trakt.tv/users/' + list_user + '/lists/' + list_key + '/items/shows',
+                                   params=payload,
+                                   headers=headers,
+                                   timeout=30)
+                log.debug("Request User: %s", authenticate_user)
+                log.debug("Request URL: %s", req.url)
+                log.debug("Request Payload: %s", payload)
+                log.debug("Response Code: %d", req.status_code)
+                log.debug("Response Page: %d of %d", payload['page'],
+                          0 if 'X-Pagination-Page-Count' not in req.headers else int(
+                              req.headers['X-Pagination-Page-Count']))
+
+                if req.status_code == 200:
+                    resp_json = req.json()
+
+                    for show in resp_json:
+                        if show not in processed_shows:
+                            processed_shows.append(show)
+
+                    # check if we have fetched the last page, break if so
+                    if 'X-Pagination-Page-Count' not in req.headers or not int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There was no more pages to retrieve")
+                        break
+                    elif payload['page'] >= int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There are no more pages to retrieve results from")
+                        break
+                    else:
+                        log.info("There are %d pages left to retrieve results from",
+                                 int(req.headers['X-Pagination-Page-Count']) - payload['page'])
+                        payload['page'] += 1
+                        time.sleep(5)
+                elif req.status_code == 401:
+                    log.error("The authentication to Trakt is revoked. Please re-authenticate.")
+
+                    exit()
+                else:
+                    log.error("Failed to retrieve shows on %s from %s, request response: %d", list_key, list_user,
+                              req.status_code)
+                    break
+
+            if len(processed_shows):
+                log.debug("Found %d shows on %s from %s", len(processed_shows), list_key, list_user)
+
+                return processed_shows
+            return None
+        except Exception:
+            log.exception("Exception retrieving shows on user list")
+        return None
+
     @backoff.on_predicate(backoff.expo, lambda x: x is None, max_tries=4, on_backoff=backoff_handler)
     def get_trending_shows(self, limit=1000, languages=None):
         try:
@@ -473,3 +775,145 @@ class Trakt:
         except Exception:
             log.exception("Exception retrieving boxoffice movies: ")
         return None
+
+    @backoff.on_predicate(backoff.expo, lambda x: x is None, max_tries=4, on_backoff=backoff_handler)
+    def get_watchlist_movies(self, authenticate_user=None, limit=1000, languages=None):
+        try:
+            processed_movies = []
+
+            if languages is None:
+                languages = ['en']
+
+            # generate payload
+            payload = {'extended': 'full', 'limit': limit, 'page': 1}
+            if languages:
+                payload['languages'] = ','.join(languages)
+
+            # make request
+            while True:
+                headers, authenticate_user = self.oauth_headers(authenticate_user)
+
+                req = requests.get('https://api.trakt.tv/users/' + authenticate_user + '/watchlist/movies',
+                                   params=payload,
+                                   headers=headers,
+                                   timeout=30)
+                log.debug("Request User: %s", authenticate_user)
+                log.debug("Request URL: %s", req.url)
+                log.debug("Request Payload: %s", payload)
+                log.debug("Response Code: %d", req.status_code)
+                log.debug("Response Page: %d of %d", payload['page'],
+                          0 if 'X-Pagination-Page-Count' not in req.headers else int(
+                              req.headers['X-Pagination-Page-Count']))
+
+                if req.status_code == 200:
+                    resp_json = req.json()
+
+                    for show in resp_json:
+                        if show not in processed_movies:
+                            processed_movies.append(show)
+
+                    # check if we have fetched the last page, break if so
+                    if 'X-Pagination-Page-Count' not in req.headers or not int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There was no more pages to retrieve")
+                        break
+                    elif payload['page'] >= int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There are no more pages to retrieve results from")
+                        break
+                    else:
+                        log.info("There are %d pages left to retrieve results from",
+                                 int(req.headers['X-Pagination-Page-Count']) - payload['page'])
+                        payload['page'] += 1
+                        time.sleep(5)
+                elif req.status_code == 401:
+                    log.error("The authentication to Trakt is revoked. Please re-authenticate.")
+
+                    exit()
+                else:
+                    log.error("Failed to retrieve movies on watchlist from %s, request response: %d", authenticate_user,
+                              req.status_code)
+                    break
+
+            if len(processed_movies):
+                log.debug("Found %d movies on watchlist from %s", len(processed_movies), authenticate_user)
+
+                return processed_movies
+            return None
+        except Exception:
+            log.exception("Exception retrieving movies on watchlist")
+        return None
+
+    @backoff.on_predicate(backoff.expo, lambda x: x is None, max_tries=4, on_backoff=backoff_handler)
+    def get_user_list_movies(self, list_url, authenticate_user=None, limit=1000, languages=None):
+        try:
+            processed_movies = []
+
+            if languages is None:
+                languages = ['en']
+
+            # generate payload
+            payload = {'extended': 'full', 'limit': limit, 'page': 1}
+            if languages:
+                payload['languages'] = ','.join(languages)
+
+            try:
+                import re
+                list_user = re.search('\/users\/([^/]*)', list_url).group(1)
+                list_key = re.search('\/lists\/([^/]*)', list_url).group(1)
+            except:
+                log.error('The URL "%s" is not in the correct format', list_url)
+
+            log.debug('Fetching %s from %s', list_key, list_user)
+
+            # make request
+            while True:
+                headers, authenticate_user = self.oauth_headers(authenticate_user)
+
+                req = requests.get('https://api.trakt.tv/users/' + list_user + '/lists/' + list_key + '/items/movies',
+                                   params=payload,
+                                   headers=headers,
+                                   timeout=30)
+                log.debug("Request User: %s", authenticate_user)
+                log.debug("Request URL: %s", req.url)
+                log.debug("Request Payload: %s", payload)
+                log.debug("Response Code: %d", req.status_code)
+                log.debug("Response Page: %d of %d", payload['page'],
+                          0 if 'X-Pagination-Page-Count' not in req.headers else int(
+                              req.headers['X-Pagination-Page-Count']))
+
+                if req.status_code == 200:
+                    resp_json = req.json()
+
+                    for show in resp_json:
+                        if show not in processed_movies:
+                            processed_movies.append(show)
+
+                    # check if we have fetched the last page, break if so
+                    if 'X-Pagination-Page-Count' not in req.headers or not int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There was no more pages to retrieve")
+                        break
+                    elif payload['page'] >= int(req.headers['X-Pagination-Page-Count']):
+                        log.debug("There are no more pages to retrieve results from")
+                        break
+                    else:
+                        log.info("There are %d pages left to retrieve results from",
+                                 int(req.headers['X-Pagination-Page-Count']) - payload['page'])
+                        payload['page'] += 1
+                        time.sleep(5)
+                elif req.status_code == 401:
+                    log.error("The authentication to Trakt is revoked. Please re-authenticate.")
+
+                    exit()
+                else:
+                    log.error("Failed to retrieve movies on %s from %s, request response: %d", list_key,
+                              authenticate_user,
+                              req.status_code)
+                    break
+
+            if len(processed_movies):
+                log.debug("Found %d movies on %s from %s", len(processed_movies), list_key, authenticate_user)
+
+                return processed_movies
+            return None
+        except Exception:
+            log.exception("Exception retrieving movies on user list")
+        return None

misc/config.py

@@ -33,13 +33,13 @@ class AttrConfig(AttrDict):
 
 
 class Config(object, metaclass=Singleton):
-
     base_config = {
         'core': {
             'debug': False
         },
         'trakt': {
-            'api_key': ''
+            'api_key': '',
+            'api_secret': ''
         },
         'sonarr': {
             'url': 'http://localhost:8989/',
@@ -150,36 +150,55 @@ class Config(object, metaclass=Singleton):
         with open(self.config_path, 'r') as fp:
             return AttrConfig(json.load(fp))
 
+    def __inner_upgrade(self, settings1, settings2, key=None, overwrite=False):
+        sub_upgraded = False
+        merged = settings2.copy()
+
+        # print(settings1)
+        # print(settings2)
+        # print(overwrite)
+        # print("_______________")
+
+        if isinstance(settings1, dict):
+            for k, v in settings1.items():
+                # missing k
+                if k not in settings2:
+                    merged[k] = v
+                    sub_upgraded = True
+                    if not key:
+                        print("Added %r config option: %s" % (str(k), str(v)))
+                    else:
+                        print("Added %r to config option %r: %s" % (str(k), str(key), str(v)))
+                    continue
+
+                # iterate children
+                if isinstance(v, dict) or isinstance(v, list):
+                    merged[k], did_upgrade = self.__inner_upgrade(settings1[k], settings2[k], key=k,
+                                                                  overwrite=overwrite)
+                    sub_upgraded = did_upgrade if did_upgrade else sub_upgraded
+                elif settings1[k] != settings2[k] and overwrite:
+                    merged = settings1
+                    sub_upgraded = True
+        elif isinstance(settings1, list) and key:
+            for v in settings1:
+                if v not in settings2:
+                    merged.append(v)
+                    sub_upgraded = True
+                    print("Added to config option %r: %s" % (str(key), str(v)))
+                    continue
+
+        return merged, sub_upgraded
+
     def upgrade_settings(self, currents):
-        upgraded = False
+        upgraded_settings, upgraded = self.__inner_upgrade(self.base_config, currents)
-
+        return AttrConfig(upgraded_settings), upgraded
-        def inner_upgrade(default, current, key=None):
+
-            sub_upgraded = False
+    def merge_settings(self, settings_to_merge):
-            merged = current.copy()
+        upgraded_settings, upgraded = self.__inner_upgrade(settings_to_merge, self.conf, overwrite=True)
-            if isinstance(default, dict):
+
-                for k, v in default.items():
+        self.conf = upgraded_settings
-                    # missing k
+
-                    if k not in current:
+        if upgraded:
-                        merged[k] = v
+            self.dump_config()
-                        sub_upgraded = True
+
-                        if not key:
-                            print("Added %r config option: %s" % (str(k), str(v)))
-                        else:
-                            print("Added %r to config option %r: %s" % (str(k), str(key), str(v)))
-                        continue
-                    # iterate children
-                    if isinstance(v, dict) or isinstance(v, list):
-                        merged[k], did_upgrade = inner_upgrade(default[k], current[k], key=k)
-                        sub_upgraded = did_upgrade if did_upgrade else sub_upgraded
-
-            elif isinstance(default, list) and key:
-                for v in default:
-                    if v not in current:
-                        merged.append(v)
-                        sub_upgraded = True
-                        print("Added to config option %r: %s" % (str(key), str(v)))
-                        continue
-            return merged, sub_upgraded
-
-        upgraded_settings, upgraded = inner_upgrade(self.base_config, currents)
         return AttrConfig(upgraded_settings), upgraded

traktarr.py

@@ -53,20 +53,40 @@ def app(config, logfile):
     init_notifications()
 
 
+############################################################
+# Trakt OAuth
+############################################################
+
+@app.command(help='Authenticate Traktrarr to index your personal lists')
+def trakt_authentication():
+    from media.trakt import Trakt
+    trakt = Trakt(cfg)
+
+    response = trakt.oauth_authentication()
+
+    if response:
+        log.info("Authentication information saved; please restart the application")
+        exit()
+
+
 ############################################################
 # SHOWS
 ############################################################
 
 @app.command(help='Add new shows to Sonarr.')
-@click.option('--list-type', '-t', type=click.Choice(['anticipated', 'trending', 'popular']),
+@click.option('--list-type', '-t',
-              help='Trakt list to process.', required=True)
+              help='Trakt list to process. For example, anticipated, trending, popular, watchlist or any URL to a list',
+              required=True)
 @click.option('--add-limit', '-l', default=0, help='Limit number of shows added to Sonarr.', show_default=True)
 @click.option('--add-delay', '-d', default=2.5, help='Seconds between each add request to Sonarr.', show_default=True)
 @click.option('--genre', '-g', default=None, help='Only add shows from this genre to Sonarr.')
 @click.option('--folder', '-f', default=None, help='Add shows with this root folder to Sonarr.')
 @click.option('--no-search', is_flag=True, help='Disable search when adding shows to Sonarr.')
 @click.option('--notifications', is_flag=True, help='Send notifications.')
-def shows(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_search=False, notifications=False):
+@click.option('--authenticate-user',
+              help='Specify which user to authenticate with to retrieve Trakt lists. Default: first user in the config')
+def shows(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_search=False, notifications=False,
+          authenticate_user=None):
     from media.sonarr import Sonarr
     from media.trakt import Trakt
     from misc import helpers
@@ -82,7 +102,7 @@ def shows(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_sea
         cfg['sonarr']['root_folder'] = folder
 
     # validate trakt api_key
-    trakt = Trakt(cfg.trakt.api_key)
+    trakt = Trakt(cfg)
     if not trakt.validate_api_key():
         log.error("Aborting due to failure to validate Trakt API Key")
         if notifications:
@@ -144,12 +164,11 @@ def shows(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_sea
         trakt_series_list = trakt.get_trending_shows()
     elif list_type.lower() == 'popular':
         trakt_series_list = trakt.get_popular_shows()
+    elif list_type.lower() == 'watchlist':
+        trakt_series_list = trakt.get_watchlist_shows(authenticate_user)
     else:
-        log.error("Aborting due to unknown Trakt list type")
+        trakt_series_list = trakt.get_user_list_shows(list_type, authenticate_user)
-        if notifications:
+
-            callback_notify({'event': 'abort', 'type': 'shows', 'list_type': list_type,
-                             'reason': 'Failure to determine Trakt list type'})
-        return None
     if not trakt_series_list:
         log.error("Aborting due to failure to retrieve Trakt %s shows list", list_type)
         if notifications:
@@ -232,15 +251,19 @@ def shows(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_sea
 ############################################################
 
 @app.command(help='Add new movies to Radarr.')
-@click.option('--list-type', '-t', type=click.Choice(['anticipated', 'trending', 'popular', 'boxoffice']),
+@click.option('--list-type', '-t',
-              help='Trakt list to process.', required=True)
+              help='Trakt list to process. For example, anticipated, trending, popular, boxoffice, watchlist or any URL to a list',
+              required=True)
 @click.option('--add-limit', '-l', default=0, help='Limit number of movies added to Radarr.', show_default=True)
 @click.option('--add-delay', '-d', default=2.5, help='Seconds between each add request to Radarr.', show_default=True)
 @click.option('--genre', '-g', default=None, help='Only add movies from this genre to Radarr.')
 @click.option('--folder', '-f', default=None, help='Add movies with this root folder to Radarr.')
 @click.option('--no-search', is_flag=True, help='Disable search when adding movies to Radarr.')
 @click.option('--notifications', is_flag=True, help='Send notifications.')
-def movies(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_search=False, notifications=False):
+@click.option('--authenticate-user',
+              help='Specify which user to authenticate with to retrieve Trakt lists. Default: first user in the config')
+def movies(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_search=False, notifications=False,
+           authenticate_user=None):
     from media.radarr import Radarr
     from media.trakt import Trakt
     from misc import helpers
@@ -256,7 +279,7 @@ def movies(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_se
         cfg['radarr']['root_folder'] = folder
 
     # validate trakt api_key
-    trakt = Trakt(cfg.trakt.api_key)
+    trakt = Trakt(cfg)
     if not trakt.validate_api_key():
         log.error("Aborting due to failure to validate Trakt API Key")
         if notifications:
@@ -310,12 +333,11 @@ def movies(list_type, add_limit=0, add_delay=2.5, genre=None, folder=None, no_se
         trakt_movies_list = trakt.get_popular_movies()
     elif list_type.lower() == 'boxoffice':
         trakt_movies_list = trakt.get_boxoffice_movies()
+    elif list_type.lower() == 'watchlist':
+        trakt_movies_list = trakt.get_watchlist_movies(authenticate_user)
     else:
-        log.error("Aborting due to unknown Trakt list type")
+        trakt_movies_list = trakt.get_user_list_movies(list_type, authenticate_user)
-        if notifications:
+
-            callback_notify({'event': 'abort', 'type': 'movies', 'list_type': list_type,
-                             'reason': 'Failure to determine Trakt list type'})
-        return None
     if not trakt_movies_list:
         log.error("Aborting due to failure to retrieve Trakt %s movies list", list_type)
         if notifications:
@@ -412,23 +434,57 @@ def callback_notify(data):
 
 
 def automatic_shows(add_delay=2.5, no_search=False, notifications=False):
+    from media.trakt import Trakt
+
     total_shows_added = 0
     try:
         log.info("Started")
 
-        for list_type, type_amount in cfg.automatic.shows.items():
+        for list_type, value in cfg.automatic.shows.items():
+            added_shows = None
+
             if list_type.lower() == 'interval':
                 continue
-            elif type_amount <= 0:
-                log.info("Skipped Trakt's %s shows list", list_type)
-                continue
-            else:
-                log.info("Adding %d shows from Trakt's %s list", type_amount, list_type)
 
-            # run shows
+            if list_type.lower() in Trakt.non_user_lists:
-            added_shows = shows.callback(list_type=list_type, add_limit=type_amount,
+                limit = value
-                                         add_delay=add_delay, no_search=no_search,
+
-                                         notifications=notifications)
+                if limit <= 0:
+                    log.info("Skipped Trakt's %s shows list", list_type)
+                    continue
+                else:
+                    log.info("Adding %d shows from Trakt's %s list", limit, list_type)
+
+                # run shows
+                added_shows = shows.callback(list_type=list_type, add_limit=limit,
+                                             add_delay=add_delay, no_search=no_search,
+                                             notifications=notifications)
+            elif list_type.lower() == 'watchlist':
+                for authenticate_user, limit in value.items():
+                    if limit <= 0:
+                        log.info("Skipped Trakt's %s for %s", list_type, authenticate_user)
+                        continue
+                    else:
+                        log.info("Adding %d shows from the %s from %s", limit, list_type, authenticate_user)
+
+                    # run shows
+                    added_shows = shows.callback(list_type=list_type, add_limit=limit,
+                                                 add_delay=add_delay, no_search=no_search,
+                                                 notifications=notifications, authenticate_user=authenticate_user)
+            elif list_type.lower() == 'lists':
+                for list, v in value.items():
+                    if isinstance(v, dict):
+                        authenticate_user = v['authenticate_user']
+                        limit = v['limit']
+                    else:
+                        authenticate_user = None
+                        limit = v
+
+                    # run shows
+                    added_shows = shows.callback(list_type=list, add_limit=limit,
+                                                 add_delay=add_delay, no_search=no_search,
+                                                 notifications=notifications, authenticate_user=authenticate_user)
+
             if added_shows is None:
                 log.error("Failed adding shows from Trakt's %s list", list_type)
                 time.sleep(10)
@@ -449,23 +505,57 @@ def automatic_shows(add_delay=2.5, no_search=False, notifications=False):
 
 
 def automatic_movies(add_delay=2.5, no_search=False, notifications=False):
+    from media.trakt import Trakt
+
     total_movies_added = 0
     try:
         log.info("Started")
 
-        for list_type, type_amount in cfg.automatic.movies.items():
+        for list_type, value in cfg.automatic.movies.items():
+            added_movies = None
+
             if list_type.lower() == 'interval':
                 continue
-            elif type_amount <= 0:
-                log.info("Skipped Trakt's %s movies list", list_type)
-                continue
-            else:
-                log.info("Adding %d movies from Trakt's %s list", type_amount, list_type)
 
-            # run movies
+            if list_type.lower() in Trakt.non_user_lists:
-            added_movies = movies.callback(list_type=list_type, add_limit=type_amount,
+                limit = value
-                                           add_delay=add_delay, no_search=no_search,
+
-                                           notifications=notifications)
+                if limit <= 0:
+                    log.info("Skipped Trakt's %s movies list", list_type)
+                    continue
+                else:
+                    log.info("Adding %d movies from Trakt's %s list", limit, list_type)
+
+                # run movies
+                added_movies = movies.callback(list_type=list_type, add_limit=limit,
+                                               add_delay=add_delay, no_search=no_search,
+                                               notifications=notifications)
+            elif list_type.lower() == 'watchlist':
+                for authenticate_user, limit in value.items():
+                    if limit <= 0:
+                        log.info("Skipped Trakt's %s for %s", list_type, authenticate_user)
+                        continue
+                    else:
+                        log.info("Adding %d movies from the %s from %s", limit, list_type, authenticate_user)
+
+                    # run movies
+                    added_movies = movies.callback(list_type=list_type, add_limit=limit,
+                                                   add_delay=add_delay, no_search=no_search,
+                                                   notifications=notifications, authenticate_user=authenticate_user)
+            elif list_type.lower() == 'lists':
+                for list, v in value.items():
+                    if isinstance(v, dict):
+                        authenticate_user = v['authenticate_user']
+                        limit = v['limit']
+                    else:
+                        authenticate_user = None
+                        limit = v
+
+                    # run shows
+                    added_movies = movies.callback(list_type=list, add_limit=limit,
+                                                   add_delay=add_delay, no_search=no_search,
+                                                   notifications=notifications, authenticate_user=authenticate_user)
+
             if added_movies is None:
                 log.error("Failed adding movies from Trakt's %s list", list_type)
                 time.sleep(10)