잔재미코딩

파이썬 surprise 라이브러리의 SVD 모델

풀스택을 위한 서버 기술과 도커 기본

혹시 본 자료가 도움이 되셨다면, 이런 식으로 잘 정리된 코드/자료로 만든 제 강의 한번 봐주실래요?

풀스택을 위한 서버 기술과 도커 기본

도커, 리눅스, AWS, nginx, HTTPS, flask 배포 기본

자세히 알아보기 모든 강좌 보기

7. 파이썬 surprise 라이브러리의 SVD 모델

  • The prediction $\hat{r}_{ui}$ is set as:

$\hat{r}_{ui} = \mu + b_u + b_i + q_i^Tp_u$

If user \(u\) is unknown, then the bias \(b_u\) and the factors \(p_u\) are assumed to be zero. The same applies for item \(i\) with \(b_i\) and \(q_i\).

  • 최소화는 SGD 사용 $$\begin{split}b_u \leftarrow b_u + \gamma (e_{ui} - \lambda b_u)\\ b_i \leftarrow b_i + \gamma (e_{ui} - \lambda b_i)\\ p_u \leftarrow p_u + \gamma (e_{ui} \cdot q_i - \lambda p_u)\\ q_i \leftarrow q_i + \gamma (e_{ui} \cdot p_u - \lambda q_i)\end{split}$$
  • 주요 파라미터
    • n_factors – The number of factors. Default is 100.
    • n_epochs – The number of iteration of the SGD procedure. Default is 20.
    • lr_all – The learning rate for all parameters. Default is 0.005

7.1. MovieLens 데이터를 기반으로 하는 실제 예제

MovieLens 데이터 로드

In [2]:
import os
import pandas as pd
from surprise import SVD
from surprise import Dataset
from surprise import dump


data = Dataset.load_builtin('ml-100k')
풀스택을 위한 서버 기술과 도커 기본

혹시 본 자료가 도움이 되셨다면, 이런 식으로 잘 정리된 코드/자료로 만든 제 강의 한번 봐주실래요?

풀스택을 위한 서버 기술과 도커 기본

도커, 리눅스, AWS, nginx, HTTPS, flask 배포 기본

자세히 알아보기 모든 강좌 보기
In [3]:
df = pd.DataFrame(data.raw_ratings, columns=["user", "item", "rate", "id"])
In [4]:
df.head()
Out[4]:
user item rate id
0 196 242 3.0 881250949
1 186 302 3.0 891717742
2 22 377 1.0 878887116
3 244 51 2.0 880606923
4 166 346 1.0 886397596

트레이닝 + 모델 저장

In [5]:
import os

from surprise import SVD
from surprise import Dataset
from surprise import dump


data = Dataset.load_builtin('ml-100k')
trainset = data.build_full_trainset()

algo = SVD()
algo.train(trainset)

# Compute predictions of the 'original' algorithm.
predictions = algo.test(trainset.build_testset())

# Dump algorithm and reload it.
file_name = os.path.expanduser('dump_file')
dump.dump(file_name, algo=algo)
_, loaded_algo = dump.load(file_name)

# We now ensure that the algo is still the same by checking the predictions.
predictions_loaded_algo = loaded_algo.test(trainset.build_testset())
assert predictions == predictions_loaded_algo
print('Predictions are the same')

Predictions are the same

정확도 계산

In [6]:
from surprise import Dataset
from surprise import SVD
from surprise import accuracy


data = Dataset.load_builtin('ml-100k')

algo = SVD()

trainset = data.build_full_trainset()
algo.train(trainset)

testset = trainset.build_testset()
predictions = algo.test(testset)

accuracy.rmse(predictions)

RMSE: 0.6763
Out[6]:
0.6763421071136434

모델 최적화 (파라미터 튜닝)

풀스택을 위한 서버 기술과 도커 기본

혹시 본 자료가 도움이 되셨다면, 이런 식으로 잘 정리된 코드/자료로 만든 제 강의 한번 봐주실래요?

풀스택을 위한 서버 기술과 도커 기본

도커, 리눅스, AWS, nginx, HTTPS, flask 배포 기본

자세히 알아보기 모든 강좌 보기
  • surprise의 GridSearch class
    • 주요 파라미터
      • algo_class (AlgoBase) – A class object of of the algorithm to evaluate.
      • param_grid (dict) – The dictionary has algo_class parameters as keys (string) and list of parameters as the desired values to try. All combinations will be evaluated with desired algorithm.
      • measures (list of string) – The performance measures to compute. Allowed names are function names as defined in the accuracy module. Default is ['rmse', 'mae'].
      • verbose (int) – Level of verbosity. If 0, nothing is printed. If 1, accuracy measures for each parameters combination are printed, with combination values. If 2, folds accuracy values are also printed. Default is 1
In [7]:
import random

from surprise import SVD
from surprise import Dataset
from surprise import accuracy
from surprise import GridSearch


# Load the full dataset.
data = Dataset.load_builtin('ml-100k')
data.split(n_folds=3)

# Select your best algo with grid search.
print('Grid Search...')
param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005]}
grid_search = GridSearch(SVD, param_grid, measures=['RMSE'], verbose=1)
print(grid_search.best_params)
grid_search.evaluate(data)

Grid Search...
defaultdict(<class 'list'>, {})
Running grid search for the following parameter combinations:
{'n_epochs': 5, 'lr_all': 0.002}
{'n_epochs': 5, 'lr_all': 0.005}
{'n_epochs': 10, 'lr_all': 0.002}
{'n_epochs': 10, 'lr_all': 0.005}
Resulsts:
{'n_epochs': 5, 'lr_all': 0.002}
{'RMSE': 0.98969008899623423}
----------
{'n_epochs': 5, 'lr_all': 0.005}
{'RMSE': 0.96388170867799128}
----------
{'n_epochs': 10, 'lr_all': 0.002}
{'RMSE': 0.96890656349975923}
----------
{'n_epochs': 10, 'lr_all': 0.005}
{'RMSE': 0.95219313886797485}
----------

사용자별 영화 추천 예 - offline 방식으로 별도 테이블을 만들어서 저장하고, 해당 사용자 로그인시 웹에서 추천 시나리오

In [16]:
from collections import defaultdict

from surprise import SVD
from surprise import Dataset


def get_top_n(predictions, n=10):
    '''Return the top-N recommendation for each user from a set of predictions.

    Args:
        predictions(list of Prediction objects): The list of predictions, as
            returned by the test method of an algorithm.
        n(int): The number of recommendation to output for each user. Default
            is 10.

    Returns:
    A dict where keys are user (raw) ids and values are lists of tuples:
        [(raw item id, rating estimation), ...] of size n.
    '''

    # First map the predictions to each user.
    top_n = defaultdict(list)
    for uid, iid, true_r, est, _ in predictions:
        top_n[uid].append((iid, est))

    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]

    return top_n


# First train an SVD algorithm on the movielens dataset.
data = Dataset.load_builtin('ml-100k')
trainset = data.build_full_trainset()
algo = SVD()
algo.train(trainset)

# Than predict ratings for all pairs (u, i) that are NOT in the training set.
testset = trainset.build_anti_testset()
predictions = algo.test(testset)

top_n = get_top_n(predictions, n=10)

# Print the recommended items for each user
for uid, user_ratings in top_n.items():
    print(uid, [iid for (iid, _) in user_ratings])

196 ['197', '64', '408', '474', '318', '963', '484', '127', '480', '193']
186 ['114', '611', '166', '656', '318', '83', '432', '59', '408', '169']
22 ['100', '183', '180', '135', '408', '12', '603', '178', '483', '357']
244 ['408', '515', '285', '480', '12', '127', '474', '1007', '483', '23']
166 ['172', '318', '169', '181', '408', '511', '357', '64', '515', '166']
298 ['173', '64', '408', '515', '513', '12', '89', '661', '258', '169']
115 ['179', '276', '285', '169', '408', '134', '180', '603', '1449', '483']
253 ['515', '174', '114', '493', '963', '603', '408', '251', '169', '357']
305 ['515', '657', '603', '513', '213', '496', '114', '694', '604', '659']
6 ['603', '657', '641', '190', '1449', '57', '654', '48', '958', '114']
62 ['169', '408', '427', '313', '603', '178', '223', '194', '963', '175']
286 ['480', '318', '427', '515', '474', '8', '641', '12', '306', '923']
200 ['302', '144', '114', '603', '59', '488', '493', '64', '178', '12']
210 ['318', '169', '64', '408', '513', '285', '272', '178', '191', '474']
224 ['64', '923', '513', '59', '496', '161', '742', '95', '357', '427']
303 ['169', '509', '313', '199', '48', '180', '498', '178', '193', '242']
122 ['285', '408', '483', '512', '515', '603', '59', '272', '528', '654']
194 ['169', '408', '1142', '313', '251', '525', '190', '963', '484', '603']
291 ['474', '408', '318', '169', '657', '190', '170', '114', '963', '320']
234 ['57', '302', '59', '408', '251', '169', '522', '856', '114', '1007']
119 ['408', '169', '513', '657', '474', '60', '603', '483', '48', '285']
167 ['483', '183', '515', '484', '488', '174', '127', '647', '1449', '71']
299 ['180', '64', '9', '530', '427', '199', '178', '223', '528', '750']
308 ['137', '1039', '1142', '251', '923', '474', '86', '114', '246', '963']
95 ['258', '408', '313', '169', '318', '12', '166', '272', '963', '19']
38 ['64', '333', '480', '462', '223', '316', '479', '197', '805', '553']
102 ['318', '64', '408', '169', '657', '474', '190', '178', '357', '480']
63 ['169', '357', '45', '318', '127', '223', '12', '474', '64', '178']
160 ['357', '12', '318', '98', '180', '134', '427', '64', '178', '654']
50 ['408', '923', '127', '169', '114', '483', '187', '318', '170', '64']
301 ['408', '169', '657', '114', '963', '923', '480', '223', '306', '485']
225 ['169', '59', '114', '611', '189', '313', '513', '515', '483', '168']
290 ['169', '178', '173', '963', '166', '251', '12', '427', '408', '513']
97 ['56', '12', '170', '64', '483', '480', '427', '48', '178', '318']
157 ['611', '474', '408', '511', '318', '98', '223', '56', '483', '512']
181 ['82', '692', '174', '64', '98', '963', '316', '204', '79', '210']
278 ['483', '12', '57', '318', '114', '59', '484', '657', '408', '474']
276 ['134', '191', '178', '114', '480', '190', '657', '511', '520', '663']
7 ['313', '1142', '302', '856', '1039', '293', '272', '408', '493', '242']
10 ['318', '427', '187', '272', '408', '923', '114', '79', '313', '1449']
284 ['178', '408', '483', '657', '427', '357', '480', '64', '318', '513']
201 ['493', '488', '657', '661', '641', '494', '169', '487', '498', '963']
287 ['169', '408', '285', '515', '134', '59', '603', '483', '189', '493']
246 ['89', '603', '302', '116', '474', '134', '114', '1142', '191', '408']
242 ['474', '515', '480', '174', '96', '272', '216', '483', '408', '178']
249 ['474', '641', '190', '1142', '180', '657', '530', '316', '127', '178']
99 ['318', '357', '136', '525', '408', '169', '114', '195', '272', '187']
178 ['496', '272', '705', '169', '513', '408', '923', '485', '251', '114']
251 ['318', '174', '357', '657', '195', '191', '98', '483', '169', '208']
81 ['64', '302', '286', '483', '272', '187', '480', '180', '265', '174']
260 ['408', '169', '190', '511', '223', '963', '12', '603', '205', '923']
25 ['483', '64', '59', '515', '318', '603', '513', '22', '172', '178']
59 ['114', '408', '124', '1073', '1142', '223', '302', '157', '178', '246']
72 ['408', '286', '169', '316', '143', '523', '657', '923', '251', '302']
87 ['313', '272', '483', '603', '169', '28', '178', '114', '59', '12']
42 ['313', '520', '178', '483', '170', '191', '22', '169', '657', '963']
292 ['318', '357', '12', '191', '474', '114', '427', '192', '498', '651']
20 ['528', '316', '408', '487', '173', '923', '275', '484', '315', '313']
13 ['136', '408', '1021', '644', '479', '169', '496', '945', '246', '1137']
138 ['408', '114', '50', '520', '205', '169', '189', '127', '923', '174']
60 ['408', '114', '169', '190', '318', '1194', '657', '198', '127', '963']
57 ['408', '169', '174', '172', '963', '114', '313', '483', '923', '165']
223 ['513', '174', '265', '59', '64', '172', '50', '187', '483', '482']
189 ['302', '427', '408', '12', '223', '169', '98', '357', '251', '515']
243 ['408', '474', '272', '603', '513', '427', '114', '479', '528', '313']
92 ['127', '187', '357', '178', '603', '114', '427', '272', '483', '657']
241 ['64', '480', '318', '479', '483', '408', '178', '474', '603', '657']
254 ['318', '483', '427', '223', '170', '963', '651', '520', '96', '528']
293 ['408', '197', '191', '178', '641', '269', '919', '318', '268', '515']
127 ['603', '480', '174', '96', '79', '172', '64', '169', '333', '429']
222 ['187', '169', '177', '663', '963', '511', '1142', '42', '129', '114']
267 ['134', '178', '357', '513', '316', '199', '641', '205', '302', '603']
11 ['272', '408', '512', '474', '285', '114', '64', '483', '59', '124']
8 ['64', '408', '169', '168', '178', '191', '100', '483', '98', '134']
162 ['127', '603', '963', '100', '64', '59', '169', '483', '178', '657']
279 ['199', '98', '613', '178', '496', '657', '936', '527', '701', '520']
145 ['127', '511', '318', '205', '179', '498', '272', '434', '208', '168']
28 ['169', '272', '172', '183', '408', '127', '963', '114', '318', '520']
135 ['169', '318', '408', '50', '483', '474', '64', '657', '172', '22']
32 ['483', '12', '603', '173', '318', '64', '515', '513', '480', '490']
90 ['484', '1449', '408', '487', '653', '114', '963', '655', '745', '510']
216 ['89', '176', '187', '484', '203', '180', '963', '641', '511', '478']
250 ['483', '513', '488', '657', '603', '178', '522', '512', '427', '205']
271 ['483', '251', '611', '1449', '484', '166', '641', '93', '223', '646']
265 ['923', '318', '134', '169', '488', '1039', '190', '8', '357', '197']
198 ['513', '408', '603', '124', '657', '178', '169', '483', '134', '12']
168 ['59', '513', '611', '114', '483', '480', '530', '408', '316', '923']
110 ['318', '513', '316', '59', '483', '511', '1194', '169', '427', '480']
58 ['285', '14', '114', '179', '154', '60', '23', '316', '86', '513']
237 ['318', '657', '170', '272', '515', '313', '251', '285', '57', '641']
94 ['285', '137', '919', '124', '511', '114', '1142', '116', '408', '198']
128 ['963', '272', '408', '313', '511', '661', '316', '12', '513', '285']
44 ['169', '178', '127', '12', '408', '199', '484', '1007', '661', '302']
264 ['427', '480', '496', '132', '174', '172', '483', '178', '127', '318']
41 ['114', '169', '89', '187', '923', '172', '480', '408', '483', '513']
82 ['165', '408', '478', '196', '59', '98', '136', '187', '114', '285']
262 ['408', '127', '12', '170', '603', '272', '357', '479', '513', '483']
174 ['22', '116', '64', '180', '516', '316', '921', '524', '48', '307']
43 ['483', '963', '515', '96', '1194', '481', '178', '520', '136', '357']
84 ['316', '923', '483', '515', '513', '603', '272', '169', '480', '251']
269 ['86', '958', '408', '652', '511', '60', '709', '275', '1137', '753']
259 ['127', '408', '603', '272', '483', '511', '50', '318', '191', '205']
85 ['603', '408', '178', '169', '285', '484', '302', '114', '1449', '653']
213 ['427', '480', '408', '178', '191', '498', '169', '316', '114', '302']
121 ['483', '64', '603', '511', '191', '302', '251', '657', '474', '170']
49 ['654', '474', '127', '603', '340', '23', '180', '357', '582', '169']
155 ['318', '480', '357', '127', '408', '114', '427', '64', '12', '603']
68 ['223', '190', '169', '64', '191', '408', '318', '483', '100', '427']
172 ['318', '64', '272', '22', '12', '963', '169', '408', '479', '83']
19 ['169', '64', '603', '483', '357', '316', '127', '408', '114', '511']
268 ['199', '657', '187', '318', '513', '430', '1194', '488', '64', '603']
5 ['484', '511', '134', '156', '187', '12', '246', '190', '7', '203']
80 ['169', '174', '408', '603', '657', '357', '98', '251', '173', '56']
66 ['318', '172', '174', '513', '923', '657', '483', '511', '114', '408']
18 ['272', '315', '114', '481', '124', '653', '657', '320', '251', '525']
26 ['318', '511', '603', '474', '357', '480', '513', '498', '483', '12']
130 ['408', '318', '520', '169', '923', '513', '59', '114', '320', '481']
256 ['272', '524', '661', '1194', '641', '923', '525', '603', '408', '318']
1 ['483', '1137', '408', '302', '320', '285', '318', '515', '1449', '512']
56 ['12', '318', '313', '923', '963', '302', '427', '657', '496', '223']
15 ['513', '657', '923', '172', '603', '528', '87', '511', '199', '963']
207 ['474', '963', '603', '496', '165', '519', '480', '408', '178', '498']
232 ['169', '427', '114', '408', '480', '190', '479', '134', '12', '89']
52 ['603', '408', '12', '272', '483', '64', '169', '50', '199', '114']
161 ['169', '357', '603', '114', '12', '64', '480', '190', '251', '512']
148 ['513', '483', '515', '657', '963', '480', '272', '170', '512', '64']
125 ['313', '265', '651', '480', '178', '169', '237', '603', '121', '12']
83 ['496', '313', '143', '923', '272', '528', '318', '427', '173', '487']
272 ['408', '114', '318', '64', '513', '316', '511', '488', '180', '480']
151 ['165', '48', '513', '269', '313', '285', '923', '936', '511', '316']
54 ['408', '89', '178', '515', '172', '64', '474', '479', '427', '134']
16 ['408', '169', '187', '313', '251', '186', '1039', '165', '132', '82']
91 ['496', '272', '173', '923', '178', '100', '196', '251', '316', '302']
294 ['318', '169', '172', '174', '64', '480', '173', '513', '98', '114']
229 ['357', '50', '318', '603', '100', '98', '483', '475', '496', '285']
36 ['408', '64', '169', '12', '513', '114', '923', '50', '178', '357']
70 ['318', '64', '12', '251', '515', '98', '178', '513', '603', '199']
14 ['169', '178', '114', '480', '136', '484', '246', '651', '185', '89']
295 ['474', '165', '514', '408', '189', '169', '923', '963', '114', '223']
233 ['513', '657', '479', '59', '316', '170', '474', '496', '408', '963']
214 ['923', '663', '183', '589', '272', '48', '654', '611', '963', '199']
192 ['169', '134', '12', '318', '408', '98', '64', '511', '136', '603']
100 ['318', '513', '515', '169', '480', '64', '59', '1137', '530', '190']
307 ['657', '223', '134', '603', '512', '190', '513', '199', '480', '285']
297 ['408', '276', '169', '483', '134', '189', '488', '60', '318', '190']
193 ['923', '169', '408', '318', '480', '483', '165', '603', '657', '427']
113 ['313', '272', '169', '513', '198', '285', '178', '12', '515', '474']
275 ['313', '483', '194', '59', '488', '178', '316', '641', '272', '190']
219 ['474', '408', '603', '483', '50', '173', '169', '1449', '190', '178']
218 ['318', '483', '357', '511', '272', '59', '316', '515', '479', '64']
123 ['169', '474', '211', '180', '519', '408', '611', '318', '603', '272']
158 ['603', '64', '357', '134', '169', '654', '192', '191', '515', '178']
302 ['12', '483', '318', '114', '657', '480', '408', '98', '185', '156']
23 ['12', '114', '474', '168', '169', '1142', '135', '198', '513', '963']
296 ['318', '12', '197', '302', '408', '1449', '603', '173', '64', '511']
33 ['12', '603', '169', '191', '64', '272', '127', '408', '318', '483']
154 ['56', '127', '408', '114', '64', '179', '318', '12', '657', '192']
77 ['408', '12', '169', '136', '525', '61', '469', '178', '603', '116']
270 ['302', '474', '1137', '515', '427', '512', '480', '132', '648', '479']
187 ['408', '169', '316', '50', '483', '511', '923', '199', '513', '603']
170 ['408', '178', '169', '611', '513', '114', '511', '923', '357', '483']
101 ['64', '408', '272', '520', '513', '923', '169', '178', '22', '136']
184 ['474', '603', '408', '169', '59', '921', '190', '1449', '269', '493']
112 ['408', '480', '59', '603', '511', '474', '483', '513', '189', '114']
133 ['483', '603', '169', '513', '64', '50', '187', '174', '474', '180']
215 ['657', '199', '318', '12', '408', '169', '178', '302', '498', '1019']
69 ['127', '483', '187', '64', '199', '173', '603', '661', '135', '318']
104 ['172', '174', '923', '408', '251', '12', '511', '483', '318', '22']
240 ['357', '318', '483', '12', '408', '64', '50', '172', '474', '114']
144 ['496', '408', '603', '169', '178', '132', '483', '513', '136', '512']
191 ['408', '134', '603', '64', '480', '127', '50', '185', '513', '511']
61 ['64', '318', '408', '242', '603', '98', '483', '169', '923', '100']
142 ['923', '313', '318', '50', '174', '172', '511', '513', '641', '59']
177 ['408', '483', '134', '199', '185', '114', '603', '285', '1142', '511']
203 ['408', '64', '12', '483', '169', '480', '657', '515', '174', '427']
21 ['197', '64', '318', '357', '12', '654', '208', '179', '603', '423']
197 ['496', '318', '64', '483', '194', '178', '28', '98', '12', '486']
134 ['169', '64', '174', '318', '408', '22', '603', '168', '173', '12']
180 ['513', '175', '479', '272', '89', '178', '313', '488', '64', '923']
236 ['190', '408', '357', '198', '208', '169', '114', '963', '923', '515']
263 ['603', '173', '178', '657', '408', '12', '474', '169', '479', '251']
109 ['313', '169', '333', '114', '1137', '963', '302', '408', '300', '59']
64 ['483', '408', '357', '657', '654', '611', '513', '641', '285', '170']
114 ['50', '12', '169', '64', '114', '178', '511', '603', '408', '181']
239 ['127', '316', '131', '275', '86', '285', '856', '515', '1142', '490']
117 ['408', '169', '114', '513', '89', '483', '318', '183', '64', '178']
65 ['313', '513', '302', '483', '59', '493', '515', '482', '923', '199']
137 ['474', '98', '603', '1137', '515', '512', '480', '132', '479', '272']
257 ['1449', '483', '513', '192', '318', '357', '611', '178', '169', '519']
111 ['169', '474', '174', '64', '318', '408', '513', '483', '178', '50']
285 ['12', '127', '316', '134', '408', '511', '657', '98', '190', '483']
96 ['114', '357', '408', '302', '251', '272', '511', '285', '480', '515']
116 ['169', '134', '513', '657', '654', '12', '182', '96', '64', '197']
73 ['408', '483', '603', '192', '134', '513', '641', '428', '114', '427']
221 ['474', '169', '89', '408', '9', '179', '190', '316', '480', '189']
235 ['127', '513', '657', '408', '357', '64', '641', '178', '663', '611']
164 ['480', '172', '318', '64', '22', '511', '963', '180', '657', '79']
281 ['483', '169', '515', '114', '480', '488', '408', '923', '64', '320']
182 ['513', '483', '408', '169', '192', '519', '285', '114', '174', '611']
129 ['483', '408', '64', '178', '357', '657', '513', '603', '285', '511']
45 ['427', '318', '174', '302', '603', '313', '513', '651', '191', '178']
131 ['318', '50', '169', '408', '357', '480', '427', '132', '172', '603']
230 ['318', '165', '59', '170', '483', '313', '408', '133', '692', '513']
126 ['408', '318', '169', '657', '127', '251', '285', '603', '519', '480']
231 ['408', '316', '64', '272', '496', '603', '513', '114', '318', '169']
280 ['475', '189', '175', '921', '408', '427', '285', '251', '190', '169']
288 ['483', '179', '479', '191', '611', '408', '474', '603', '498', '172']
152 ['603', '515', '480', '496', '178', '318', '705', '64', '357', '197']
217 ['197', '12', '318', '316', '170', '216', '272', '275', '64', '481']
79 ['89', '474', '408', '64', '169', '483', '603', '513', '178', '179']
75 ['50', '174', '169', '127', '272', '64', '187', '520', '195', '134']
245 ['513', '408', '923', '515', '519', '656', '483', '199', '657', '64']
282 ['408', '313', '483', '169', '318', '114', '50', '513', '134', '64']
78 ['50', '174', '114', '132', '318', '172', '511', '483', '480', '169']
118 ['302', '515', '480', '498', '96', '648', '195', '479', '272', '483']
283 ['408', '318', '357', '169', '923', '114', '320', '474', '127', '187']
171 ['408', '169', '64', '127', '187', '98', '79', '22', '511', '318']
107 ['318', '98', '185', '169', '483', '357', '178', '408', '251', '64']
226 ['64', '50', '190', '318', '173', '285', '228', '178', '114', '22']
306 ['657', '357', '603', '408', '50', '134', '483', '64', '169', '48']
173 ['474', '98', '193', '603', '196', '515', '512', '480', '23', '317']
185 ['408', '320', '169', '474', '483', '168', '191', '134', '657', '64']
150 ['408', '169', '12', '64', '178', '156', '427', '318', '513', '272']
274 ['174', '483', '408', '64', '169', '114', '12', '513', '657', '22']
188 ['496', '251', '603', '169', '408', '178', '604', '515', '427', '165']
48 ['12', '100', '272', '316', '318', '178', '408', '313', '64', '22']
311 ['169', '408', '272', '87', '481', '529', '316', '632', '659', '513']
165 ['50', '408', '64', '178', '89', '313', '603', '659', '12', '173']
208 ['69', '496', '64', '318', '603', '513', '408', '480', '178', '169']
2 ['12', '169', '174', '22', '408', '64', '603', '96', '427', '173']
205 ['64', '408', '483', '170', '515', '169', '12', '923', '173', '114']
248 ['408', '59', '169', '12', '483', '48', '285', '134', '513', '272']
93 ['318', '64', '89', '183', '515', '483', '251', '408', '173', '480']
159 ['429', '511', '223', '316', '923', '172', '521', '313', '735', '98']
146 ['50', '127', '603', '483', '98', '480', '318', '89', '12', '172']
29 ['408', '483', '474', '169', '603', '513', '178', '64', '60', '114']
156 ['169', '408', '603', '114', '483', '174', '657', '316', '127', '98']
37 ['318', '483', '12', '100', '208', '408', '64', '199', '603', '132']
141 ['272', '22', '963', '64', '318', '210', '692', '174', '511', '633']
195 ['511', '318', '272', '172', '133', '64', '603', '50', '12', '657']
108 ['64', '474', '408', '480', '169', '302', '479', '641', '197', '178']
47 ['603', '483', '169', '199', '512', '318', '127', '511', '480', '178']
255 ['318', '169', '12', '114', '357', '127', '190', '285', '498', '408']
89 ['603', '513', '483', '185', '199', '178', '408', '98', '657', '515']
140 ['127', '483', '480', '178', '316', '408', '169', '427', '603', '50']
190 ['172', '12', '408', '64', '22', '168', '114', '963', '174', '480']
24 ['174', '50', '185', '513', '963', '114', '89', '923', '199', '408']
17 ['50', '483', '408', '318', '515', '474', '963', '64', '174', '169']
313 ['512', '1194', '513', '169', '408', '648', '528', '179', '641', '510']
53 ['318', '357', '313', '513', '427', '272', '483', '98', '127', '133']
124 ['169', '189', '408', '654', '511', '114', '285', '313', '923', '611']
149 ['318', '474', '483', '169', '923', '657', '488', '613', '132', '86']
176 ['169', '483', '179', '603', '408', '134', '114', '98', '178', '199']
106 ['98', '408', '483', '316', '357', '178', '488', '484', '272', '513']
312 ['272', '318', '59', '313', '963', '958', '707', '64', '1449', '12']
175 ['174', '178', '143', '480', '318', '603', '513', '191', '169', '302']
153 ['408', '603', '185', '318', '192', '114', '474', '512', '169', '528']
220 ['318', '169', '98', '408', '64', '56', '474', '127', '483', '513']
143 ['64', '427', '127', '480', '657', '174', '408', '357', '963', '170']
199 ['474', '483', '318', '357', '180', '64', '89', '127', '199', '169']
202 ['169', '313', '963', '528', '316', '427', '83', '190', '513', '511']
277 ['169', '190', '114', '408', '318', '64', '98', '272', '513', '57']
206 ['316', '12', '191', '480', '318', '568', '64', '513', '427', '136']
76 ['408', '127', '611', '178', '357', '187', '483', '169', '242', '185']
314 ['313', '480', '316', '79', '272', '488', '191', '923', '531', '133']
136 ['178', '64', '187', '98', '474', '251', '603', '285', '12', '357']
179 ['98', '64', '318', '194', '223', '480', '496', '651', '170', '178']
4 ['302', '474', '603', '100', '515', '496', '479', '483', '408', '178']
304 ['172', '174', '127', '64', '12', '408', '169', '98', '511', '480']
3 ['98', '657', '493', '478', '190', '513', '923', '86', '1039', '191']
227 ['357', '318', '192', '483', '98', '56', '199', '611', '48', '511']
252 ['474', '408', '318', '64', '169', '12', '190', '657', '185', '357']
212 ['480', '114', '169', '258', '408', '603', '474', '659', '513', '1007']
310 ['408', '318', '127', '69', '427', '114', '657', '513', '357', '496']
35 ['483', '272', '511', '923', '316', '114', '513', '427', '57', '189']
147 ['357', '169', '114', '408', '483', '318', '657', '64', '170', '963']
105 ['169', '408', '114', '12', '174', '50', '64', '318', '178', '483']
34 ['603', '272', '172', '48', '483', '408', '357', '191', '134', '169']
71 ['318', '513', '408', '474', '12', '127', '480', '114', '192', '659']
51 ['12', '483', '199', '169', '511', '100', '98', '89', '318', '190']
204 ['513', '923', '127', '661', '657', '408', '963', '528', '641', '57']
315 ['483', '50', '134', '474', '191', '169', '357', '89', '192', '14']
31 ['50', '64', '173', '89', '408', '479', '603', '513', '172', '483']
316 ['169', '408', '653', '657', '528', '488', '272', '958', '285', '603']
103 ['12', '285', '603', '89', '64', '483', '318', '313', '190', '408']
318 ['313', '511', '169', '316', '603', '272', '408', '963', '661', '276']
30 ['408', '272', '357', '169', '64', '12', '513', '603', '474', '114']
120 ['318', '474', '64', '178', '480', '641', '408', '483', '479', '603']
46 ['187', '169', '408', '316', '64', '318', '9', '513', '180', '12']
289 ['64', '318', '22', '513', '48', '216', '474', '197', '178', '963']
209 ['169', '318', '199', '98', '513', '496', '172', '603', '174', '187']
261 ['603', '100', '427', '512', '480', '498', '479', '272', '483', '408']
88 ['318', '357', '64', '178', '659', '408', '12', '511', '923', '513']
9 ['603', '408', '127', '318', '64', '313', '357', '12', '427', '480']
247 ['408', '483', '114', '513', '199', '318', '169', '59', '12', '22']
321 ['169', '272', '172', '178', '488', '408', '168', '318', '114', '285']
266 ['408', '169', '318', '483', '480', '498', '64', '657', '603', '59']
74 ['357', '318', '923', '603', '316', '512', '480', '483', '427', '479']
238 ['483', '98', '318', '408', '488', '484', '1194', '197', '223', '64']
319 ['169', '12', '513', '408', '603', '89', '98', '357', '50', '114']
323 ['12', '187', '408', '169', '654', '313', '427', '513', '192', '603']
67 ['408', '127', '318', '56', '169', '12', '963', '657', '98', '251']
211 ['474', '408', '98', '513', '603', '496', '189', '488', '272', '1007']
98 ['89', '511', '316', '64', '272', '357', '57', '223', '12', '513']
12 ['272', '64', '22', '313', '483', '408', '12', '59', '190', '316']
40 ['50', '408', '64', '174', '318', '187', '483', '657', '172', '183']
258 ['50', '172', '483', '603', '318', '479', '89', '12', '181', '127']
228 ['408', '513', '318', '185', '12', '483', '114', '488', '487', '480']
325 ['652', '89', '124', '64', '169', '513', '512', '479', '657', '603']
320 ['408', '64', '603', '513', '169', '474', '178', '190', '657', '272']
326 ['19', '607', '707', '641', '100', '57', '432', '408', '251', '83']
327 ['483', '223', '488', '134', '519', '427', '661', '316', '1449', '59']
183 ['657', '190', '178', '408', '318', '64', '114', '923', '98', '707']
328 ['173', '496', '513', '603', '479', '136', '709', '190', '648', '408']
322 ['408', '180', '657', '199', '647', '474', '114', '480', '98', '515']
330 ['265', '165', '480', '272', '79', '483', '408', '134', '169', '923']
27 ['313', '923', '408', '64', '169', '318', '480', '272', '127', '22']
331 ['318', '474', '316', '169', '657', '197', '98', '513', '480', '357']
332 ['474', '603', '515', '512', '480', '408', '178', '318', '199', '169']
329 ['318', '64', '603', '178', '513', '480', '511', '484', '479', '315']
86 ['318', '483', '127', '178', '169', '408', '603', '98', '479', '493']
139 ['357', '318', '408', '169', '313', '134', '603', '511', '1449', '513']
300 ['515', '483', '318', '64', '169', '173', '357', '50', '923', '98']
163 ['114', '168', '313', '408', '483', '169', '132', '963', '488', '659']
333 ['272', '169', '603', '313', '408', '479', '12', '178', '114', '64']
334 ['114', '251', '511', '589', '64', '611', '641', '480', '651', '156']
39 ['50', '483', '496', '174', '98', '318', '178', '657', '515', '64']
324 ['474', '98', '603', '515', '165', '480', '496', '174', '272', '79']
132 ['408', '114', '169', '64', '480', '923', '603', '318', '98', '197']
336 ['169', '48', '12', '64', '203', '430', '408', '177', '174', '60']
335 ['641', '89', '474', '180', '483', '357', '223', '318', '480', '272']
169 ['513', '923', '285', '169', '64', '318', '408', '496', '173', '313']
338 ['127', '114', '64', '89', '285', '190', '14', '12', '515', '272']
339 ['408', '285', '114', '272', '1142', '223', '169', '647', '116', '513']
309 ['174', '50', '318', '313', '22', '172', '173', '64', '132', '483']
342 ['127', '603', '185', '302', '187', '474', '178', '272', '64', '180']
340 ['318', '64', '659', '313', '527', '519', '513', '515', '496', '210']
317 ['272', '483', '174', '50', '64', '963', '515', '603', '357', '169']
341 ['98', '480', '174', '96', '483', '408', '127', '318', '56', '89']
343 ['190', '285', '172', '251', '178', '114', '192', '647', '603', '171']
344 ['134', '320', '242', '185', '483', '56', '603', '200', '180', '187']
345 ['127', '513', '134', '603', '496', '408', '474', '654', '178', '190']
346 ['408', '963', '663', '357', '169', '483', '480', '191', '522', '529']
347 ['42', '169', '408', '429', '64', '114', '135', '480', '705', '178']
273 ['127', '50', '513', '199', '187', '57', '603', '483', '98', '661']
55 ['178', '318', '603', '661', '408', '483', '513', '64', '169', '134']
349 ['480', '408', '169', '318', '64', '134', '178', '357', '275', '483']
348 ['98', '272', '483', '318', '64', '12', '173', '22', '513', '316']
354 ['114', '408', '178', '921', '641', '474', '923', '1103', '856', '488']
351 ['515', '15', '480', '174', '272', '172', '483', '408', '178', '318']
358 ['190', '169', '89', '408', '189', '923', '513', '474', '488', '168']
352 ['185', '191', '169', '127', '318', '313', '963', '180', '480', '199']
360 ['484', '318', '923', '316', '603', '514', '190', '922', '12', '114']
363 ['483', '132', '1137', '427', '963', '48', '135', '480', '199', '14']
355 ['174', '272', '483', '408', '178', '318', '64', '169', '12', '531']
362 ['169', '98', '603', '408', '272', '64', '12', '483', '178', '190']
357 ['144', '98', '603', '427', '15', '480', '498', '174', '195', '79']
356 ['408', '169', '64', '657', '318', '603', '474', '479', '114', '48']
361 ['318', '480', '357', '408', '114', '134', '302', '169', '127', '483']
365 ['114', '408', '318', '320', '56', '169', '474', '657', '223', '488']
350 ['408', '12', '318', '178', '513', '272', '357', '963', '64', '114']
367 ['603', '272', '48', '408', '127', '318', '11', '198', '64', '357']
368 ['64', '169', '22', '12', '318', '657', '603', '357', '408', '496']
371 ['318', '169', '498', '480', '657', '483', '114', '408', '923', '172']
373 ['657', '483', '57', '408', '61', '515', '963', '60', '251', '513']
370 ['318', '515', '483', '496', '316', '191', '357', '59', '127', '251']
374 ['59', '313', '519', '923', '300', '169', '663', '1137', '966', '603']
372 ['603', '427', '512', '480', '174', '648', '479', '272', '48', '483']
337 ['408', '12', '114', '64', '169', '313', '318', '48', '190', '480']
378 ['483', '169', '515', '408', '513', '603', '178', '170', '493', '357']
366 ['474', '86', '603', '427', '512', '480', '479', '272', '483', '408']
377 ['50', '12', '483', '408', '318', '357', '64', '302', '169', '89']
375 ['408', '169', '114', '174', '480', '64', '127', '318', '483', '50']
359 ['169', '923', '272', '511', '483', '12', '513', '114', '171', '318']
379 ['408', '170', '318', '132', '302', '134', '483', '316', '647', '169']
380 ['178', '488', '275', '57', '408', '513', '169', '657', '641', '478']
381 ['474', '408', '12', '286', '185', '64', '515', '169', '302', '199']
385 ['10', '179', '647', '86', '315', '14', '516', '921', '170', '709']
382 ['603', '313', '316', '169', '318', '480', '657', '513', '170', '709']
387 ['45', '9', '185', '792', '134', '240', '480', '525', '137', '654']
364 ['483', '318', '174', '357', '114', '64', '480', '513', '963', '511']
369 ['174', '515', '64', '603', '657', '963', '12', '357', '318', '272']
388 ['603', '143', '515', '427', '480', '496', '174', '79', '172', '408']
386 ['357', '603', '511', '64', '169', '98', '408', '12', '190', '520']
389 ['408', '169', '522', '114', '313', '223', '357', '511', '647', '512']
383 ['408', '56', '175', '169', '127', '98', '187', '192', '12', '114']
390 ['169', '174', '64', '178', '483', '474', '50', '318', '511', '98']
393 ['515', '656', '513', '923', '641', '699', '528', '516', '1007', '963']
392 ['483', '135', '187', '408', '427', '603', '124', '496', '357', '484']
376 ['313', '50', '513', '318', '134', '169', '174', '483', '408', '515']
394 ['127', '64', '603', '357', '474', '408', '187', '258', '318', '178']
391 ['408', '316', '169', '285', '709', '189', '519', '515', '654', '114']
398 ['169', '318', '210', '313', '513', '302', '251', '963', '408', '272']
397 ['64', '114', '180', '89', '187', '408', '191', '169', '179', '190']
399 ['513', '474', '169', '657', '408', '1194', '272', '921', '963', '483']
396 ['657', '64', '513', '488', '408', '483', '59', '923', '316', '705']
401 ['59', '657', '169', '487', '15', '313', '427', '79', '496', '190']
402 ['963', '318', '313', '657', '114', '272', '169', '302', '316', '59']
384 ['100', '318', '64', '50', '174', '169', '603', '127', '408', '172']
395 ['173', '408', '483', '169', '496', '603', '114', '513', '178', '79']
353 ['318', '483', '134', '169', '474', '223', '513', '100', '603', '408']
403 ['169', '408', '89', '64', '48', '1194', '178', '963', '480', '657']
405 ['164', '705', '144', '1039', '237', '496', '531', '270', '490', '405']
400 ['603', '98', '483', '100', '127', '318', '64', '173', '285', '169']
406 ['313', '603', '484', '192', '963', '178', '646', '169', '1019', '525']
407 ['480', '187', '318', '178', '64', '12', '192', '488', '114', '963']
409 ['169', '408', '641', '443', '313', '488', '272', '258', '515', '651']
404 ['191', '169', '204', '318', '12', '174', '657', '736', '50', '641']
413 ['483', '169', '127', '134', '318', '174', '165', '709', '513', '223']
416 ['208', '963', '408', '89', '198', '513', '641', '483', '1063', '516']
408 ['318', '98', '483', '520', '515', '496', '169', '50', '408', '64']
410 ['50', '513', '172', '199', '114', '474', '205', '179', '12', '318']
411 ['483', '64', '513', '357', '169', '657', '519', '479', '178', '661']
417 ['408', '318', '114', '316', '59', '272', '523', '657', '22', '187']
412 ['50', '272', '12', '89', '187', '127', '483', '511', '513', '181']
420 ['56', '114', '156', '192', '474', '511', '169', '180', '654', '134']
422 ['169', '483', '190', '134', '64', '187', '493', '923', '316', '318']
425 ['657', '528', '408', '169', '320', '114', '919', '135', '251', '483']
419 ['318', '127', '12', '64', '172', '187', '178', '316', '408', '357']
415 ['408', '169', '50', '178', '513', '114', '318', '302', '474', '127']
423 ['169', '496', '64', '50', '318', '178', '483', '408', '174', '251']
429 ['135', '169', '408', '9', '923', '488', '657', '478', '14', '313']
428 ['474', '100', '483', '408', '127', '318', '56', '64', '357', '169']
427 ['242', '474', '95', '246', '98', '193', '194', '603', '201', '1137']
418 ['64', '483', '173', '318', '603', '114', '12', '168', '169', '316']
424 ['408', '64', '22', '318', '651', '481', '98', '169', '197', '493']
432 ['923', '64', '318', '513', '408', '480', '316', '487', '488', '272']
421 ['357', '511', '180', '963', '661', '480', '651', '316', '169', '318']
435 ['514', '515', '963', '205', '478', '1007', '475', '529', '530', '189']
433 ['483', '127', '408', '357', '427', '318', '611', '169', '134', '187']
426 ['272', '408', '114', '59', '515', '513', '963', '169', '57', '64']
436 ['169', '89', '12', '272', '175', '190', '408', '514', '171', '56']
430 ['603', '474', '511', '357', '316', '114', '484', '408', '483', '156']
434 ['174', '50', '513', '96', '313', '429', '64', '169', '172', '568']
437 ['427', '647', '661', '272', '513', '285', '604', '317', '136', '313']
438 ['172', '408', '114', '313', '64', '178', '483', '174', '132', '316']
431 ['318', '483', '408', '64', '169', '657', '12', '199', '190', '132']
442 ['179', '205', '357', '488', '50', '178', '483', '127', '135', '180']
440 ['483', '357', '169', '12', '513', '187', '22', '89', '156', '320']
445 ['357', '169', '285', '483', '180', '923', '275', '192', '659', '178']
447 ['172', '511', '272', '479', '199', '136', '427', '923', '173', '317']
449 ['134', '474', '483', '408', '114', '89', '187', '178', '169', '654']
450 ['408', '513', '963', '165', '285', '958', '524', '515', '48', '945']
446 ['408', '114', '657', '169', '603', '496', '483', '528', '427', '479']
439 ['408', '483', '169', '64', '302', '50', '923', '603', '191', '651']
451 ['22', '313', '174', '82', '318', '963', '210', '735', '520', '497']
452 ['315', '487', '251', '657', '12', '205', '511', '429', '651', '200']
454 ['208', '963', '923', '513', '216', '1007', '178', '170', '265', '83']
453 ['127', '187', '89', '135', '64', '96', '176', '195', '169', '1142']
414 ['474', '513', '56', '408', '178', '169', '180', '134', '190', '603']
455 ['483', '408', '479', '302', '187', '655', '169', '603', '251', '285']
444 ['179', '56', '318', '114', '480', '474', '187', '98', '357', '172']
448 ['50', '483', '127', '318', '100', '478', '98', '169', '498', '174']
457 ['603', '427', '496', '408', '178', '480', '316', '272', '198', '657']
456 ['169', '408', '657', '709', '134', '223', '48', '513', '64', '190']
458 ['197', '156', '653', '177', '606', '175', '923', '856', '504', '646']
462 ['318', '64', '114', '408', '511', '357', '483', '134', '513', '12']
459 ['210', '483', '963', '318', '272', '64', '923', '513', '189', '313']
460 ['178', '169', '483', '408', '318', '603', '474', '12', '657', '480']
461 ['169', '100', '12', '408', '127', '272', '64', '114', '318', '357']
467 ['603', '285', '474', '169', '56', '12', '134', '357', '483', '318']
468 ['483', '657', '480', '641', '511', '496', '134', '302', '479', '474']
466 ['8', '408', '189', '427', '483', '611', '169', '272', '114', '480']
472 ['474', '246', '98', '515', '512', '165', '480', '209', '479', '272']
465 ['178', '515', '89', '272', '480', '479', '176', '427', '483', '242']
463 ['179', '185', '272', '512', '408', '182', '191', '316', '197', '318']
471 ['316', '272', '275', '134', '651', '408', '48', '603', '169', '114']
474 ['408', '272', '169', '156', '223', '251', '114', '153', '512', '429']
469 ['427', '318', '408', '169', '178', '657', '98', '480', '357', '528']
464 ['427', '496', '318', '64', '480', '357', '174', '22', '98', '132']
476 ['272', '318', '474', '603', '408', '515', '133', '98', '169', '480']
478 ['185', '127', '191', '474', '661', '657', '192', '488', '408', '180']
473 ['64', '316', '513', '318', '483', '519', '657', '641', '59', '178']
470 ['318', '357', '313', '191', '64', '127', '474', '923', '408', '427']
480 ['513', '408', '22', '318', '14', '357', '427', '238', '134', '285']
441 ['318', '483', '285', '408', '513', '64', '515', '488', '923', '484']
479 ['170', '114', '14', '963', '64', '56', '484', '246', '87', '603']
484 ['12', '64', '483', '603', '408', '272', '169', '178', '251', '302']
486 ['654', '483', '603', '183', '512', '98', '318', '474', '185', '114']
487 ['513', '633', '187', '511', '59', '496', '603', '483', '523', '611']
482 ['483', '318', '64', '172', '519', '657', '480', '408', '174', '169']
481 ['483', '357', '603', '127', '178', '515', '496', '169', '285', '64']
492 ['520', '144', '174', '183', '22', '79', '12', '98', '195', '166']
493 ['313', '496', '64', '427', '511', '144', '510', '966', '169', '513']
490 ['357', '172', '174', '223', '12', '408', '659', '318', '191', '169']
489 ['265', '98', '496', '174', '96', '318', '64', '173', '22', '923']
483 ['483', '603', '496', '169', '408', '427', '251', '657', '285', '493']
496 ['169', '171', '474', '175', '269', '134', '272', '45', '189', '12']
494 ['318', '408', '172', '178', '511', '963', '12', '114', '173', '169']
495 ['408', '169', '48', '484', '137', '180', '223', '923', '190', '963']
477 ['302', '474', '86', '98', '194', '603', '515', '427', '512', '480']
497 ['515', '513', '272', '313', '205', '490', '484', '963', '474', '483']
488 ['313', '169', '12', '408', '272', '195', '204', '194', '657', '316']
498 ['114', '408', '513', '641', '427', '86', '316', '357', '1137', '654']
499 ['114', '64', '513', '408', '22', '45', '170', '302', '197', '169']
491 ['272', '316', '114', '427', '132', '496', '480', '169', '318', '191']
500 ['408', '512', '1007', '114', '663', '12', '187', '64', '474', '647']
502 ['272', '603', '316', '484', '408', '515', '12', '64', '480', '488']
503 ['483', '513', '589', '493', '647', '64', '657', '921', '923', '745']
504 ['923', '190', '659', '198', '272', '657', '313', '528', '189', '408']
505 ['59', '318', '272', '515', '427', '603', '483', '513', '57', '143']
506 ['313', '318', '144', '22', '64', '480', '513', '511', '1194', '427']
443 ['357', '64', '483', '50', '178', '963', '174', '172', '511', '48']
507 ['242', '346', '474', '265', '86', '144', '1', '95', '98', '194']
514 ['56', '57', '223', '171', '603', '251', '657', '316', '489', '963']
508 ['408', '603', '483', '64', '12', '14', '127', '479', '156', '285']
511 ['127', '64', '199', '169', '285', '603', '483', '408', '185', '318']
515 ['603', '64', '178', '511', '408', '169', '272', '89', '357', '480']
512 ['169', '408', '483', '357', '190', '114', '474', '64', '98', '89']
513 ['98', '480', '174', '272', '483', '408', '318', '64', '484', '22']
475 ['318', '483', '251', '98', '515', '208', '174', '357', '511', '12']
523 ['603', '515', '496', '174', '272', '483', '178', '318', '64', '22']
518 ['513', '483', '59', '316', '166', '15', '265', '71', '174', '272']
509 ['174', '318', '64', '12', '178', '98', '483', '127', '195', '480']
516 ['408', '64', '318', '114', '513', '480', '483', '168', '12', '223']
510 ['174', '50', '169', '408', '191', '963', '64', '133', '483', '480']
524 ['315', '1137', '489', '272', '313', '61', '675', '316', '114', '1194']
501 ['187', '12', '357', '408', '178', '513', '180', '64', '50', '134']
525 ['172', '169', '50', '174', '408', '114', '12', '318', '302', '64']
521 ['169', '408', '199', '479', '251', '511', '316', '603', '272', '657']
520 ['169', '408', '603', '318', '64', '528', '483', '275', '657', '513']
519 ['242', '302', '474', '1', '98', '194', '603', '100', '143', '515']
528 ['172', '169', '480', '357', '483', '313', '318', '199', '205', '89']
532 ['174', '172', '408', '178', '69', '64', '169', '50', '28', '190']
530 ['316', '318', '923', '408', '169', '272', '114', '86', '315', '656']
531 ['169', '50', '408', '172', '478', '174', '64', '176', '483', '79']
529 ['408', '480', '114', '513', '313', '272', '169', '603', '12', '474']
517 ['178', '169', '408', '483', '318', '513', '192', '64', '174', '114']
527 ['89', '272', '79', '923', '521', '408', '98', '313', '589', '194']
485 ['318', '100', '98', '483', '357', '50', '251', '511', '64', '12']
533 ['656', '515', '482', '79', '963', '178', '923', '1194', '513', '166']
535 ['313', '408', '513', '199', '474', '641', '606', '169', '208', '272']
536 ['12', '173', '64', '963', '657', '223', '114', '178', '923', '513']
526 ['114', '483', '134', '56', '64', '178', '357', '156', '705', '12']
537 ['408', '1240', '223', '169', '9', '246', '156', '154', '114', '656']
534 ['474', '603', '143', '515', '427', '165', '480', '174', '272', '483']
541 ['64', '22', '178', '923', '966', '45', '483', '169', '12', '61']
538 ['180', '302', '179', '64', '408', '132', '513', '313', '657', '520']
542 ['408', '185', '169', '603', '483', '178', '611', '114', '134', '657']
545 ['12', '64', '513', '923', '114', '313', '178', '318', '657', '190']
539 ['64', '169', '318', '480', '408', '515', '98', '493', '114', '178']
547 ['408', '169', '483', '127', '603', '12', '64', '114', '657', '357']
543 ['408', '657', '483', '127', '963', '661', '641', '511', '484', '48']
548 ['195', '174', '96', '172', '230', '228', '59', '173', '265', '484']
546 ['480', '496', '174', '127', '69', '357', '169', '187', '180', '657']
522 ['603', '408', '169', '114', '127', '187', '484', '483', '64', '923']
551 ['190', '189', '169', '114', '197', '515', '165', '480', '484', '1143']
544 ['923', '169', '513', '408', '318', '190', '114', '178', '64', '134']
553 ['185', '127', '64', '318', '408', '180', '169', '179', '12', '114']
552 ['513', '408', '657', '483', '313', '178', '318', '169', '923', '64']
540 ['127', '134', '64', '483', '408', '603', '169', '513', '480', '187']
554 ['357', '64', '496', '657', '313', '170', '483', '169', '427', '480']
550 ['96', '603', '12', '318', '483', '228', '169', '488', '64', '493']
556 ['100', '174', '169', '272', '50', '661', '191', '194', '285', '357']
559 ['178', '488', '483', '59', '114', '408', '169', '64', '192', '657']
560 ['64', '408', '178', '169', '285', '488', '513', '114', '603', '190']
561 ['408', '114', '137', '647', '192', '525', '654', '641', '653', '190']
563 ['603', '408', '169', '173', '64', '127', '134', '483', '178', '22']
566 ['408', '169', '9', '357', '654', '515', '923', '272', '185', '603']
557 ['603', '134', '515', '480', '169', '923', '408', '483', '98', '318']
558 ['114', '603', '50', '408', '483', '169', '183', '513', '12', '589']
564 ['318', '483', '178', '603', '519', '611', '498', '174', '98', '79']
565 ['474', '603', '479', '483', '408', '64', '183', '169', '12', '173']
573 ['169', '59', '483', '603', '98', '173', '64', '357', '12', '515']
549 ['64', '408', '318', '169', '12', '483', '603', '22', '178', '963']
567 ['408', '515', '86', '189', '169', '923', '57', '1143', '528', '921']
569 ['173', '474', '408', '480', '318', '135', '127', '272', '114', '169']
562 ['487', '520', '64', '316', '496', '408', '519', '265', '192', '641']
576 ['318', '22', '64', '513', '408', '169', '12', '173', '190', '483']
577 ['483', '513', '474', '178', '515', '316', '963', '408', '59', '484']
579 ['313', '272', '174', '64', '22', '318', '316', '963', '648', '166']
574 ['50', '172', '178', '511', '483', '318', '169', '64', '174', '114']
555 ['408', '134', '64', '127', '483', '963', '1194', '923', '190', '251']
572 ['318', '357', '127', '963', '511', '98', '12', '64', '408', '923']
575 ['64', '515', '169', '923', '22', '114', '242', '408', '511', '12']
584 ['318', '483', '208', '169', '178', '195', '408', '190', '513', '174']
588 ['513', '408', '64', '641', '182', '187', '474', '175', '427', '487']
587 ['474', '98', '483', '318', '1449', '513', '488', '59', '185', '199']
568 ['50', '169', '166', '318', '285', '197', '124', '172', '64', '408']
586 ['513', '272', '474', '316', '306', '169', '189', '286', '313', '408']
585 ['187', '318', '408', '64', '114', '127', '513', '483', '515', '480']
582 ['272', '483', '603', '127', '180', '135', '172', '179', '156', '183']
591 ['178', '515', '169', '318', '474', '124', '316', '251', '483', '59']
581 ['272', '318', '169', '313', '98', '166', '14', '64', '12', '603']
592 ['175', '171', '488', '513', '661', '199', '190', '156', '478', '520']
580 ['357', '318', '12', '127', '496', '64', '22', '483', '174', '316']
590 ['408', '169', '302', '483', '12', '223', '64', '657', '474', '357']
593 ['513', '474', '96', '408', '64', '22', '195', '316', '176', '483']
583 ['318', '64', '169', '50', '172', '114', '127', '89', '408', '480']
596 ['408', '169', '474', '64', '114', '657', '12', '316', '318', '302']
570 ['272', '114', '169', '408', '316', '513', '12', '134', '178', '511']
599 ['98', '603', '318', '357', '191', '185', '923', '511', '657', '83']
589 ['174', '963', '12', '513', '96', '114', '318', '22', '186', '181']
594 ['272', '513', '318', '64', '480', '170', '408', '603', '657', '1194']
597 ['657', '272', '408', '64', '513', '190', '169', '480', '45', '663']
578 ['408', '318', '427', '64', '489', '169', '191', '174', '12', '963']
601 ['169', '603', '199', '408', '187', '493', '136', '134', '474', '14']
602 ['318', '357', '480', '174', '64', '12', '408', '172', '178', '963']
600 ['483', '408', '180', '64', '603', '173', '318', '178', '191', '1194']
605 ['511', '603', '272', '474', '251', '48', '114', '134', '178', '23']
603 ['169', '408', '318', '114', '302', '657', '480', '483', '513', '272']
595 ['56', '169', '302', '318', '64', '357', '134', '98', '173', '190']
606 ['520', '272', '513', '169', '357', '603', '318', '205', '114', '515']
608 ['316', '156', '272', '285', '488', '896', '430', '198', '315', '519']
607 ['12', '603', '408', '318', '427', '64', '134', '285', '923', '192']
610 ['603', '100', '513', '64', '57', '132', '654', '178', '511', '134']
611 ['169', '408', '59', '923', '513', '192', '483', '64', '318', '520']
617 ['432', '114', '59', '1449', '408', '512', '603', '487', '315', '182']
618 ['357', '520', '511', '272', '488', '194', '208', '611', '199', '480']
614 ['474', '318', '313', '496', '657', '513', '64', '483', '132', '178']
609 ['191', '427', '923', '302', '511', '59', '896', '318', '513', '494']
615 ['50', '511', '513', '169', '408', '174', '64', '12', '313', '134']
616 ['98', '483', '408', '515', '657', '64', '169', '496', '178', '513']
620 ['515', '272', '318', '474', '408', '316', '64', '12', '513', '427']
571 ['963', '513', '272', '408', '483', '641', '169', '515', '479', '603']
619 ['178', '318', '191', '204', '496', '199', '196', '498', '210', '185']
613 ['483', '515', '178', '313', '496', '657', '169', '511', '474', '512']
622 ['318', '923', '191', '357', '203', '285', '124', '251', '14', '1142']
621 ['12', '190', '114', '199', '408', '42', '169', '513', '318', '272']
604 ['483', '172', '12', '515', '170', '50', '64', '174', '318', '114']
624 ['318', '357', '483', '114', '64', '511', '187', '223', '923', '427']
612 ['169', '483', '408', '515', '178', '59', '302', '603', '313', '657']
627 ['127', '408', '483', '114', '192', '603', '50', '480', '657', '519']
623 ['89', '174', '480', '515', '318', '178', '172', '272', '169', '427']
628 ['474', '86', '1', '246', '98', '193', '603', '1137', '423', '515']
625 ['98', '318', '12', '114', '513', '511', '178', '272', '89', '316']
629 ['318', '169', '313', '408', '483', '480', '657', '211', '114', '963']
633 ['169', '272', '408', '185', '603', '483', '480', '178', '316', '512']
632 ['178', '313', '496', '427', '187', '408', '603', '659', '511', '272']
631 ['64', '169', '603', '178', '408', '98', '480', '318', '474', '483']
634 ['98', '318', '357', '511', '603', '520', '133', '169', '64', '1039']
639 ['480', '408', '657', '169', '963', '316', '89', '603', '50', '530']
630 ['313', '57', '488', '513', '169', '481', '189', '512', '921', '633']
642 ['408', '316', '963', '513', '520', '515', '265', '923', '509', '9']
637 ['483', '313', '178', '641', '216', '114', '272', '513', '657', '487']
640 ['98', '603', '480', '272', '483', '408', '127', '923', '285', '511']
626 ['100', '474', '318', '50', '511', '127', '134', '185', '513', '89']
643 ['178', '191', '313', '134', '64', '135', '272', '661', '480', '475']
598 ['50', '169', '64', '174', '408', '318', '191', '513', '89', '285']
638 ['169', '318', '64', '178', '641', '513', '963', '28', '408', '313']
635 ['223', '50', '173', '603', '172', '1007', '483', '515', '474', '480']
644 ['408', '318', '513', '114', '657', '313', '515', '272', '480', '963']
636 ['603', '512', '480', '479', '483', '408', '178', '189', '64', '430']
645 ['408', '127', '192', '480', '923', '603', '316', '12', '647', '187']
648 ['313', '258', '1039', '493', '270', '651', '333', '237', '408', '655']
647 ['272', '169', '318', '603', '408', '64', '483', '114', '515', '316']
650 ['178', '12', '318', '513', '169', '114', '408', '64', '963', '1449']
651 ['50', '496', '178', '357', '318', '174', '64', '513', '527', '483']
654 ['966', '64', '923', '190', '707', '487', '528', '427', '493', '485']
653 ['483', '114', '169', '408', '513', '963', '241', '484', '651', '59']
655 ['169', '408', '488', '487', '641', '168', '151', '177', '430', '189']
649 ['408', '318', '357', '12', '114', '64', '169', '272', '98', '483']
658 ['114', '474', '483', '513', '657', '64', '134', '12', '251', '272']
656 ['64', '127', '187', '480', '656', '313', '923', '427', '180', '174']
660 ['169', '408', '513', '199', '480', '12', '246', '127', '528', '114']
659 ['169', '923', '59', '12', '408', '427', '641', '318', '114', '1063']
646 ['603', '657', '483', '515', '169', '408', '12', '963', '641', '513']
663 ['172', '511', '195', '166', '169', '657', '408', '963', '114', '482']
664 ['178', '515', '474', '488', '114', '199', '923', '511', '170', '661']
657 ['316', '127', '318', '12', '59', '64', '313', '963', '98', '196']
665 ['408', '169', '178', '64', '114', '963', '513', '318', '199', '657']
666 ['408', '512', '165', '641', '922', '921', '923', '170', '285', '528']
661 ['12', '22', '483', '98', '641', '187', '963', '136', '661', '251']
662 ['64', '169', '408', '190', '56', '318', '170', '132', '474', '114']
667 ['483', '127', '320', '512', '169', '923', '276', '408', '100', '134']
641 ['272', '408', '169', '56', '923', '12', '313', '474', '114', '512']
668 ['12', '169', '603', '127', '513', '285', '515', '89', '64', '611']
673 ['923', '178', '169', '408', '483', '64', '427', '480', '223', '963']
671 ['313', '133', '496', '64', '318', '272', '513', '178', '317', '205']
669 ['513', '185', '134', '488', '317', '316', '1449', '606', '178', '285']
676 ['513', '408', '478', '178', '479', '199', '89', '484', '694', '963']
674 ['22', '963', '513', '174', '923', '59', '187', '318', '408', '657']
652 ['169', '483', '12', '493', '64', '515', '661', '114', '603', '22']
677 ['480', '12', '318', '313', '64', '11', '169', '474', '89', '203']
682 ['515', '603', '408', '134', '199', '178', '114', '606', '641', '513']
679 ['114', '408', '513', '657', '12', '251', '198', '59', '923', '48']
684 ['511', '357', '272', '313', '316', '114', '512', '603', '427', '480']
685 ['963', '483', '169', '657', '408', '100', '59', '513', '318', '1194']
683 ['50', '496', '603', '483', '12', '134', '174', '98', '173', '408']
691 ['515', '127', '12', '272', '313', '169', '483', '520', '199', '89']
672 ['169', '408', '114', '474', '483', '178', '427', '603', '963', '64']
692 ['169', '190', '318', '64', '98', '178', '408', '483', '50', '12']
690 ['96', '318', '527', '493', '515', '22', '408', '169', '603', '963']
689 ['408', '169', '657', '511', '603', '963', '114', '83', '528', '513']
686 ['479', '408', '923', '511', '513', '657', '272', '1194', '488', '169']
693 ['408', '169', '114', '657', '190', '603', '316', '189', '513', '481']
688 ['242', '51', '346', '474', '265', '86', '257', '387', '392', '486']
697 ['408', '169', '285', '493', '98', '64', '483', '1449', '519', '116']
698 ['318', '169', '223', '813', '64', '320', '98', '272', '475', '79']
670 ['313', '408', '357', '64', '114', '513', '427', '169', '963', '28']
694 ['313', '64', '272', '488', '512', '12', '408', '963', '1449', '170']
680 ['56', '483', '12', '174', '114', '484', '178', '963', '606', '48']
705 ['963', '709', '966', '923', '166', '515', '519', '57', '490', '125']
701 ['483', '603', '480', '169', '427', '223', '408', '923', '515', '98']
699 ['603', '480', '427', '357', '12', '408', '474', '483', '178', '519']
704 ['515', '483', '169', '408', '64', '513', '199', '190', '114', '474']
707 ['357', '169', '114', '408', '709', '98', '520', '272', '213', '512']
700 ['483', '603', '172', '513', '496', '89', '64', '479', '12', '22']
687 ['657', '318', '480', '408', '169', '474', '513', '187', '127', '98']
695 ['127', '98', '318', '12', '408', '483', '187', '175', '657', '64']
675 ['170', '64', '12', '513', '408', '357', '187', '114', '50', '180']
708 ['174', '963', '173', '64', '22', '318', '204', '169', '483', '172']
709 ['313', '408', '272', '114', '963', '12', '357', '133', '169', '134']
711 ['515', '507', '474', '14', '513', '357', '1449', '603', '285', '659']
710 ['474', '408', '513', '657', '647', '427', '488', '923', '480', '661']
712 ['272', '22', '318', '313', '923', '114', '357', '659', '480', '427']
715 ['169', '408', '357', '483', '127', '191', '199', '178', '478', '114']
713 ['64', '408', '474', '657', '12', '318', '114', '513', '357', '169']
716 ['302', '313', '170', '408', '657', '513', '272', '1449', '963', '251']
681 ['318', '513', '64', '190', '483', '408', '169', '923', '45', '251']
678 ['408', '12', '174', '318', '169', '114', '64', '272', '963', '511']
719 ['316', '483', '143', '480', '59', '521', '496', '22', '174', '190']
702 ['173', '169', '50', '408', '603', '64', '178', '651', '199', '172']
721 ['483', '178', '169', '205', '210', '515', '257', '183', '408', '1194']
714 ['480', '174', '22', '515', '172', '318', '79', '513', '316', '408']
717 ['318', '59', '511', '657', '169', '651', '408', '197', '603', '191']
718 ['98', '408', '318', '64', '427', '22', '169', '528', '178', '603']
696 ['408', '318', '169', '603', '512', '192', '474', '64', '528', '480']
722 ['169', '603', '408', '657', '513', '59', '48', '357', '483', '64']
724 ['963', '174', '12', '114', '318', '64', '408', '479', '474', '520']
727 ['480', '513', '313', '64', '963', '166', '430', '478', '641', '59']
725 ['178', '169', '483', '48', '511', '923', '408', '170', '480', '1194']
706 ['408', '169', '515', '313', '172', '12', '480', '64', '483', '22']
720 ['408', '98', '12', '488', '169', '175', '64', '56', '127', '114']
729 ['603', '64', '22', '89', '169', '174', '178', '479', '50', '136']
726 ['483', '408', '173', '603', '114', '169', '174', '479', '480', '50']
728 ['318', '483', '59', '12', '64', '114', '178', '199', '316', '169']
703 ['174', '513', '408', '603', '178', '511', '272', '172', '64', '96']
738 ['12', '114', '483', '520', '513', '302', '657', '963', '251', '272']
736 ['98', '114', '318', '480', '272', '408', '474', '169', '315', '657']
734 ['178', '64', '427', '251', '513', '480', '79', '963', '520', '12']
730 ['318', '64', '483', '963', '127', '657', '199', '603', '510', '661']
743 ['127', '357', '318', '511', '64', '56', '169', '251', '50', '427']
742 ['318', '64', '192', '474', '483', '12', '515', '316', '484', '197']
737 ['408', '318', '190', '316', '603', '134', '60', '9', '178', '286']
733 ['178', '483', '50', '64', '169', '134', '603', '1194', '272', '357']
745 ['22', '318', '408', '474', '313', '484', '525', '496', '657', '199']
740 ['318', '98', '483', '603', '197', '480', '169', '515', '513', '114']
735 ['357', '64', '96', '313', '513', '187', '191', '178', '705', '89']
747 ['520', '191', '193', '512', '177', '114', '251', '657', '1039', '340']
723 ['408', '603', '313', '127', '483', '12', '199', '657', '498', '513']
739 ['174', '169', '181', '408', '520', '511', '114', '659', '189', '313']
749 ['318', '272', '513', '493', '265', '520', '525', '427', '479', '657']
748 ['12', '127', '190', '513', '87', '1142', '11', '178', '136', '246']
746 ['313', '318', '169', '520', '272', '357', '143', '178', '191', '12']
731 ['178', '513', '169', '479', '657', '519', '318', '408', '251', '199']
750 ['50', '12', '318', '272', '483', '408', '64', '520', '174', '963']
741 ['318', '483', '127', '64', '511', '12', '474', '191', '408', '169']
751 ['357', '169', '318', '657', '199', '520', '64', '604', '474', '251']
756 ['302', '483', '408', '169', '285', '114', '963', '64', '199', '657']
757 ['12', '178', '408', '318', '357', '480', '641', '187', '427', '114']
752 ['318', '657', '12', '178', '515', '143', '169', '1007', '513', '921']
758 ['408', '318', '89', '357', '169', '114', '511', '178', '515', '661']
732 ['657', '408', '318', '483', '511', '98', '480', '223', '114', '50']
762 ['127', '483', '357', '50', '205', '178', '64', '474', '603', '114']
744 ['408', '64', '474', '318', '169', '98', '513', '1194', '480', '114']
754 ['169', '408', '50', '64', '190', '963', '48', '318', '114', '172']
753 ['318', '59', '493', '603', '57', '496', '498', '143', '169', '519']
763 ['408', '64', '169', '114', '178', '657', '45', '479', '641', '513']
764 ['169', '657', '515', '59', '603', '408', '357', '480', '165', '190']
767 ['408', '64', '169', '923', '318', '114', '272', '357', '603', '48']
769 ['127', '89', '50', '192', '603', '488', '185', '114', '408', '513']
755 ['22', '408', '169', '313', '173', '114', '64', '318', '272', '923']
771 ['285', '512', '963', '513', '659', '64', '856', '45', '657', '519']
768 ['408', '64', '318', '114', '169', '963', '22', '174', '12', '186']
773 ['483', '48', '199', '647', '474', '513', '114', '514', '657', '178']
765 ['474', '603', '169', '64', '408', '515', '479', '484', '483', '191']
772 ['603', '427', '480', '483', '408', '127', '318', '64', '357', '169']
766 ['56', '513', '408', '488', '190', '603', '169', '511', '100', '1007']
774 ['169', '408', '513', '483', '657', '302', '480', '603', '285', '487']
760 ['513', '178', '603', '187', '519', '408', '483', '515', '127', '514']
761 ['172', '174', '210', '22', '89', '114', '333', '684', '64', '480']
777 ['963', '64', '272', '408', '169', '735', '22', '83', '82', '520']
759 ['178', '318', '64', '513', '480', '199', '196', '22', '174', '483']
776 ['12', '172', '316', '64', '197', '42', '180', '482', '313', '963']
780 ['923', '482', '651', '64', '483', '963', '484', '114', '178', '408']
779 ['408', '313', '22', '64', '483', '114', '178', '12', '318', '317']
778 ['169', '114', '182', '408', '127', '489', '192', '156', '89', '12']
782 ['64', '22', '169', '318', '174', '385', '357', '191', '172', '511']
786 ['64', '272', '483', '427', '963', '408', '12', '613', '251', '474']
784 ['318', '64', '50', '22', '174', '172', '483', '169', '12', '181']
770 ['474', '98', '603', '515', '512', '480', '272', '48', '483', '408']
788 ['515', '136', '57', '169', '190', '493', '242', '408', '484', '923']
789 ['511', '172', '64', '313', '357', '318', '483', '408', '187', '661']
790 ['12', '169', '923', '963', '483', '114', '180', '272', '198', '478']
787 ['169', '168', '318', '50', '513', '173', '474', '511', '174', '172']
783 ['318', '483', '657', '480', '513', '511', '64', '12', '272', '357']
785 ['657', '480', '483', '408', '1194', '357', '172', '169', '179', '98']
794 ['98', '169', '12', '483', '223', '408', '64', '56', '190', '178']
781 ['285', '427', '12', '611', '493', '511', '408', '166', '114', '641']
796 ['166', '651', '963', '657', '923', '656', '207', '1642', '513', '242']
795 ['114', '285', '23', '408', '127', '483', '190', '192', '530', '223']
793 ['408', '169', '318', '64', '474', '483', '302', '190', '223', '114']
798 ['513', '408', '519', '127', '64', '963', '22', '479', '178', '657']
791 ['483', '114', '169', '316', '408', '511', '199', '223', '172', '174']
802 ['313', '50', '22', '172', '515', '272', '174', '181', '64', '318']
800 ['318', '64', '98', '169', '191', '190', '493', '483', '603', '134']
804 ['169', '178', '313', '963', '223', '923', '408', '190', '613', '272']
803 ['100', '483', '357', '318', '98', '12', '302', '285', '127', '190']
775 ['64', '357', '127', '318', '408', '483', '657', '169', '98', '427']
792 ['272', '515', '98', '64', '513', '318', '408', '603', '87', '480']
799 ['12', '100', '169', '98', '511', '408', '603', '114', '64', '357']
805 ['483', '302', '1449', '657', '114', '512', '647', '1194', '242', '474']
806 ['603', '285', '479', '134', '657', '173', '64', '513', '251', '165']
807 ['923', '963', '513', '64', '57', '611', '166', '306', '482', '603']
797 ['174', '313', '408', '496', '22', '427', '483', '172', '12', '318']
801 ['603', '480', '174', '272', '172', '408', '318', '64', '169', '12']
809 ['515', '408', '169', '12', '513', '923', '114', '134', '480', '178']
815 ['64', '170', '223', '8', '480', '497', '963', '272', '12', '408']
817 ['12', '483', '169', '408', '427', '521', '606', '657', '646', '489']
821 ['408', '199', '12', '285', '657', '50', '316', '515', '83', '923']
818 ['12', '483', '479', '169', '408', '484', '272', '641', '173', '474']
814 ['64', '50', '313', '174', '96', '22', '318', '178', '408', '963']
812 ['313', '64', '483', '170', '168', '318', '183', '479', '172', '50']
823 ['357', '603', '483', '179', '169', '9', '132', '272', '496', '185']
825 ['313', '318', '69', '22', '169', '963', '328', '300', '408', '64']
827 ['318', '64', '169', '408', '483', '1194', '98', '657', '603', '251']
829 ['127', '483', '12', '64', '654', '357', '56', '187', '191', '180']
811 ['603', '480', '174', '79', '172', '483', '408', '178', '127', '318']
830 ['318', '169', '496', '483', '603', '604', '199', '659', '64', '178']
826 ['64', '169', '963', '408', '483', '513', '641', '166', '480', '28']
831 ['127', '169', '408', '172', '357', '318', '513', '483', '180', '515']
819 ['480', '483', '408', '127', '318', '56', '64', '169', '114', '474']
828 ['603', '48', '98', '169', '474', '64', '189', '318', '511', '285']
808 ['174', '172', '408', '127', '64', '169', '12', '50', '963', '316']
835 ['657', '603', '479', '483', '493', '64', '170', '59', '169', '408']
833 ['86', '603', '482', '169', '318', '1449', '171', '131', '693', '661']
836 ['127', '64', '408', '483', '484', '50', '480', '199', '98', '169']
816 ['515', '480', '174', '479', '483', '318', '89', '199', '187', '513']
838 ['64', '483', '603', '178', '488', '493', '963', '269', '251', '923']
839 ['64', '659', '408', '318', '603', '14', '169', '496', '474', '519']
840 ['318', '408', '178', '124', '641', '251', '192', '258', '488', '1021']
832 ['603', '12', '657', '272', '64', '318', '963', '169', '408', '357']
810 ['474', '144', '98', '194', '603', '1137', '100', '515', '427', '512']
844 ['169', '64', '408', '114', '57', '357', '528', '479', '483', '657']
843 ['11', '272', '156', '315', '513', '707', '483', '12', '475', '178']
834 ['357', '64', '114', '318', '483', '172', '12', '174', '170', '169']
846 ['169', '408', '313', '750', '251', '963', '223', '114', '189', '646']
837 ['408', '64', '474', '315', '272', '134', '480', '641', '83', '513']
813 ['169', '318', '408', '64', '483', '657', '114', '178', '474', '190']
842 ['318', '127', '357', '603', '408', '56', '498', '657', '197', '1449']
847 ['515', '408', '316', '169', '483', '469', '59', '83', '64', '189']
848 ['272', '408', '64', '169', '513', '114', '313', '60', '515', '22']
822 ['483', '318', '657', '50', '114', '474', '199', '498', '178', '98']
852 ['318', '12', '173', '513', '64', '134', '272', '114', '923', '603']
851 ['96', '89', '195', '520', '488', '178', '205', '513', '197', '69']
849 ['242', '302', '474', '144', '98', '194', '603', '181', '515', '165']
854 ['408', '114', '474', '169', '251', '1142', '172', '320', '480', '654']
850 ['272', '64', '923', '313', '498', '178', '651', '258', '169', '483']
858 ['178', '64', '408', '272', '56', '169', '316', '488', '513', '480']
853 ['313', '513', '57', '134', '408', '923', '114', '170', '96', '169']
855 ['318', '22', '357', '180', '64', '316', '511', '408', '12', '173']
824 ['483', '427', '603', '114', '318', '169', '178', '12', '513', '64']
845 ['357', '64', '169', '14', '98', '114', '318', '603', '483', '408']
841 ['318', '64', '474', '408', '169', '963', '657', '483', '427', '12']
859 ['174', '64', '169', '50', '513', '173', '172', '318', '408', '12']
862 ['408', '318', '611', '190', '513', '316', '320', '30', '607', '114']
856 ['100', '178', '174', '474', '483', '408', '172', '114', '64', '12']
820 ['64', '12', '285', '127', '515', '603', '511', '98', '272', '483']
863 ['60', '513', '408', '223', '48', '11', '190', '12', '921', '189']
860 ['187', '483', '127', '318', '50', '12', '480', '357', '64', '511']
857 ['313', '272', '169', '483', '496', '64', '408', '603', '134', '223']
864 ['285', '187', '313', '320', '272', '170', '887', '480', '512', '302']
865 ['483', '12', '64', '514', '114', '652', '173', '48', '172', '480']
868 ['483', '134', '137', '171', '175', '48', '190', '514', '428', '60']
867 ['408', '169', '513', '114', '302', '127', '285', '1007', '357', '190']
861 ['923', '480', '483', '169', '223', '318', '427', '963', '528', '515']
870 ['408', '114', '173', '251', '515', '187', '205', '320', '285', '512']
871 ['169', '408', '963', '64', '318', '657', '496', '1007', '320', '144']
875 ['408', '199', '114', '316', '483', '178', '127', '285', '318', '513']
876 ['408', '169', '357', '474', '963', '190', '64', '12', '56', '98']
872 ['318', '190', '169', '174', '50', '963', '64', '513', '98', '408']
866 ['483', '515', '64', '357', '657', '408', '50', '603', '174', '318']
877 ['187', '127', '174', '64', '318', '172', '511', '963', '654', '100']
873 ['174', '318', '496', '50', '513', '195', '484', '12', '657', '64']
880 ['408', '169', '9', '313', '474', '963', '134', '480', '178', '340']
878 ['313', '408', '480', '520', '272', '169', '963', '134', '133', '513']
869 ['169', '474', '408', '285', '114', '483', '190', '172', '170', '603']
881 ['318', '923', '516', '656', '12', '272', '258', '603', '190', '607']
879 ['98', '64', '357', '169', '408', '318', '272', '12', '515', '285']
883 ['179', '654', '1007', '23', '474', '32', '114', '483', '200', '268']
882 ['318', '64', '178', '480', '408', '603', '483', '657', '320', '513']
884 ['483', '318', '168', '474', '496', '64', '178', '132', '657', '513']
886 ['169', '408', '199', '511', '478', '1062', '210', '705', '114', '251']
885 ['64', '22', '190', '313', '114', '923', '198', '408', '963', '480']
889 ['316', '176', '490', '136', '963', '272', '251', '478', '661', '205']
874 ['318', '56', '64', '657', '12', '474', '408', '192', '483', '169']
892 ['199', '474', '408', '512', '513', '223', '520', '923', '185', '603']
890 ['56', '178', '475', '100', '318', '408', '180', '191', '12', '169']
893 ['357', '408', '114', '513', '318', '169', '480', '127', '657', '427']
887 ['165', '275', '174', '483', '507', '189', '169', '60', '306', '707']
891 ['603', '515', '272', '64', '22', '923', '408', '316', '318', '79']
894 ['192', '187', '474', '483', '408', '180', '178', '169', '185', '654']
896 ['194', '169', '357', '242', '521', '498', '178', '408', '523', '165']
897 ['318', '192', '242', '357', '64', '251', '480', '100', '519', '427']
901 ['318', '313', '496', '143', '657', '483', '64', '178', '79', '603']
899 ['22', '191', '313', '12', '316', '251', '963', '513', '143', '165']
903 ['923', '169', '603', '408', '178', '174', '498', '661', '474', '480']
904 ['483', '199', '647', '515', '98', '318', '1449', '64', '641', '484']
907 ['242', '302', '265', '451', '603', '1137', '515', '480', '498', '208']
905 ['408', '483', '50', '513', '127', '603', '657', '515', '474', '114']
902 ['408', '173', '132', '64', '169', '357', '190', '496', '114', '98']
898 ['50', '174', '173', '483', '169', '64', '318', '172', '12', '178']
895 ['318', '127', '474', '483', '64', '357', '302', '98', '285', '408']
906 ['603', '483', '318', '357', '427', '169', '89', '178', '64', '657']
900 ['114', '408', '1007', '513', '134', '172', '515', '514', '12', '1194']
908 ['272', '64', '169', '178', '285', '480', '313', '520', '659', '705']
916 ['127', '285', '124', '513', '272', '408', '963', '114', '187', '116']
911 ['127', '169', '64', '318', '408', '515', '286', '527', '251', '493']
912 ['408', '603', '22', '480', '165', '923', '657', '8', '313', '651']
914 ['64', '100', '357', '483', '173', '475', '251', '127', '50', '318']
918 ['513', '114', '408', '357', '515', '483', '474', '528', '657', '169']
919 ['474', '603', '134', '493', '512', '190', '357', '496', '185', '427']
921 ['22', '64', '318', '208', '1063', '12', '451', '216', '513', '258']
910 ['408', '64', '318', '169', '114', '132', '191', '480', '208', '8']
913 ['496', '251', '124', '242', '313', '513', '199', '165', '194', '1039']
915 ['483', '480', '132', '134', '657', '318', '654', '169', '408', '427']
922 ['483', '169', '479', '603', '923', '134', '480', '511', '12', '165']
923 ['98', '194', '603', '427', '512', '480', '132', '479', '272', '483']
928 ['474', '603', '515', '427', '512', '480', '275', '498', '132', '174']
927 ['333', '57', '479', '1194', '136', '656', '603', '89', '357', '272']
924 ['136', '199', '22', '483', '513', '657', '479', '48', '651', '603']
929 ['408', '64', '657', '603', '513', '173', '272', '114', '285', '357']
931 ['408', '480', '427', '185', '603', '169', '483', '474', '170', '488']
917 ['12', '64', '275', '408', '589', '23', '59', '603', '357', '185']
932 ['318', '50', '408', '132', '127', '604', '187', '12', '268', '272']
909 ['480', '132', '483', '408', '318', '64', '169', '50', '114', '657']
934 ['318', '923', '166', '480', '8', '408', '98', '306', '357', '114']
933 ['191', '192', '520', '496', '663', '134', '185', '272', '525', '657']
935 ['657', '513', '114', '480', '64', '408', '641', '272', '515', '59']
938 ['357', '483', '963', '12', '318', '511', '657', '174', '615', '172']
940 ['408', '603', '479', '528', '512', '114', '169', '923', '483', '22']
888 ['408', '318', '169', '483', '189', '64', '127', '187', '512', '272']
925 ['64', '12', '318', '515', '22', '603', '520', '169', '408', '178']
942 ['408', '64', '169', '114', '483', '178', '190', '98', '251', '223']
937 ['169', '174', '12', '483', '181', '302', '963', '127', '923', '172']
926 ['12', '408', '483', '515', '50', '45', '134', '169', '603', '64']
943 ['272', '223', '169', '114', '180', '175', '1142', '209', '480', '357']
939 ['242', '144', '246', '98', '603', '100', '196', '526', '427', '165']
936 ['483', '302', '603', '169', '513', '191', '474', '98', '185', '208']
930 ['318', '272', '513', '603', '528', '496', '313', '98', '83', '660']
920 ['316', '275', '64', '923', '483', '427', '169', '408', '516', '83']
941 ['64', '285', '483', '12', '127', '169', '190', '657', '515', '663']

추천성능 평가

  1. RMSE (Root Mean Squared Error) : 평균 제곱근 오차 $$ \text{RMSE} = \sqrt{\frac{1}{|\hat{R}|} \sum_{\hat{r}_{ui} \in \hat{R}}(r_{ui} - \hat{r}_{ui})^2} $$
  2. MAE (Mean Absolute Error) : 평균 절대 오차 $$ \text{MAE} = \frac{1}{|\hat{R}|} \sum_{\hat{r}_{ui} \in \hat{R}}|r_{ui} - \hat{r}_{ui}| $$

알고리즘 평가

풀스택을 위한 서버 기술과 도커 기본

혹시 본 자료가 도움이 되셨다면, 이런 식으로 잘 정리된 코드/자료로 만든 제 강의 한번 봐주실래요?

풀스택을 위한 서버 기술과 도커 기본

도커, 리눅스, AWS, nginx, HTTPS, flask 배포 기본

자세히 알아보기 모든 강좌 보기
In [17]:
import surprise
from surprise import Dataset
data = Dataset.load_builtin('ml-100k')
data.split(n_folds=3)
In [23]:
sim_options = {'name': 'msd'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

Evaluating RMSE, MAE of algorithm KNNBasic.

------------
Fold 1
Computing the msd similarity matrix...
Done computing similarity matrix.
RMSE: 0.9911
MAE:  0.7833
------------
Fold 2
Computing the msd similarity matrix...
Done computing similarity matrix.
RMSE: 0.9833
MAE:  0.7771
------------
Fold 3
Computing the msd similarity matrix...
Done computing similarity matrix.
RMSE: 0.9894
MAE:  0.7819
------------
------------
Mean RMSE: 0.9879
Mean MAE : 0.7808
------------
------------
Out[23]:
CaseInsensitiveDefaultDict(list,
                           {'mae': [0.78332085007914287,
                             0.7771041490260826,
                             0.78185859406389302],
                            'rmse': [0.99109548719657858,
                             0.98332811659672703,
                             0.9893776110540401]})
In [24]:
sim_options = {'name': 'cosine'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

Evaluating RMSE, MAE of algorithm KNNBasic.

------------
Fold 1
Computing the cosine similarity matrix...
Done computing similarity matrix.
RMSE: 1.0244
MAE:  0.8108
------------
Fold 2
Computing the cosine similarity matrix...
Done computing similarity matrix.
RMSE: 1.0187
MAE:  0.8061
------------
Fold 3
Computing the cosine similarity matrix...
Done computing similarity matrix.
RMSE: 1.0226
MAE:  0.8094
------------
------------
Mean RMSE: 1.0219
Mean MAE : 0.8088
------------
------------
Out[24]:
CaseInsensitiveDefaultDict(list,
                           {'mae': [0.81079510431791924,
                             0.80605631128339117,
                             0.80941768920884594],
                            'rmse': [1.0243634073960175,
                             1.0187482414191331,
                             1.0225720777877443]})
In [25]:
sim_options = {'name': 'pearson'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

Evaluating RMSE, MAE of algorithm KNNBasic.

------------
Fold 1
Computing the pearson similarity matrix...
Done computing similarity matrix.
RMSE: 1.0239
MAE:  0.8120
------------
Fold 2
Computing the pearson similarity matrix...
Done computing similarity matrix.
RMSE: 1.0167
MAE:  0.8083
------------
Fold 3
Computing the pearson similarity matrix...
Done computing similarity matrix.
RMSE: 1.0214
MAE:  0.8099
------------
------------
Mean RMSE: 1.0207
Mean MAE : 0.8101
------------
------------
Out[25]:
CaseInsensitiveDefaultDict(list,
                           {'mae': [0.81200632767328063,
                             0.80826931547507119,
                             0.80989597847789219],
                            'rmse': [1.0239246403571562,
                             1.0167128227144424,
                             1.0213664721488831]})
In [58]:
sim_options = {'name': 'pearson_baseline'}
algo = surprise.KNNBasic(sim_options=sim_options)
surprise.evaluate(algo, data)

Evaluating RMSE, MAE of algorithm KNNBasic.

------------
Fold 1
Estimating biases using als...
Computing the pearson_baseline similarity matrix...
Done computing similarity matrix.
RMSE: 1.0049
MAE:  0.7968
------------
Fold 2
Estimating biases using als...
Computing the pearson_baseline similarity matrix...
Done computing similarity matrix.
RMSE: 1.0103
MAE:  0.7985
------------
Fold 3
Estimating biases using als...
Computing the pearson_baseline similarity matrix...
Done computing similarity matrix.
RMSE: 1.0110
MAE:  0.8002
------------
------------
Mean RMSE: 1.0087
Mean MAE : 0.7985
------------
------------
Out[58]:
CaseInsensitiveDefaultDict(list,
                           {'mae': [0.79675695245350209,
                             0.7984790061210334,
                             0.8002162404915526],
                            'rmse': [1.0048624637086425,
                             1.0102925000197331,
                             1.0110090350939811]})
In [26]:
algo = surprise.SVD()
surprise.evaluate(algo, data)

Evaluating RMSE, MAE of algorithm SVD.

------------
Fold 1
RMSE: 0.9498
MAE:  0.7498
------------
Fold 2
RMSE: 0.9458
MAE:  0.7463
------------
Fold 3
RMSE: 0.9446
MAE:  0.7453
------------
------------
Mean RMSE: 0.9467
Mean MAE : 0.7471
------------
------------
Out[26]:
CaseInsensitiveDefaultDict(list,
                           {'mae': [0.749794336106132,
                             0.7462662006765739,
                             0.74533450362095799],
                            'rmse': [0.94976543487583887,
                             0.94580670524240762,
                             0.94459061787140985]})

SVD (SVD와 Latent Factor 모형) 정확도(Precision), 재현율(Recall)과 참고사항