반응형
Naver API
- 네이버 개발자 센터
- https://developers.naver.com/main/
- Application
- 어플리케이션 등록
- 어플리케이션 이름 ds_study
- 사용 API
- 검색
- 데이터랩(검색어트렌드)
- 데이터랩(쇼핑인사이트)
- 환경추가
- WEB 설정
- http://localhost
- Clienct ID: H2_6lcavpVyHW8211rUq
- Clienct Secret: D1XBjnKTe9
- https://developers.naver.com/apps/#/myapps/H2_6lcavpVyHW8211rUq/overview
2. 네이버 검색 API 사용하기
In [ ]:
- urllib: http 프로토콜에 따라서 서버의 요청/응답을 처리하기 위한 모듈
- urllib.request: 클라이언트의 요청을 처리하는 모듈
- urllib.parse: url 주소에 대한 분석
검색: 블로그(blog)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/blog?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
response, response.getcode(), response.code, response.status
Out[4]:
(<http.client.HTTPResponse at 0x160a37430>, 200, 200, 200)
In [10]:
# 글자로 읽을 경우, decode utf-8 설정
print(response_body.decode("utf-8"))
검색: 책(book)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/book?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
검색: 영화(movie)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/movie?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
검색: 카페(cafearticle)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/cafearticle?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
검색: 쇼핑(shop)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/shop?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
검색: 백과사전(encyc)
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("파이썬")
url = "https://openapi.naver.com/v1/search/encyc?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
3. 상품 검색
"몰스킨
# 네이버 검색 API예제는 블로그를 비롯 전문자료까지 호출방법이 동일하므로 blog검색만 대표로 예제를 올렸습니다.
# 네이버 검색 Open API 예제 - 블로그 검색
import os
import sys
import urllib.request
client_id =
client_secret =
encText = urllib.parse.quote("몰스킨")
url = "https://openapi.naver.com/v1/search/shop?query=" + encText # json 결과
# url = "https://openapi.naver.com/v1/search/blog.xml?query=" + encText # xml 결과
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id",client_id)
request.add_header("X-Naver-Client-Secret",client_secret)
response = urllib.request.urlopen(request)
rescode = response.getcode()
if(rescode==200):
response_body = response.read()
print(response_body.decode('utf-8'))
else:
print("Error Code:" + rescode)
(1) gen_search_url()
encText = urllib.parse.quote("몰스킨")
url = "https://openapi.naver.com/v1/search/shop?query=" + encText # json 결과
def gen_search_url(api_node, search_text, start_num, disp_num):
base = "https://openapi.naver.com/v1/search"
node = "/" + api_node + ".json"
param_query = "?query=" + urllib.parse.quote(search_text)
param_start = "&start=" + str(start_num)
param_disp = "&display=" + str(disp_num)
return base + node + param_query + param_start + param_disp
In [136]:
gen_search_url("shop", "TEST", 10, 3)
(2) get_result_onepage()
In [114]:
import json
import datetime
def get_result_onpage(url):
request = urllib.request.Request(url)
request.add_header("X-Naver-Client-Id", client_id)
request.add_header("X-Naver-Client-Secret", client_secret)
response = urllib.request.urlopen(request)
print("[%s] Url Request Success" % datetime.datetime.now())
return json.loads(response.read().decode("utf-8"))
In [115]:
datetime.datetime.now()
url = gen_search_url("shop", "몰스킨", 1, 5)
one_result = get_result_onpage(url)
one_result
one_result["items"][0]["title"]
Out[118]:
'한정판 <b>몰스킨</b> 2022 해리포터 데일리 다이어리 포캣'
In [119]:
one_result["items"][0]["link"]
Out[119]:
'https://search.shopping.naver.com/gate.nhn?id=28719894754'
In [120]:
one_result["items"][0]["lprice"]
Out[120]:
'31700'
In [121]:
one_result["items"][0]["mallName"]
Out[121]:
'네이버'
one_result["items"][0]
Out[141]:
{'title': '한정판 몰스킨 2022 해리포터 데일리 다이어리 포캣',
'link': 'https://search.shopping.naver.com/gate.nhn?id=28719894754',
'image': 'https://shopping-phinf.pstatic.net/main_2871989/28719894754.20210906035419.jpg',
'lprice': '31700',
'hprice': '',
'mallName': '네이버',
'productId': '28719894754',
'productType': '1',
'brand': '몰스킨',
'maker': '몰스킨',
'category1': '생활/건강',
'category2': '문구/사무용품',
'category3': '다이어리/플래너',
'category4': '다이어리'}
In [142]:
import pandas as pd
def get_fields(json_data):
title = [each["title"] for each in json_data["items"]]
link = [each["link"] for each in json_data["items"]]
lprice = [each["lprice"] for each in json_data["items"]]
mall_name = [each["mallName"] for each in json_data["items"]]
result_pd = pd.DataFrame({
"title": title,
"link": link,
"lprice": lprice,
"mall": mall_name,
}, columns=["title", "lprice", "link", "mall"])
return result_pd
In [143]:
get_fields(one_result)
(4) delete_tag()
In [144]:
def delete_tag(input_str):
input_str = input_str.replace("<b>", "")
input_str = input_str.replace("</b>", "")
return input_str
In [145]:
import pandas as pd
def get_fields(json_data):
title = [delete_tag(each["title"]) for each in json_data["items"]]
link = [each["link"] for each in json_data["items"]]
lprice = [each["lprice"] for each in json_data["items"]]
mall_name = [each["mallName"] for each in json_data["items"]]
result_pd = pd.DataFrame({
"title": title,
"link": link,
"lprice": lprice,
"mall": mall_name,
}, columns=["title", "lprice", "link", "mall"])
return result_pd
In [146]:
get_fields(one_result)
url = gen_search_url("shop", "몰스킨", 1, 5)
json_result = get_result_onpage(url)
pd_result = get_fields(json_result)
In [148]:
pd_result
(5) actMain()
In [149]:
for n in range(1, 1000, 100):
print(n)
1
101
201
301
401
501
601
701
801
901
In [155]:
result_mol = []
for n in range(1, 1000, 100):
url = gen_search_url("shop", "몰스킨", n, 100)
json_result = get_result_onpage(url)
pd_result = get_fields(json_result)
result_mol.append(pd_result)
result_mol = pd.concat(result_mol)
[2021-09-30 05:09:47.325324] Url Request Success
[2021-09-30 05:09:47.609312] Url Request Success
[2021-09-30 05:09:47.913627] Url Request Success
[2021-09-30 05:09:48.227613] Url Request Success
[2021-09-30 05:09:48.533253] Url Request Success
[2021-09-30 05:09:48.816997] Url Request Success
[2021-09-30 05:09:49.125801] Url Request Success
[2021-09-30 05:09:49.397631] Url Request Success
[2021-09-30 05:09:49.715529] Url Request Success
[2021-09-30 05:09:49.987900] Url Request Success
In [156]:
result_mol.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 0 to 99
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 title 1000 non-null object
1 lprice 1000 non-null object
2 link 1000 non-null object
3 mall 1000 non-null object
dtypes: object(4)
memory usage: 39.1+ KB
In [159]:
result_mol.reset_index(drop=True, inplace=True)
result_mol.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 title 1000 non-null object
1 lprice 1000 non-null object
2 link 1000 non-null object
3 mall 1000 non-null object
dtypes: object(4)
memory usage: 31.4+ KB
result_mol.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 title 1000 non-null object
1 lprice 1000 non-null object
2 link 1000 non-null object
3 mall 1000 non-null object
dtypes: object(4)
memory usage: 31.4+ KB
In [170]:
result_mol["lprice"] = result_mol["lprice"].astype("float")
result_mol.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 title 1000 non-null object
1 lprice 1000 non-null float64
2 link 1000 non-null object
3 mall 1000 non-null object
dtypes: float64(1), object(3)
memory usage: 31.4+ KB
시각화
import set_matplotlib_hangul
In [179]:
plt.figure(figsize=(15, 6))
sns.countplot(
result_mol["mall"],
data=result_mol,
palette="RdYlGn",
order=result_mol["mall"].value_counts().index
)
plt.xticks(rotation=90)
plt.show()
'파이썬 이것저것 > 크롤링' 카테고리의 다른 글
[파이썬] 서울시 주유소 크롤링 (0) | 2022.06.05 |
---|---|
[파이썬] 네이버 영화 평점 데이터 분석해보기 (0) | 2022.06.04 |
[파이썬] 스타벅스, 이디야 매장 크롤링 및 분석 (0) | 2022.04.24 |
[크롤링] GS25 매장 크롤링 -1 (0) | 2022.04.17 |