View source for Scrapping: Save text setiap pencarian

Simpan konten '''per URL ke file `.txt`''', 1 file per halaman  
Jadi tiap hasil pencarian disimpan jadi file `.txt` sendiri biar bisa dibaca atau dianalisis dengan mudah.

==FITUR-FITUR:==
* Baca `keywords.txt` → cari di Google
* Ambil `top-N` hasil pencarian tiap keyword
* Kunjungi tiap URL, ambil '''judul + isi artikel (5 paragraf pertama)'''
* Simpan ke folder `outputs/` dalam format `.txt`:
** Nama file: `keyword_rank_judul.txt`

==PERSIAPAN:==

 pip install googlesearch-python requests beautifulsoup4

Buat file `keywords.txt`:

 berita teknologi Indonesia
 politik 2025
 game PS5 terbaru

==SCRIPT PYTHON SUPER LENGKAP:==

 import os
 import requests
 from googlesearch import search
 from bs4 import BeautifulSoup
 import re
 import time
 
 def load_keywords(filename):
     with open(filename, 'r', encoding='utf-8') as f:
         return [line.strip() for line in f if line.strip()]
 
 def clean_filename(text):
     # Hilangkan karakter ilegal untuk nama file
     return re.sub(r'[\\/*?:"<>|]', '', text).strip().replace(' ', '_')[:50]
 
 def get_page_content(url):
     try:
         headers = {
             "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
         }
         response = requests.get(url, headers=headers, timeout=10)
         soup = BeautifulSoup(response.content, 'html.parser')
         title = soup.title.string if soup.title else 'No Title'
         paragraphs = soup.find_all('p')
         content = '\n\n'.join([p.get_text() for p in paragraphs[:5]])
         return title.strip(), content.strip()
     except Exception as e:
         return 'Error', f"Gagal mengambil konten: {e}"
 
 def save_to_txt(keyword, rank, title, url, content, folder='outputs'):
     os.makedirs(folder, exist_ok=True)
     filename = f"{clean_filename(keyword)}_{rank}_{clean_filename(title)}.txt"
     filepath = os.path.join(folder, filename)
 
     with open(filepath, 'w', encoding='utf-8') as f:
         f.write(f"Keyword   : {keyword}\n")
         f.write(f"Peringkat : {rank}\n")
         f.write(f"Judul     : {title}\n")
         f.write(f"URL       : {url}\n\n")
         f.write(content)
 
 def scrape_and_save_txt(keywords, num_results=5):
     for keyword in keywords:
         print(f"\n🔍 Searching: {keyword}")
         try:
             results = search(keyword, num_results=num_results)
             for i, url in enumerate(results):
                 print(f"  → ({i+1}) Fetching: {url}")
                 title, content = get_page_content(url)
                 save_to_txt(keyword, i+1, title, url, content)
                 time.sleep(2)
         except Exception as e:
             print(f"❌ Error saat mencari '{keyword}': {e}")
 
     print("\n✅ Semua konten telah disimpan di folder 'outputs/'")
 
 # Main
 if __name__ == '__main__':
     keywords = load_keywords('keywords.txt')
     scrape_and_save_txt(keywords, num_results=5)

==Output:==
Folder `outputs/` akan berisi file seperti:

 berita_teknologi_Indonesia_1_Tekno_Terbaru_dari_Tempo.txt
 politik_2025_2_Media_Indonesia_Politik.txt

Isi filenya:

 Keyword   : berita teknologi Indonesia
 Peringkat : 1
 Judul     : Berita Teknologi Terbaru Hari Ini - Tempo
 URL       : https://tekno.tempo.co/...
 
 [ISI PARAGRAF]


==Pranala Menarik==

* [[Scrapping]]