Post

100 Days Of Python - Day 34

Posted
By Ndimofor Aretas 6 min read

Day 37

Automating Chrome Browser with Selenium

  • Selenium is a portable framework for testing web applications.
  • It is open-source software released under the Apache License 2.0.
  • With selenium, we can automate the browser to do the repetitive tasks such as filling forms, clicking buttons, etc.

Installation & Setup

  • Install selenium using pip
1

pip install selenium
  • Install Chrome Browser

Working with Selenium

  • Import the selenium webdriver
1

from selenium import webdriver
  • Create a webdriver object
1

driver = webdriver.Chrome()
  • Open a website
1

driver.get("https://www.google.com")
  • To keep the chrome browser open, we need to use the chrome options in the webdriver
1
2
3
4
5
6
7
8
9
10
11

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)

# Open a website like amazon.de

driver.get("https://www.amazon.de")
  • To close a browser tab we can use the following code
1

driver.close()
  • To close the entire browser we can use the following code
1

driver.quit()
  • With selenium, we can find elements on a website by first importing the By module as followS:
1

from selenium.webdriver.common.by import By

  • Then we can use the following methods to find elements on a website:

    • id: driver.find_element(By.ID, "id")
    • name: driver.find_element(By.NAME, "name")
    • class name: driver.find_element(By.CLASS_NAME, "class_name")
    • xpath: driver.find_element(By.XPATH, "xpath")

  • After selecting an element, we can get the text of the element using the text attribute

1
2

element = driver.find_element(By.ID, "id")
print(element.text)

Printing the events from python.org

  • In the following example, we will use selenium to print the events from python.org
  • We will first of all select the element that contains the events and then save the events in a dictionary
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()

driver.get("https://www.python.org")

events_times = driver.find_elements(By.CSS_SELECTOR, ".event-widget time")
events_names = driver.find_elements(By.CSS_SELECTOR, ".event-widget li a")

events = {}

for n in range(len(events_times)):
    events[n] = {
        "time": events_times[n].text,
        "name": events_names[n].text,
    }

print(events)

driver.quit()

# Output as of 05.10.2023

{0: {'time': '2023-10-06', 'name': 'Django Day Copenhagen 2023'}, 1: {'time': '2023-10-06', 'name': 'PyCon ES Canarias 2023'}, 2: {'time': '2023-10-07', 'name': 'DjangoCongress JP 2023'}, 3: {'time': '2023-10-08', 'name': 'TOUFU - Parent-Child Python Programming Workshop'}, 4: {'time': '2023-10-09', 'name': 'PyHEP 2023'}}

Clicking a button

  • To click a button we can use the click() method
1
2

button = driver.find_element(By.ID, "id")
button.click()

Wikipedia main page

In the following example, we will use selenium to click on the article count button on the wikipedia main page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()
# get the wikipedia main page
driver.get("https://en.wikipedia.org/wiki/Main_Page")
# get the article count using the css class selector
article_count = driver.find_element(By.CSS_SELECTOR, "#articlecount a")
# click on the article count link
article_count.click()
  • In the following example, we will use selenium to:
    • find an element using the link text
    • then click on the link
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from selenium import webdriver
from selenium.webdriver.common.by import By

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()
# get the wikipedia main page
driver.get("https://en.wikipedia.org/wiki/Main_Page")
# get the english link using the link text
english_link = driver.find_element(By.LINK_TEXT, "English")
# click on the all portals link
english_link.click()
  • In the following example, we will use selenium to:
  • find the search box using the name attribute
  • then type in the search box
  • then press the enter key
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()
# get the wikipedia main page
driver.get("https://en.wikipedia.org/wiki/Main_Page")
# get the search box using the name attribute
search_box = driver.find_element(By.NAME, "search")
# type in the search box
search_box.send_keys("Python")
# press the enter key
search_box.send_keys(Keys.ENTER)
  • In the following example, we will use selenium to:
    • fill in a form with our first name, last name and email address
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()

driver.get("http://secure-retreat-92358.herokuapp.com/")
first_name = driver.find_element(By.NAME, "fName")
first_name.send_keys("John")
last_name = driver.find_element(By.NAME, "lName")
last_name.send_keys("Doe")
email = driver.find_element(By.NAME, "email")
email.send_keys("johndoe@email.com")
submit = driver.find_element(By.CSS_SELECTOR, "form button")
submit.click()
  • In the following example, we will use selenium to:
  • open the cookie clicker game
  • then click on the cookie
  • then click on the upgrades
  • the game will run for 5 minutes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time

chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("detach", True)

driver = webdriver.Chrome(options=chrome_options)
driver.maximize_window()

driver.get("https://orteil.dashnet.org/experiments/cookie/")
driver.implicitly_wait(5)

cookie = driver.find_element(By.ID, "cookie")

# get upgrade item ids
items = driver.find_elements(By.CSS_SELECTOR, "#store div")
item_ids = [item.get_attribute("id") for item in items]

timeout = time.time() + 5
five_min = time.time() + 60 * 5

while True:
    cookie.click()

    # every 5 seconds
    if time.time() > timeout:
        # get all upgrade <b> tags
        all_prices = driver.find_elements(By.CSS_SELECTOR, "#store b")
        item_prices = []

        # convert <b> text into an integer price
        for price in all_prices:
            element_text = price.text
            if element_text != "":
                cost = int(element_text.split("-")[1].strip().replace(",", ""))
                item_prices.append(cost)

        # create dictionary of store items and prices
        cookie_upgrades = {}
        for n in range(len(item_prices)):
            cookie_upgrades[item_prices[n]] = item_ids[n]

        # get current cookie count
        money_element = driver.find_element(By.ID, "money").text
        if "," in money_element:
            money_element = money_element.replace(",", "")
        cookie_count = int(money_element)

        # find upgrades that we can currently afford
        affordable_upgrades = {}
        for cost, id in cookie_upgrades.items():
            if cookie_count > cost:
                affordable_upgrades[cost] = id

        # purchase the most expensive affordable upgrade
        highest_price_affordable_upgrade = max(affordable_upgrades)
        print(highest_price_affordable_upgrade)
        to_purchase_id = affordable_upgrades[highest_price_affordable_upgrade]

        driver.find_element(By.ID, to_purchase_id).click()

        # add another 5 seconds until the next check
        timeout = time.time() + 5

    # after 5 minutes stop the bot and check the cookies per second count
    if time.time() > five_min:
        cookie_per_s = driver.find_element(By.ID, "cps").text
        print(cookie_per_s)
        break
Python Tutorial, Day 34
day 34 selenium web automation cookie clicker
This post is licensed under CC BY 4.0 by the author.