Python Automation

5.

Web Automation and Scraping

5.1.

Fundamentals of Web Technologies

5.1.1.

Basic HTML Structure

5.1.1.1.

HTML Elements and Tags

5.1.1.2.

5.1.1.3.

Document Object Model

5.1.1.4.

Common HTML Elements

5.1.2.

5.1.2.1.

Element Selectors

5.1.2.2.

Class Selectors

5.1.2.3.

5.1.2.4.

Attribute Selectors

5.1.2.5.

Descendant Selectors

5.1.2.6.

Pseudo-selectors

5.1.3.

Understanding HTTP

5.1.3.1.

5.1.3.1.1.

5.1.3.1.2.

5.1.3.1.3.

5.1.3.1.4.

DELETE Requests

5.1.3.2.

HTTP Status Codes

5.1.3.3.

Request Headers

5.1.3.4.

Response Headers

5.1.3.5.

Cookies and Sessions

5.2.

Ethical Considerations and Best Practices

5.2.1.

Understanding robots.txt

5.2.2.

Respecting Rate Limits

5.2.3.

User-Agent Headers

5.2.4.

Avoiding Server Overload

5.2.5.

Legal Considerations

5.2.6.

Privacy and Data Protection

5.2.7.

Terms of Service Compliance

5.3.

Making Web Requests with requests

5.3.1.

Installing the requests Library

5.3.2.

Basic GET Requests

5.3.3.

Basic POST Requests

5.3.4.

Handling Request Parameters

5.3.5.

Working with Headers

5.3.6.

Managing Cookies

5.3.7.

Session Objects

5.3.8.

Handling Redirects

5.3.9.

Setting Timeouts

5.3.10.

SSL Verification

5.3.11.

5.3.12.

Error Handling in Requests

5.3.13.

Response Object Methods

5.4.

Parsing HTML with BeautifulSoup

5.4.1.

Installing BeautifulSoup

5.4.2.

Creating Soup Objects

5.4.3.

Parsing HTML Content

5.4.4.

Finding Elements

5.4.4.1.

5.4.4.2.

find_all Method

5.4.4.3.

CSS Selectors with select

5.4.5.

Navigating the HTML Tree

5.4.5.1.

Parent Elements

5.4.5.2.

5.4.5.3.

Sibling Elements

5.4.6.

Extracting Data

5.4.6.1.

5.4.6.2.

Attribute Values

5.4.6.3.

5.4.7.

5.4.8.

Handling Different Parsers

5.4.9.

Encoding Issues

5.5.

Browser Automation with Selenium

5.5.1.

Understanding Browser Automation

5.5.2.

Installing Selenium

5.5.3.

WebDriver Setup

5.5.3.1.

Chrome WebDriver

5.5.3.2.

Firefox WebDriver

5.5.3.3.

5.5.4.

Basic Browser Control

5.5.4.1.

Opening Browsers

5.5.4.2.

Navigating to URLs

5.5.4.3.

Closing Browsers

5.5.5.

Locating Web Elements

5.5.5.1.

5.5.5.2.

5.5.5.3.

5.5.5.4.

5.5.5.5.

By CSS Selector

5.5.5.6.

5.5.5.7.

5.5.6.

Interacting with Elements

5.5.6.1.

Clicking Elements

5.5.6.2.

Sending Text Input

5.5.6.3.

Clearing Input Fields

5.5.6.4.

Submitting Forms

5.5.6.5.

Selecting from Dropdowns

5.5.6.6.

Handling Checkboxes and Radio Buttons

5.5.7.

Handling Dynamic Content

5.5.7.1.

5.5.7.2.

5.5.7.3.

Expected Conditions

5.5.7.4.

Custom Wait Conditions

5.5.8.

Advanced Browser Operations

5.5.8.1.

Executing JavaScript

5.5.8.2.

Taking Screenshots

5.5.8.3.

Managing Windows and Tabs

5.5.8.4.

Handling Alerts and Pop-ups

5.5.8.5.

File Upload and Download

5.5.9.

Headless Browser Mode

5.5.10.

Browser Options and Preferences

Previous

4. Working with Common Data Formats

Go to top

Next

6. Interacting with APIs