Web Scraping

  1. Core Web Technologies for Scraping
    1. HyperText Transfer Protocol
      1. The Request-Response Model
        1. Client-Server Architecture
          1. Lifecycle of a Request
            1. Connection Management
            2. HTTP Methods
              1. GET Requests
                1. Retrieving Resources
                  1. Query Parameters
                    1. Caching Considerations
                    2. POST Requests
                      1. Submitting Data
                        1. Form Data Encoding
                          1. Request Body Formats
                          2. Other HTTP Methods
                            1. PUT Method
                              1. DELETE Method
                                1. PATCH Method
                              2. HTTP Headers
                                1. User-Agent Header
                                  1. Customizing User-Agent Strings
                                    1. Implications for Scraping
                                      1. Browser Fingerprinting
                                      2. Referer Header
                                        1. Tracking Navigation Paths
                                          1. Security Implications
                                          2. Accept and Content-Type Headers
                                            1. Content Negotiation
                                              1. MIME Types
                                            2. HTTP Status Codes
                                              1. Success Codes
                                                1. 200 OK
                                                  1. 201 Created
                                                    1. 204 No Content
                                                    2. Redirection Codes
                                                      1. 301 Moved Permanently
                                                        1. 302 Found
                                                          1. 304 Not Modified
                                                          2. Client Error Codes
                                                            1. 400 Bad Request
                                                              1. 401 Unauthorized
                                                                1. 403 Forbidden
                                                                  1. 404 Not Found
                                                                    1. 429 Too Many Requests
                                                                    2. Server Error Codes
                                                                      1. 500 Internal Server Error
                                                                        1. 502 Bad Gateway
                                                                          1. 503 Service Unavailable
                                                                      2. HyperText Markup Language
                                                                        1. Document Object Model
                                                                          1. Tree Structure
                                                                            1. Nodes and Elements
                                                                              1. DOM Manipulation
                                                                              2. HTML Tags and Attributes
                                                                                1. Common Structural Tags
                                                                                  1. Content Tags
                                                                                    1. Form Elements
                                                                                      1. Attribute Usage
                                                                                      2. Document Structure
                                                                                        1. HTML Document Declaration
                                                                                          1. Head Section Elements
                                                                                            1. Body Section Organization
                                                                                            2. Semantic HTML
                                                                                              1. Semantic Elements
                                                                                                1. Accessibility Considerations
                                                                                                2. HTML Parsing Challenges
                                                                                                  1. Malformed HTML
                                                                                                    1. Browser Compatibility
                                                                                                  2. Cascading Style Sheets
                                                                                                    1. CSS Selectors
                                                                                                      1. Basic Selectors
                                                                                                        1. Attribute Selectors
                                                                                                          1. Pseudo-classes
                                                                                                            1. Combinator Selectors
                                                                                                            2. CSS Impact on Scraping
                                                                                                              1. Hidden Elements
                                                                                                                1. Dynamic Classes and IDs
                                                                                                                  1. CSS-Generated Content
                                                                                                                  2. Responsive Design Considerations
                                                                                                                    1. Media Queries
                                                                                                                      1. Mobile vs. Desktop Layouts

                                                                                                                  Previous

                                                                                                                  1. Fundamentals of Web Scraping

                                                                                                                  Go to top

                                                                                                                  Next

                                                                                                                  3. The Web Scraping Process