What is Parsel in Python?

Parsel is a Python library used to extract structured data from HTML, XML, and JSON documents. It provides a Selector API that makes it easy to locate and extract specific content.

• Supports XPath and CSS selectors for HTML and XML.
• Supports JMESPath for extracting data from JSON.
• Can extract text, attributes, and HTML content.
• Commonly used for web scraping and is also used by Scrapy.
• Provides methods such as get() and getall() for retrieving extracted data.

Installing Parsel

It can be installed using pip:

pip install parsel

After installation, import the Selector class to begin parsing content.

from parsel import Selector

Basic Usage

Parsel works by creating a Selector object from HTML or XML content. The selector can then be used with XPath or CSS expressions to find the required elements.

from parsel import Selector

html = """
<html>
    <body>
        <h1>Python Web Scraping</h1>
        <p>Learn data extraction with Parsel.</p>
    </body>
</html>
"""

selector = Selector(text=html)

print(selector.xpath('//h1/text()').get())

Output

Python Web Scraping

Explanation:

• Selector parses the HTML string and creates a selectable document.
• xpath() locates the h1 element and its text.
• get() returns the first matching result.

Extracting Data Using XPath

XPath can be used to locate elements based on their position, attributes, relationships, and other conditions.

from parsel import Selector

html = """
<html>
    <body>
        <h1>Python Web Scraping</h1>
        <p class="description">Parsel makes data extraction easier.</p>
    </body>
</html>
"""

selector = Selector(text=html)

description = selector.xpath('//p/text()').get()

print(description)

Output

Parsel makes data extraction easier.

Explanation:

• xpath() searches the document using an XPath expression.
• //p/text() selects the text inside p elements.
• get() returns the first matched value.

Extracting Data Using CSS Selectors

Parsel also supports CSS selectors, which can be more readable for common HTML selections. Text can be selected using the Parsel-specific ::text pseudo-element.

from parsel import Selector

html = """
<html>
    <body>
        <h1>Python Web Scraping</h1>
        <p class="description">Parsel makes data extraction easier.</p>
    </body>
</html>
"""

selector = Selector(text=html)

description = selector.css('p.description::text').get()

print(description)

Output

Parsel makes data extraction easier.

Explanation:

• css() selects elements using CSS selector syntax.
• p.description selects a p element having the description class.
• ::text extracts the text contained by the selected element.

Extracting Multiple Elements

When multiple elements match a selector, getall() can be used to return all results as a list. In contrast, get() returns only the first matching result.

from parsel import Selector

html = """
<ul>
    <li>Python</li>
    <li>Java</li>
    <li>C++</li>
</ul>
"""

selector = Selector(text=html)

languages = selector.css('li::text').getall()

print(languages)

Output

['Python', 'Java', 'C++']

Explanation:

• css() finds all li elements.
• ::text selects their text values.
• getall() returns every matched value in a list.

Advanced Usage of Parsel

Extracting Attributes

Parsel can extract attributes such as href, src, class, and id from HTML elements. CSS selectors can use the ::attr() extension for this purpose.

from parsel import Selector

html = """
<a href="https://example.com">Visit Example</a>
"""

selector = Selector(text=html)

link = selector.css('a::attr(href)').get()

print(link)

Output

https://example.com

Explanation:

• a selects the anchor element.
• ::attr(href) selects its href attribute.
• get() returns the first matching attribute value.

Chaining Selectors

Selection methods can be chained to move from a parent element to its nested elements. This is useful when a page contains repeated structures and the required data exists inside a specific section.

from parsel import Selector

html = """
<div id="content">
    <h1>Parsel Tutorial</h1>
    <p>Learn Parsel in Python.</p>
</div>
"""

selector = Selector(text=html)

title = selector.css('#content').css('h1::text').get()

print(title)

Output

Parsel Tutorial

Explanation:

• First selector locates the content container.
• Second selector searches for h1 inside that container.
• get() extracts the resulting text.

Using get() and getall()

Two commonly used extraction methods in Parsel are get() and getall(). The current Parsel documentation recommends these methods because their return behavior is predictable.

from parsel import Selector

html = """
<ul>
    <li>Python</li>
    <li>Java</li>
    <li>C++</li>
</ul>
"""

selector = Selector(text=html)

print(selector.css('li::text').get())
print(selector.css('li::text').getall())

Output

Python
['Python', 'Java', 'C++']

Explanation:

• get() returns one result, or None when no result is found.
• getall() returns all matching results as a list.
• get() is useful when only one value is required, while getall() is useful for multiple values.

Using Parsel with Scrapy

Parsel is closely associated with Scrapy. Scrapy uses Parsel selectors to select and extract data from responses using CSS and XPath expressions.

import scrapy

class ExampleSpider(scrapy.Spider):
    name = "example"
    start_urls = ["https://example.com/"]

    def parse(self, response):
        title = response.css("title::text").get()
        yield {"title": title}

Output

{'title': 'Example Domain'}

Explanation:

• Scrapy downloads the webpage and provides its response to parse().
• response.css() uses the Parsel selector API.
• get() extracts the page title.

Parsel vs BeautifulSoup vs lxml