Python XML Processing

Python XML Processing

XML (Extensible Markup Language) is a markup language designed to store and transport data in a format that is both human-readable and machine-readable. It’s widely used in configuration files, data exchange, and document formats.

What is XML?

XML defines a set of rules for encoding documents. It uses tags (like HTML) to define elements and attributes to provide metadata about those elements. Unlike HTML, XML does not predefine tags; users define their own tags relevant to their data.

Key Features:

• Extensible: You define your own tags.
• Hierarchical: Data is structured in a tree format.
• Text-based: Uses Unicode characters.
• Self-descriptive: Tags often describe the data they contain.
• Platform/Language Independent: An open standard.

While JSON has become more popular for web APIs due to its simplicity and direct mapping to JavaScript objects, XML remains important for many enterprise systems, document standards (like DocBook, Office Open XML), and configuration.

Python XML Parsing APIs

Python’s standard library provides several modules for working with XML, supporting different parsing strategies:

• xml.etree.ElementTree: (Recommended for most common tasks) Provides a simple and Pythonic way to parse and create XML. It loads the entire document into a tree structure in memory (similar to DOM) but offers a more lightweight and often easier-to-use API.
• xml.dom.minidom: Implements the Document Object Model (DOM) API, a W3C standard. It parses the entire XML document into a tree of objects in memory, allowing full access and modification. Can be memory-intensive for very large files.
• xml.sax: Implements the Simple API for XML (SAX), an event-driven parser. It reads the XML file sequentially, triggering callback functions (handlers) for events like start tags, end tags, and character data. It’s memory-efficient as it doesn’t load the whole document, making it suitable for very large files or stream processing, but can be more complex to work with for random access.

Choosing an API:

• For general XML parsing and creation, start with ElementTree.
• If you need strict DOM compliance or complex tree modifications, use minidom.
• For processing very large XML files with limited memory, use SAX.

Sample XML File (movies.xml)

We’ll use the following XML file for examples:

<?xml version="1.0" encoding="UTF-8"?>
<collection shelf="New Arrivals">
    <movie title="Enemy Behind">
        <type>War, Thriller</type>
        <format>DVD</format>
        <year>2003</year>
        <rating>PG</rating>
        <stars>10</stars>
        <description>Talk about a US-Japan war</description>
    </movie>
    <movie title="Transformers">
        <type>Anime, Science Fiction</type>
        <format>DVD</format>
        <year>1989</year>
        <rating>R</rating>
        <stars>8</stars>
        <description>A scientific fiction</description>
    </movie>
    <movie title="Trigun">
        <type>Anime, Action</type>
        <format>DVD</format>
        <episodes>4</episodes>
        <rating>PG</rating>
        <stars>10</stars>
        <description>Vash the Stampede!</description>
    </movie>
    <movie title="Ishtar">
        <type>Comedy</type>
        <format>VHS</format>
        <rating>PG</rating>
        <stars>2</stars>
        <description>Viewable boredom</description>
    </movie>
</collection>

Parsing XML with ElementTree (Recommended)

ElementTree represents the XML document as a tree of Element objects.

Example:

#!/usr/bin/env python3

import xml.etree.ElementTree as ET

xml_file = 'movies.xml'

try:
    tree = ET.parse(xml_file)
    root = tree.getroot() # Get the root element ('collection')

    print(f"Root element: {root.tag}, Shelf: {root.get('shelf')}")

    # Iterate through 'movie' elements under the root
    for movie in root.findall('movie'): # findall finds direct children
        title = movie.get('title') # Get attribute value
        year_element = movie.find('year') # find gets the first matching child
        type_element = movie.find('type')

        # Element text content is in .text attribute
        movie_type = type_element.text if type_element is not None else 'N/A'
        year = year_element.text if year_element is not None else 'N/A'

        print(f"\nMovie Title: {title}")
        print(f"  Type: {movie_type}")
        print(f"  Year: {year}")

        # Accessing potentially missing elements safely
        episodes_element = movie.find('episodes')
        if episodes_element is not None:
            print(f"  Episodes: {episodes_element.text}")

except ET.ParseError as e:
    print(f"Error parsing XML: {e}")
except FileNotFoundError:
    print(f"Error: File '{xml_file}' not found.")

Output:

Root element: collection, Shelf: New Arrivals

Movie Title: Enemy Behind
  Type: War, Thriller
  Year: 2003

Movie Title: Transformers
  Type: Anime, Science Fiction
  Year: 1989

Movie Title: Trigun
  Type: Anime, Action
  Year: N/A
  Episodes: 4

Movie Title: Ishtar
  Type: Comedy
  Year: N/A

Key ElementTree Concepts:

• ET.parse(source): Parses XML from a file path or file-like object.
• ET.fromstring(text): Parses XML from a string.
• tree.getroot(): Returns the root element of the parsed tree.
• element.tag: The element’s tag name (string).
• element.attrib: A dictionary of the element’s attributes.
• element.text: The text content directly within the element.
• element.get(attr_name): Gets the value of an attribute.
• element.find(match): Finds the first direct child element matching the tag name or path.
• element.findall(match): Finds all direct child elements matching the tag name or path.
• element.iter(tag): Creates an iterator over all sub-elements (direct and nested) matching the tag.

Parsing XML with DOM (minidom)

minidom parses the XML into a standard DOM tree structure.

Example:

#!/usr/bin/env python3

from xml.dom import minidom
import xml.parsers.expat # Often needed to catch specific parse errors

xml_file = 'movies.xml'

try:
    # Parse the XML file into a DOM tree
    dom_tree = minidom.parse(xml_file)

    # Get the collection element (root)
    collection = dom_tree.documentElement # The root <collection> element

    if collection.hasAttribute("shelf"):
        print(f"Root element: <{collection.tagName}>, Shelf: {collection.getAttribute('shelf')}")

    # Get all 'movie' elements
    movies = collection.getElementsByTagName("movie")

    # Print detail of each movie
    for movie in movies:
        print("\n***** Movie *****")
        if movie.hasAttribute("title"):
            print(f"Title: {movie.getAttribute('title')}")

        # Helper function to get text from first child element
        def get_element_text(parent, tag_name):
            elements = parent.getElementsByTagName(tag_name)
            if elements and elements[0].firstChild:
                # Check if firstChild exists and is a text node
                if elements[0].firstChild.nodeType == elements[0].TEXT_NODE:
                     return elements[0].firstChild.data.strip()
            return 'N/A'

        print(f"Type: {get_element_text(movie, 'type')}")
        print(f"Format: {get_element_text(movie, 'format')}")
        print(f"Rating: {get_element_text(movie, 'rating')}")
        print(f"Description: {get_element_text(movie, 'description')}")

except xml.parsers.expat.ExpatError as e:
     print(f"Error parsing XML: {e}")
except FileNotFoundError:
    print(f"Error: File '{xml_file}' not found.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

Output (similar to ElementTree, demonstrates DOM access):

Root element: <collection>, Shelf: New Arrivals

***** Movie *****
Title: Enemy Behind
Type: War, Thriller
Format: DVD
Rating: PG
Description: Talk about a US-Japan war

***** Movie *****
Title: Transformers
Type: Anime, Science Fiction
Format: DVD
Rating: R
Description: A scientific fiction
...

Key DOM Concepts:

• minidom.parse(source): Parses the XML.
• dom.documentElement: The root element node.
• element.tagName: Element’s tag name.
• element.getAttribute(name): Gets attribute value.
• element.hasAttribute(name): Checks if attribute exists.
• element.getElementsByTagName(name): Gets a list of all descendant elements with the given tag name.
• node.firstChild, node.childNodes, node.parentNode: Navigation through the tree.
• node.nodeType: Type of node (ELEMENT_NODE, TEXT_NODE, etc.).
• node.data: Text content of a TEXT_NODE.

Parsing XML with SAX (xml.sax)

SAX parsing requires creating a ‘Content Handler’ class that defines methods to be called when the parser encounters specific XML structures (start/end tags, characters).

Example:

#!/usr/bin/env python3

import xml.sax

# Define the Content Handler class
class MovieHandler(xml.sax.ContentHandler):
    def __init__(self):
        super().__init__()
        self.current_data = "" # Stores the tag name being processed
        self.type = ""
        self.format = ""
        self.year = ""
        self.rating = ""
        self.stars = ""
        self.description = ""
        self.episodes = "" # Added for Trigun
        self.current_movie_title = ""

    # Called when an element starts
    def startElement(self, tag, attributes):
        self.current_data = tag
        if tag == "movie":
            print("\n***** Movie *****")
            self.current_movie_title = attributes.get("title", "N/A")
            print(f"Title: {self.current_movie_title}")
            # Reset movie-specific fields
            self.type = self.format = self.year = self.rating = ""
            self.stars = self.description = self.episodes = ""

    # Called when an element ends
    def endElement(self, tag):
        if tag == "type":
            print(f"  Type: {self.type}")
        elif tag == "format":
            print(f"  Format: {self.format}")
        elif tag == "year":
            print(f"  Year: {self.year}")
        elif tag == "rating":
            print(f"  Rating: {self.rating}")
        elif tag == "stars":
            print(f"  Stars: {self.stars}")
        elif tag == "description":
            print(f"  Description: {self.description}")
        elif tag == "episodes":
            print(f"  Episodes: {self.episodes}")

        # Reset current tag tracker
        self.current_data = ""

    # Called with character data between tags
    def characters(self, content):
        # Only store content if inside a relevant tag
        if self.current_data == "type":
            self.type += content.strip()
        elif self.current_data == "format":
            self.format += content.strip()
        elif self.current_data == "year":
            self.year += content.strip()
        elif self.current_data == "rating":
            self.rating += content.strip()
        elif self.current_data == "stars":
            self.stars += content.strip()
        elif self.current_data == "description":
            self.description += content.strip()
        elif self.current_data == "episodes":
            self.episodes += content.strip()

# --- Main execution ---
xml_file = 'movies.xml'

# Create a SAX parser
parser = xml.sax.make_parser()
# Turn off namespaces (optional, simplifies example)
parser.setFeature(xml.sax.handler.feature_namespaces, 0)

# Create an instance of our handler
handler = MovieHandler()
# Set the handler for the parser
parser.setContentHandler(handler)

print("--- Starting SAX Parse ---")
try:
    parser.parse(xml_file)
    print("\n--- SAX Parse Finished ---")
except xml.sax.SAXParseException as e:
    print(f"SAX Parsing Error: {e}")
except FileNotFoundError:
     print(f"Error: File '{xml_file}' not found.")

Output (similar content, different processing flow):

--- Starting SAX Parse ---

***** Movie *****
Title: Enemy Behind
  Type: War, Thriller
  Format: DVD
  Year: 2003
  Rating: PG
  Stars: 10
  Description: Talk about a US-Japan war

***** Movie *****
Title: Transformers
...

Key SAX Concepts:

• xml.sax.make_parser(): Creates a SAX parser object.
• xml.sax.ContentHandler: Base class for handling parsing events.
• startElement(tag, attributes): Called for each opening tag.
• endElement(tag): Called for each closing tag.
• characters(content): Called with chunks of character data between tags.
• parser.setContentHandler(handler): Associates your handler with the parser.
• parser.parse(source): Starts the parsing process.

XML Generation

ElementTree is also convenient for creating XML documents.

import xml.etree.ElementTree as ET

# Create root element
root = ET.Element("data")

# Create child elements
item1 = ET.SubElement(root, "item")
item1.set("name", "item1_name") # Set attribute
item1.text = "This is the first item."

item2 = ET.SubElement(root, "item", attrib={"name": "item2_name", "type": "testing"})
item2.text = "Second item content."

# Create a tree and write to file
tree = ET.ElementTree(root)

# Pretty print (optional, adds indentation)
ET.indent(tree, space="  ", level=0)

try:
    tree.write("output.xml", encoding="utf-8", xml_declaration=True)
    print("Generated output.xml")
except IOError as e:
    print(f"Error writing XML file: {e}")