Parsing XML Files and Extracting Data Using xidel

“xidel” is a command line tool that can scrape and retrieve data from web pages and websites in HTML, XML, and JSON formats using CSS selectors and Xpath. It can be used on the Bash programming language (scripting language) command line on Linux PC/server.

A command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern templates. It can also create new or transformed XML/HTML/JSON documents. http://www.videlibri.de/xidel.html

• benibela/xidel: A command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern templates. It can also create new or transformed XML/HTML/JSON documents.

Download and Installation

wget https://sourceforge.net/projects/videlibri/files/Xidel/Xidel%200.9.8/xidel_0.9.8-1_amd64.deb
sudo dpkg -i xidel_0.9.8-1_amd64.deb

Usage Example

With a command like:

xidel /mnt/e/_downloads/scrape_anime.xml --xpath '//myanimelist/anime/series_title'

From XML like:

<?xml version="1.0" encoding="UTF-8" ?>
<myanimelist>
    <myinfo>
        <user_id></user_id>
        <user_name>Fumiya_I</user_name>
        <user_export_type>1</user_export_type>
        <user_total_anime>261</user_total_anime>
        <user_total_watching>8</user_total_watching>
        <user_total_completed>151</user_total_completed>
        <user_total_onhold>33</user_total_onhold>
        <user_total_dropped>24</user_total_dropped>
        <user_total_plantowatch>45</user_total_plantowatch>
    </myinfo>

    <anime>
        <series_animedb_id>477</series_animedb_id>
        <series_title><![CDATA[Aria The Animation]]></series_title>
        <series_type>TV</series_type>
        <series_episodes>13</series_episodes>
        <my_id>0</my_id>
        <my_watched_episodes>6</my_watched_episodes>
        <my_start_date>0000-00-00</my_start_date>
        <my_finish_date>0000-00-00</my_finish_date>
        <my_rated></my_rated>
        <my_score>8</my_score>
        <my_dvd></my_dvd>
        <my_storage></my_storage>
        <my_status>Watching</my_status>
        <my_comments><![CDATA[]]></my_comments>
        <my_times_watched>0</my_times_watched>
        <my_rewatch_value>Low</my_rewatch_value>
        <my_tags><![CDATA[]]></my_tags>
        <my_rewatching>0</my_rewatching>
        <my_rewatching_ep>0</my_rewatching_ep>
        <update_on_import>0</update_on_import>
    </anime>
    <anime>
        <series_animedb_id>37525</series_animedb_id>
        <series_title><![CDATA[Babylon]]></series_title>
        <series_type>TV</series_type>
        <series_episodes>12</series_episodes>
        <my_id>0</my_id>
        <my_watched_episodes>0</my_watched_episodes>
        <my_start_date>0000-00-00</my_start_date>
        <my_finish_date>0000-00-00</my_finish_date>
        <my_rated></my_rated>
        <my_score>9</my_score>
        <my_dvd></my_dvd>
        <my_storage></my_storage>
        <my_status>Watching</my_status>
        <my_comments><![CDATA[]]></my_comments>
        <my_times_watched>0</my_times_watched>
        <my_rewatch_value>Low</my_rewatch_value>
        <my_tags><![CDATA[]]></my_tags>
        <my_rewatching>0</my_rewatching>
        <my_rewatching_ep>0</my_rewatching_ep>
        <update_on_import>0</update_on_import>
    </anime>

You can get data like the following:

Aria The Animation
Babylon
Beastars
Boku no Hero Academia 4th Season
Psycho-Pass 3
Shinchou Yuusha: Kono Yuusha ga Ore Tueee Kuse ni Shinchou Sugiru
Sword Art Online: Alicization - War of Underworld
Vinland Saga