Working with corpus data
========================


.. meta::
    :description lang=en:
        After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.


After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.


Generic solutions in Python
---------------------------


Data science
~~~~~~~~~~~~

- Load the input into the data analysis library Pandas:
   - `read_csv <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html>`_
   - `read_json <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html>`_


Natural language processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~

For a first hand approach to NLP pipelines, see `Textblob <https://textblob.readthedocs.io/en/dev/>`_ or the `Natural Language Toolkit (NTLK) <https://www.nltk.org/>`_.


Accessible tutorials:
    - `Part-of-Speech Tagging <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/POS-Keywords.html>`_
    - `TF-IDF with Scikit-Learn <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/TF-IDF-Scikit-Learn.html>`_


Specific tools:
    - Topic modeling, including word2vec models: `Gensim tutorials <https://radimrehurek.com/gensim/auto_examples/>`_
    - `Scattertext <https://github.com/JasonKessler/scattertext>`_ is a tool for finding distinguishing terms in corpora, and presenting them in an interactive scatter plot.



Formats and software used in corpus linguistics
-----------------------------------------------


Input/Output formats: TXT, XML and XML-TEI are quite frequent in corpus linguistics.

- Han., N.-R. (2022). "`Transforming Data <https://doi.org/10.7551/mitpress/12200.003.0010>`_", The Open Handbook of Linguistic Data.


The XML and XML-TEI formats
~~~~~~~~~~~~~~~~~~~~~~~~~~~

See `A Gentle Introduction to XML <https://tei-c.org/release/doc/tei-p5-doc/en/html/SG.html>`_ or the Python package `xmltodict <https://github.com/martinblech/xmltodict>`_ which provide ways to directly read the files and work with the data as if it were in JSON format.


Corpus analysis tools
~~~~~~~~~~~~~~~~~~~~~


- `Antconc <https://www.laurenceanthony.net/software/antconc/>`_ is expected to work with TXT files
- `CorpusExplorer <https://notes.jan-oliver-ruediger.de/software/corpusexplorer-overview/>`_ supports CSV, TXT and various XML formats
- `Corpus Workbench (CWB) <https://cwb.sourceforge.io/>`_ uses verticalized texts whose origin can be in TXT or XML format
- `LancsBox <http://corpora.lancs.ac.uk/lancsbox/>`_ support various formats, notably TXT & XML
- `TXM <http://textometrie.ens-lyon.fr/?lang=en>`_ (textometry platform) can take TXT, XML & XML-TEI files as input
- `Voyant <https://voyant-tools.org/>`_ support various formats, notably TXT, XML & XML-TEI
- `Wmatrix <http://ucrel.lancs.ac.uk/wmatrix/>`_ can work with TXT and XML
- `WordSmith <https://lexically.net/wordsmith/index.html>`_ supports TXT and XML

Further corpus analysis software can be found on `corpus-analysis.com <https://corpus-analysis.com/>`_.


Generic NLP solutions
---------------------

For natural language processing see this list of open-source/off-the-shelf `NLP tools for German <https://github.com/adbar/German-NLP>`_ and `further lists for other languages <https://github.com/adbar/German-NLP#Comparable-lists>`_.