IN5800 – Data Structure

Leif Harald Karlsen

Data Structure and Complexity

Data can be organized according to how structured it is:

Unstructured
- E.g. images, text, sound
- No metadata about the meaning of the content
- Typically use machine learning/AI/statistics to get meaning
Semi-structured
- E.g. (free) graphs, trees
- Some metadata, typically contained within the data itself
- Typically assume some structure or use semantics to get meaning
Structured
- E.g. tables, relations
- Strict separation between metadata and data
- Meaning (mostly) contained in metadata
This course focuses on semi-structured and structured.

Relational data

Data represented as rows/tuples in tables/relations organized into columns/attributes
Rich metadata describing types, identity, references, uniqueness, etc.

Relational structure

Assuming a normalized schema (BCNF), we can derive from the structure:
- All entities (denoted by (non-referencing) primary keys) and associated attributes
- All relationships (denoted by foreign keys)
Structure makes using, securing, validating and retrieving the data easier/more efficient
By far the most used structure for data

Problems with relational structure

Structure difficult to create and maintain
- Difficult to predict how DB will look in 1, 5, 10, 20 years
- When organizations evolve, so does data
- Programs/queries depends on the concrete structure
Integration of different databases difficult
- Need to agree on structure and terms
Allows only one type of structure
- E.g. difficult use graphs, trees, etc. stored as relations

NoSQL

Central idea: Loosen up the rigid structure (semi-structured)

Easier to create and maintain
Easier integration
Easier to scale horizontally (split data over multiple clusters)
Limits object-relational impendance mismatch
Allows different forms of data in same DB
Note: Often read as “Not Only SQL” (Many NoSQL-DBs support SQL)

RDF/Triples

All statements are triples: s p o
- Subject
- Predicate
- Object
E.g. Leif teaches IN5800
Data represented as sets of triples
More general than a graph
- Edges between edges, e.g. teaches type Relationship
- Also: type type Relationship

Resources and data

Elements in triples are either:
- URI/IRI (globally unique identifiers)
- Blank node (anonymous/unnamed nodes)
- Data value (strings, integers, floats, dates, etc.)
Blanks only allows in subject and object position
Data values only allowed in object position
Can use prefixes for IRIs
- Instead of <https://www.uio.no/studier/emner/matnat/ifi/IN5800> can set
```
@prefix ifi: <https://www.uio.no/studier/emner/matnat/ifi/> .
```
  and now write ifi:IN5800

Example graph (in Turtle)

@prefix ifi: <https://www.uio.no/studier/emner/matnat/ifi/> .
@prefix folk: <http://folk.uio.no/> .
@prefix ex: <http://example.com/> .

folk:leifhka ex:hasName "Leif Harald Karlsen" .
folk:leifhka ex:teaches ifi:IN5800 .
folk:leifhka ex:teaches ifi:IN2090 .
folk:leifhka ex:knows folk:martingi .
folk:martingi ex:teaches ifi:IN3060 .
ifi:IN5800 ex:hasCredits 10 .
ifi:IN3060 ex:alias ifi:IN4060 .

More compact syntax:

@prefix ifi: <https://www.uio.no/studier/emner/matnat/ifi/> .
@prefix folk: <http://folk.uio.no/> .
@prefix ex: <http://example.com/> .

folk:leifhka ex:hasName "Leif Harald Karlsen" ;
             ex:teaches ifi:IN5800, ifi:IN2090 ;
             ex:knows folk:martingi .
folk:martingi ex:teaches ifi:IN3060 .
ifi:IN5800 ex:hasCredits 10 .
ifi:IN3060 ex:alias ifi:IN4060 .

Triple’s metadata

Data can describe itself:
- ex:teaches rdfs:domain ex:Teacher and ex:teaches rdfs:range ex:Course
- rdf:type rdf:type rdfs:Property and rdf:Property rdf:type rdfs:Class
- Terms defined using other terms (just like in natural language!)
- If you know the meaning of some words/terms/IRIs, can derive meaning of others
Typically agree on the meaning of standardized vocabularies
- rdfs:Class, rdf:type, rdf:Property, rdfs:domain, rdfs:range, etc.
And relate everything else to these
Since IRIs are globally unique, no ambiguity
More about this when we get to semantics
Note: Can also use Quads/Named graphs where we assign a context to each triple

Relational vs. triples

We can just make a table/relation Triple(subject, predicate, object) in a relational database
So what is the difference between relational and triple data?
When we say “relational”, we assume that the table structure reflects the domain, e.g.
- domain entities are each represented with separate tables
- relations between entities use foreign keys

Translating between structures

Different structures have different properties
Using the structure most fit for the operation often worth it
Thus, needs to translate/map between structures (e.g. from relational to triples)
These translatsions are known as mappings
Possible to translate almost any structure to some other structure
Will have a separate lecture on this topic

Other structures (CSV, JSON, XML)

Other tabular structures: CSV
Key-value: JSON
Trees: XML
Note: Can store JSON and XML directly in many relational DBs (e.g. PostgreSQL)

Definitions: Format and Structure

Format: The concrete layout of data in memory/disk with procedures for manipulation
- SQL Tables
- CSV
- RDF
- XML, JSON, Excel, etc.
Data structure: Mathematical description of the layout and manipulation of data + data representation approach + minimal semantics
- Relational (tables/relations)
- Triple-based (RDF, RDFS, OWL)
- Hierarchical (trees)
- Graph-based (property graphs/networks)
- Key-value-based

Data Formats and Structures

The data representation approach of a structure states how the structure can be used to represent different types of informaiton
- E.g. how entities, attributes, relationships, etc. should be represented within the structure
The minimal semantics of a structure states the underlying assumptions about truth in the structure
- Closed vs. open world
- Uniqueness of names
- Truth values (boolean vs. three-valued logic)
A structure can sometimes be encoded in many formats (Triples in both SQL Tables and RDF)
A format can sometimes support many structures (SQL Tables supports both relations and triples)

Data Format vs. Data Structure: Example

Format: SQL tables – Data structure: Relational

         Company
      
 cid | name |  founded   
-----+------+------------
   1 | UiO  | 1811-09-02
   2 | DNB  | 2003-12-03


         Person         
       
 pid | name  | worksfor 
-----+-------+----------
   1 | Peter |        2
   2 | Kari  |        1
   3 | Mary  |        1
   4 | Nils  |

Format: SQL tables – Data structure: Triples

                                Nodes                                 
                                                        
 id  | ntype  |                     svalue                      | lang 
-----+--------+-------------------------------------------------+------
   1 | uri    | http://www.w3.org/1999/02/22-rdf-syntax-ns#type | 
   2 | uri    | http://www.w3.org/2001/XMLSchema#date           | 
 101 | uri    | http://example.org/comp                         | 
 102 | uri    | http://example.org/comp/cid                     | 
 103 | uri    | http://example.org/comp/cid/1                   | 
 104 | uri    | http://example.org/comp/cid/2                   | 
 105 | uri    | http://example.org/comp/founded                 | 
 106 | uri    | http://example.org/comp/name                    | 
 107 | uri    | http://example.org/pers                         | 
 108 | uri    | http://example.org/pers/name                    | 
 109 | uri    | http://example.org/pers/pid                     | 
 110 | uri    | http://example.org/pers/pid/1                   | 
 111 | uri    | http://example.org/pers/pid/2                   | 
 112 | uri    | http://example.org/pers/pid/3                   | 
 113 | uri    | http://example.org/pers/pid/4                   | 
 114 | uri    | http://example.org/pers/worksfor                | 
 115 | string | DNB                                             | 
 116 | string | Kari                                            | 
 117 | string | Mary                                            | 
 118 | string | Nils                                            | 
 119 | string | Peter                                           | 
 120 | string | UiO                                             | 
 121 | date   | 1811-09-02                                      | 
 122 | date   | 2003-12-03                                      | 


          Triples
             
 subject | predicate | object 
---------+-----------+--------
     103 |         1 |    101
     103 |       105 |    121
     103 |       106 |    120
     104 |         1 |    101
     104 |       105 |    122
     104 |       106 |    115
     110 |         1 |    107
     110 |       108 |    119
     110 |       114 |    104
     111 |         1 |    107
     111 |       108 |    116
     111 |       114 |    103
     112 |         1 |    107
     112 |       108 |    117
     112 |       114 |    103
     113 |         1 |    107
     113 |       108 |    118

Format: RDF – Data structure: Triples

@prefix ex:   <http://example.org/public/> .
@prefix ex-c: <http://example.org/public/Company/> .
@prefix ex-p: <http://example.org/public/pers/> .

ex-c:cid1 a  ex:Company ;
  ex-c:founded 1811-09-02 ;
  ex-c:name "UiO" .

ex-c:cid2 a ex:Company ;
  ex-c:founded 2003-12-03 ;
  ex-c:name "DNB" .

ex-p:pid1 a ex:Person .
  ex-p:name "Peter" ;
  ex-p:worksfor ex-c:cid2 .

ex-p:pid2 a ex:Person ;
  ex-p:name "Kari" ;
  ex-p:worksfor ex-c:cid1 .

ex-p:pid3 a ex:Person ;
  ex-p:name "Mary" ;
  ex-p:worksfor ex-c:cid1 .

ex-p:pid4 a ex:Person ;
  ex-p:name "Nils" .

Format: RDF – Data structure: Relational

ex:Company a ex:Table ;
  ex:columns ("cid", "name", "founded") ;
  ex:row (1, "UiO", "1811-09-02"^^xsd:date) ,
         (2, "DNB", "2003-12-03"^^xsd:date) .

ex:Person a ex:Table ;
  ex:columns ("pid", "name", "worksfor") ;
  ex:row (1, "Peter", 2) ,
         (2, "Kari",  1) ,
         (3, "Mary",  1) ,
         (4, "Nils",  ex:null) .

Definitions: Data Schema

A data schema (or model) is a description of data
Conceptual data schema: A description of entites/individuals, attributes, relationships, annotations, etc. data describes
Logical data schema: A description of data in terms of a particular data structure
Physical data schema: A description of data in terms of a particular format
Continous transition between these schemas, from abstract (conceptual) to concrete (physical)
For each conceptual schema, there can typically be many adhering logical schemas
For each logical schema, there can typically be many adhering physical schemas
Any concrete (meaningful) dataset has one schema of each
- But a schema can be implicit, e.g. based on the natural/intuitive interpretation of the data

Conceptual vs. Logical vs. Physical Data Schema: Relational Example

CREATE TABLE company(
    cid int PRIMARY KEY,
    name text,
    founded date
);

CREATE TABLE person(
    pid int PRIMARY KEY,
    name text,
    worksFor int REFERENCES Company(cid)
);

CREATE INDEX person_name ON person(name);

CREATE TABLE company(
    cid bigint PRIMARY KEY,
    name text,
    founded int
);

CREATE TABLE person(
    pid bigint PRIMARY KEY,
    name text,
    worksFor bigint REFERENCES Company(cid)
);

CREATE INDEX person_name ON person(name);
CREATE INDEX company_name ON company(name);

Example: Different schemas for different use

Setting: Train company selling tickets
Has two databases:
- [Operational] One for registering customers, searching and booking tickets, etc.
  - Transactional (OLTP): SELECT, UPDATE, INSERT, DELTE, etc.
  - Logical: Normalized
  - Physical: Row-oriented storage
- [Reporting] One for making reports, decision support, planning mainenance, etc.
  - Analytical (OLAP): Only INSERT and SELECT
  - Logical: Denormalized, aggregated
  - Physical: Column-oriented storage

Example cont.: Differnet physical schemas

Explicit vs. Implicit

If we have a column containing people’s birth dates, we can compute their age
Birth date is explicit, but age is implicit in this data
Implicit data can be highly valuable
- Grouping and aggregation typical for extracting implicit data
- E.g. find storms from sensor data or bills from orders
Often important to think of what implicit data there is within a dataset
The computed values can be represented via VIEWS or similar
Lots of implicit data can be derived if we make the semantics/meaning of data explicit!
- More on this later

Virtual vs. Stored

Stored data is all the data residing on disk
Virtual data is implicit data for which there is a stored definition
Thus, virtual data can be computed on demand
Views is one example of virtual data
In fact, whole databases can be virtual
We will later see other methods for virtual data

Qualitative vs. Quantitative

Quantitative data is numerical data denoting a value along one or more axis
- s1 measures 5 degrees Celsius
- T happened 10:02:13 21.01.21
- X is at POINT(1.0, 2.3)
Qualitative data
- s1 is warm
- T happened before P
- X is contained in Y
Qualitative data often only relates object to discrete values or other objects
Qualitative data is often implicit in quantitative data
In computers, data is often stored and used as quantitative data
Humans tend to think and talk qualitatively

Data structure and use

Different structures are good for different things, e.g.
- Triples good for integration and maintenance, relational for security and efficiency
- Virtual data saves space, but stored is computationally more efficient
- Implicit data might be useful, but impossible to extract/define everything
- Qualitative might be more convenient for humans, but expensive to compute/store
Need to think about these trade-offs when engineering data