next up previous contents index
Next: 6.2.4 Files & directories Up: 6.2 UIO data format Previous: 6.2.2 Example of UIO   Contents   Index


6.2.3 Structure of UIO files Data representation: ASCII or binary

While opening a file for writing, the file format (``formatted'' or ``unformatted'') and the conversion type (``native'', e.g.`ieee_4'', ...) have to be specified.

The formatted ASCII data representation allows I/O independent of platform or compiler. It is useful for parameter files which can be read and edited by hand, for the direct inspection of data, the transfer between very different systems, or for the import of data which exist e.g. in a table format. From the specified conversion type only the default output format for numbers (e.g.`E13.6'' for 4 byte reals) is determined.

The unformatted binary I/O is much faster and gives usually more compact files with higher accuracy (ideally exact) in the numerical data representation. But - in principle - the file format is machine dependent. Fortunately, the IEEE format turns out to become a quasi-standard among a variety of machines. Most workstations work internally with this format. Some CRAYS which have a different internal data representation allow the hidden transformation between the internal and IEEE format during the I/O process. The UIO routines support this feature of CRAY FORTRAN compilers by means of a module (uio_mac_module), individually designed for (two types of) CRAY machines using certain CRAY specific system calls (CRAY FFIO assign logic). Nevertheless, there is also a machine independent version of this module, written completely in standard Fortran90 but providing less features than the machine-dependent versions.

Besides the format, the conversion type (see table 6.2) has to be specified. The ``native'' conversion type is the internal binary-data representation, which is also standard for unformatted Fortran output. If this representation happens to be conformal with the IEEE standard the conversion type ``ieee_4'' should be used. It gives the same data format, but in the header of the file the term ``convert=ieee_4'' instead of ``convert=native'' describes the data format precisely - in a way also understandable by other machines. On CRAY machines the native format is equal to the conversion type ``crayxmp_8'', but also the conversion types ``ieee_4'', and ``ieee_8'' can be chosen. The last two conversion types correspond to the CRAY internal types ``ieee_32'' and ``ieee_64'', respectively.

On a machine with an internal data representation not within the list in the existing uio_mac_module.F90 file, one could use the ``native'' conversion type only. But it is better to invent an appropriate name for the new data format and to build a proper machine-dependent branch with its own compiler switch. However, it was possible in all cases so far to activate some internal conversion process, which allowed the handling of UIO files in the standard Big-Endian-IEEE format.

Table 6.2: UIO conversion types with length of integers, single precision reals, and double precision reals in bytes, and an explanation.
conversion type I R D description
native ? ? ? internal data format on all machines
        (sometimes useful but not recommended)
ieee_4 4 4 8 standard IEEE big_endian format (recommended)
ieeele_4 4 4 8 IEEE little_endian format
ieee_8 8 8 16 double precision IEEE big_endian format
        (on some machines possible)
crayxmp_8 8 8 16 CRAY internal data format
idl 4 4 8 IDL format (but IDL also supports ieee_4)
xdr 4 4 8 format possible with IDL
ieee_4_limit 4 8 8 standard IEEE format $\rightarrow$ ieee_4
ieee ? ? ? IEEE format, unknown length (not recommended)

Some attention has to be paid if weird compiler switches (as e.g.tex2html_verb_mark>971) are used to modify the accuracy and standard memory size of variables.

If an existing file is opened for reading, the file format and conversion type are determined automatically from the file if the conversion type of the data in the file is among the conversion types supported by the compiler. If the file has a conversion type ``native'' but is created on a machine with different internal data representation, the file header might be readable, but an error will probably occur during the reading of a real variable. Data file structure

The UIO routines only handle sequential files. Each file consists of a list of entries. The first entry describes the file format, conversion type, and the machine who is responsible for it. The following entries contain data (scalars and 1D ...4D arrays of type integer, real (single & double precision), complex (single precision), character; tables with columns of type integer, real (single precision), or character), or structuring information (labels).

Each entry consists of the header and the (possibly empty) data block.

Each header is a list of at most 20 terms separated by blanks or linefeeds. The first term is the entry type (e.g.tex2html_verb_mark>972, see table 6.3), followed by an identifier. This identifier should follow the standard rules for variables (lowercase letters, numbers, underscore; starting with letter). It is a name as e.g.tex2html_verb_mark>973, v_1. The rest of the terms come in the form ``keyword=value''. See Tab. 6.4 for some pre-defined keywords.

Table 6.3: UIO entry types
entry type entry contents
fileform file description (first entry)
integer scalars, 1D ...4D arrays
real scalars, 1D ...4D arrays, single & double precision
complex scalars, 1D ...4D arrays, single precision
character scalars, 1D ...4D arrays
table table with integer, real, character columns
label label entry for file structuring

Table 6.4: Standard UIO entry header keywords: The keyword is given with a short description and an example. The fourth, fifth, and sixth column indicate if the keyword is a mandatory descriptor (in the file header or for the format of an entry) or if it gives only additional information and is optional and therefore not necessary to specify.
keyword description example descriptor info.#35022#> necessary
b byte number 4 format   yes
d dimension (0:9) format   yes (arrays)
ds dimension shift (1:1)   yes  
f Fortran format E13.6 format   yes
p values per line 4 format   yes (arrays)
t transformation log10 format    
n name density   yes  
u unit g/cm^3   yes  
date date 1.1.98   yes  
c0...9 comment Dichte   yes  
form file format formatted file   yes (file header)
convert conversion ieee_4 file   yes (file header)
version version 0.1.1997.11.29   yes  
system system IRIX   yes  
machine machine name atlas   yes  
osrelease OS release 6.3   yes  
osversion OS version 12161207   yes  
hardware machine hardware IP32   yes  
language program. language Fortran90   yes  
program program uiotst   yes  
xyz... user defined source   yes  

A header line has a maximum length of 80 characters. A continuation line is indicated by & at the end of the line. A header consists of 20 lines at maximum. It can be preceded by empty lines (except for the file header entry). Example:

real time f=F9.2 b=4 n='Time' u=s c0='Simulation time in seconds' &
  c1='Time count starts at 0.0'

The entry header is followed by the entry data block. This block is empty for labels and the fileform entry but non-empty otherwise.

In an unformatted file each header line is an individual record containing a string with exactly 80 characters. The following data block (scalar or array) is one single record.

In a formatted file each header line is a string of at most 80 characters delimited by a LINEFEED of whatever the operating system decided to be appropriate as EOL character (sequence). The following data block is written as sequence of lines. The number of items per line is specified by the ``p=?'' keyword in the header. Tables

For a table the entry header is followed by a list of headers for the individual table columns, a single table header line consisting of (abbreviations of) the table entry identifiers and the table itself (see the example in section 6.2.2). The dimension keyword gives the number of columns and rows in the form ``d=(1:columns,1:rows)''.

The Holweger-Müller-Atmosphere is chosen as the following ``real world'' example for a file with a table in UIO format:

fileform uio form=formatted convert=ieee_4 version=0.0.1996.10.29 &
  date='20-Feb-1997 18:40:45' system=SunOS machine=saturn osrelease=4.1.3 &
  osversion=3 hardware=sun4m language='IDL 4.0' program='by hand'

character description d=(0:3) f=A80 p=1 b=80 d='13-Nov-1996 18:29:48'
Hartmut Holweger & Edith Mueller (1974) Solar Physics 39, 19-30, table II,
empirical solar temperature stratification to fit solar spectral lines and
limb darkening

character history d=(0:3) f=A80 p=1 b=80 d='13-Nov-1996 18:29:52'
Holweger-Mueller-Atmosphere, from 1974
uio-form:              13-Nov-1996 18:29:52
conversion type added: 20-Feb-1997 18:21:01
xi -> vmicro:          20-Feb-1997 18:23:43

real teff f=F6.1 b=4 n='effective temperature' u=K texa='T_

table atmosphere d=(1:7,1:29) f=X b=1 n=Holweger-Mueller-Atmosphere &
  c0='Hartmut Holweger & Edith Mueller (1974) Solar Physics 39, 19-30, table II' &
real tauross f=E9.2 b=4 n='optical depth (Rosseland)' u=1
real tau5000 f=F7.3 b=4 n='optical depth (lambda5000)' u=1 t=log10
real t       f=F7.0 b=4 n=temperature u=K
real pgas    f=F6.3 b=4 n='gas pressure' u=dyn/cm^2
real pel     f=F6.3 b=4 n='electron pressure' u=dyn/cm^2
real vmicro  f=F4.2 b=4 n=microturbulence u=km/s
real q       f=F8.5 b=4 n='Hopf function' u=1 c0='q=((T(tau)/Teff)^4)/0.75)-tau'
  tauross tau5000       t   pgas    pel vmic        q
 2.00E-07  -6.539   3900.  0.769 -3.140 0.00  0.27637
 2.50E-07  -6.279   3920.  1.171 -2.752 0.00  0.28208
 5.00E-07  -5.868   3970.  1.598 -2.342 0.00  0.29675
 1.00E-06  -5.588   4030.  1.842 -2.105 0.00  0.31510
 2.00E-06  -5.334   4080.  2.042 -1.910 0.00  0.33103
 5.00E-06  -5.001   4160.  2.279 -1.674 0.00  0.35776
 1.00E-05  -4.747   4210.  2.450 -1.508 0.00  0.37527
 2.00E-05  -4.486   4270.  2.618 -1.341 0.00  0.39712
 5.00E-05  -4.131   4340.  2.835 -1.128 0.00  0.42377
 1.00E-04  -3.856   4400.  2.999 -0.967 0.50  0.44765
 2.00E-04  -3.577   4460.  3.162 -0.804 0.50  0.47248
 5.00E-04  -3.200   4530.  3.377 -0.596 0.50  0.50256
 1.00E-03  -2.912   4590.  3.541 -0.437 0.50  0.52925
 2.00E-03  -2.621   4640.  3.704 -0.279 0.50  0.55173
 5.00E-03  -2.233   4720.  3.919 -0.070 0.50  0.58792
 1.00E-02  -1.939   4800.  4.083  0.094 0.50  0.62415
 2.00E-02  -1.645   4900.  4.245  0.266 0.65  0.66867
 5.00E-02  -1.256   5080.  4.460  0.504 0.85  0.74558
 1.00E-01  -0.961   5260.  4.622  0.705 1.00  0.81447
 2.50E-01  -0.571   5560.  4.830  1.002 1.25  0.89163
 4.00E-01  -0.371   5850.  4.926  1.251 1.40  0.99911
 7.00E-01  -0.133   6260.  5.022  1.613 1.50  1.13453
 1.00E+00   0.019   6570.  5.070  1.879 1.60  1.22581
 1.50E+00   0.191   6880.  5.114  2.140 1.70  1.17659
 2.00E+00   0.312   7160.  5.140  2.363 1.80  1.13964
 4.00E+00   0.597   7920.  5.191  2.889 1.80  0.70033
 6.00E+00   0.761   8250.  5.213  3.092 1.80 -0.46595
 8.00E+00   0.877   8420.  5.229  3.196 1.80 -1.99552
 1.00E+01   0.967   8500.  5.242  3.245 1.80 -3.76404 Recommendations for standard file structure

The very first entry in an UIO file is always the fileform uio entry, containing information about the file format and conversion type. Afterwards, entries can follow in any order. But it is perhaps a good idea to start the file with three special entries (file_id, description, history) as in

fileform uio form=formatted convert=ieee_4 ...             

character file_id f=A80 b=80 n='File identification'                            

character description d=(1:2) f=A80 p=1 b=80 n='File description'               
This is a file to demonstrate the recommended start entries for all
UIO files.

character history d=(1:1) f=A80 p=1 b=80 n='File history'
UIO demo file: 22-Dec-1997 14:15:15

A recommended format for sets of multi-dimensional arrays (e.g. hydrodynamics: x-axis, y-axis, z-axis, density, velocities, energy, ...) is shown in Sect.8.1.

next up previous contents index
Next: 6.2.4 Files & directories Up: 6.2 UIO data format Previous: 6.2.2 Example of UIO   Contents   Index