Commit 7ab4244b authored by Jan Kotanski's avatar Jan Kotanski
Browse files

New upstream version 2.10.0

parent 366281a4
All of the people who have made at least one contribution to h5py.
Authors are sorted by number of commits.
* Andrew Collette
* Thomas A Caswell
* James Tocknell
* Thomas Kluyver
* Андрей Парамонов
* Darren Dale
* Aleksandar Jelenak
* Valentin Valls
* Ulrik Kofoed Pedersen
* Martin Raspaud
* Anthony Scopatz
* John Kirkham
* Aaron Parsons
* Lawrence Chan
* Andrey Paramonov
* Liu JiaLin
* lhole
* Ghislain Antony Vaillant
* Yu Feng
* Seth R Johnson
* Matthieu Brucher
* Andrea Bedini
* Martin Teichmann
* Stan West
* Axel Huebl
* Pierre de Buyl
* Florian Rathgeber
* Chris Billington
* Thomas VINCENT
* Max Dietz
* Konrad Hinsen
* Toon Verstraelen
* Jerome Kieffer
* Markus Gerstel
* Machine User
* Geoff Wright
* Kit Choi
* lucasb-eyer
* Matthew Brett
* Dylan Nelson
* Scott Sanderson
* Antony Lee
* Matt Zwier
* Charles Law
* Chen Yufei
* Simon Mutch
* Christian Sachs
* ebner
* Sajid Ali
* Noel Dawe
* John Tyree
* Christoph Gohlke
* Carlos Pascual
* Joe Jevnik
* unknown
* Jason Newton
* Will Parkin
* Devin
* Nathan Goldbaum
* paulmueller
* Hameer Abbasi
* Andy Salnikov
* amcnicho
* Jonah Bernhard
* Antoine Pitrou
* Joseph Kleinhenz
* cclauss
* Colin Gavin
* Chris Meyer
* Jakob Lombacher
* Michael Boyle
* Sam Mason
* Peter H. Li
* jialin
* joydeep bhattacharjee
* pharshalp
* Matthias Geier
* Eric Larson
* Drew Parsons
* Robert David Grant
* JP Dehollain
* chrisjbillington
* Lars Viklund
* Dan Guest
* root
* bpinsard
* Barry Wardel
* Evan Wright
* Johannes Meixner
* Marc Abramowitz
* syhw
* Stuart Berg
* Michael V. DePalatis
* Kwat
* Mike Boyle
* Peter Chang
* Mikhail
* Nils Werner
* Peter Hill
* John Benediktsson
* Yuri D'Elia
* Matthew Turk
* Bradley M. Froehle
* Dan Meliza
* Artem Sanakoev
* Matthias König
* Adam J. Stewart
* Kyle Sunden
* Caleb Morse
* Peter Colberg
* James Clarke
* Sam Toyer
* Joe Zuntz
* Greg Allen
* Lawrence Mitchell
* dkriegner
* Jens Timmerman
* Niru Maheswaranathan
* Paco Hope
* Felix Yan
Copyright (c) 2008 Andrew Collette and contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
include ANN.rst
include AUTHORS
include api_gen.py
include LICENSE
include MANIFEST.in
include pylintrc
include README.rst
include setup_build.py
include setup_configure.py
include tox.ini
include pytest.ini
recursive-include docs *
prune docs/_build
......@@ -33,3 +36,8 @@ exclude *.yml
exclude *.yaml
recursive-exclude * __pycache__
recursive-exclude * *.py[co]
exclude .coveragerc
exclude .mailmap
exclude rever.xsh
prune news
Metadata-Version: 1.1
Metadata-Version: 1.2
Name: h5py
Version: 2.9.0
Version: 2.10.0
Summary: Read and write HDF5 files from Python
Home-page: http://www.h5py.org
Author: Andrew Collette
Author-email: andrew.collette@gmail.com
Maintainer: Andrew Collette
Maintainer-email: andrew.collette@gmail.com
License: BSD
Download-URL: https://pypi.python.org/pypi/h5py
Description:
......
......@@ -78,9 +78,14 @@ when building from source.
Source installation
-------------------
To install h5py from source, you need three things installed:
* A supported Python version with development headers
* HDF5 1.8.4 or newer with development headers
* A C compiler
On Unix platforms, you also need ``pkg-config`` unless you explicitly specify
a path for HDF5 as described in :ref:`custom_install`.
OS-specific instructions for installing HDF5, Python and a C compiler are in the next few
sections.
......
......@@ -58,10 +58,10 @@ copyright = u'2014, Andrew Collette and contributors'
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '2.9'
# The full version, including alpha/beta/rc tags.
release = '2.9.0'
release = '2.10.0'
# The short X.Y version.
version = '.'.join(release.split('.')[:2])
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
......@@ -116,20 +116,16 @@ Then, clone your new copy of h5py to your local machine::
Create a topic branch for your feature
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you're fixing a bug, you'll want to check out a branch against the
appropriate stable branch. For example, to fix a bug you found in version
2.1.3, you'll want to check out against branch "2.1"::
$ git checkout -b bugfix 2.1
If you're contributing a new feature, it's appropriate to develop against the
"master" branch, so you would instead do::
Check out a new branch for the bugfix or feature you're writing::
$ git checkout -b newfeature master
The exact name of the branch can be anything you want. For bug fixes, one
approach is to put the issue number in the branch name.
We develop all changes against the *master* branch.
If we're making a bugfix release, a bot will backport merged pull requests.
Implement the feature!
~~~~~~~~~~~~~~~~~~~~~~
......@@ -156,8 +152,9 @@ Your pull request might be accepted right away. More commonly, the maintainers
will post comments asking you to fix minor things, like add a few tests, clean
up the style to be PEP-8 compliant, etc.
The pull request page also shows whether the project builds correctly,
using Travis CI. Check to see if the build succeeded (takes about 5 minutes),
The pull request page also shows the results of building and testing the
modified code on Travis and Appveyor CI.
Check back after about 30 minutes to see if the build succeeded,
and if not, try to modify your changes to make it work.
When making changes after creating your pull request, just add commits to
......@@ -199,7 +196,7 @@ opening the various files as we work through the example.
First, get ahold of
the function signature; the easiest place for this is at the `online
HDF5 Reference Manual <http://www.hdfgroup.org/HDF5/doc/RM/RM_H5Front.html>`_.
HDF5 Reference Manual <https://support.hdfgroup.org/HDF5/doc/RM/RM_H5Front.html>`_.
Then, add the function's C signature to the file ``api_functions.txt``::
hsize_t H5Dget_storage_size(hid_t dset_id)
......
......@@ -91,6 +91,11 @@ Reference
Retrieve `name`, or `default` if no such attribute exists.
.. method:: get_id(name)
Get the low-level :class:`AttrID <low:h5py.h5a.AttrID>` for the named
attribute.
.. method:: create(name, data, shape=None, dtype=None)
Create a new attribute, with control over the shape and type. Any
......
......@@ -166,7 +166,7 @@ The ``compression_opts`` parameter will then be passed to this filter.
Scale-Offset filter
~~~~~~~~~~~~~~~~~~~
Filters enabled with the ``compression`` keywords are _lossless_; what comes
Filters enabled with the ``compression`` keywords are *lossless*; what comes
out of the dataset is exactly what you put in. HDF5 also includes a lossy
filter which trades precision for storage space.
......@@ -256,6 +256,10 @@ Broadcasting is implemented using repeated hyperslab selections, and is
safe to use with very large target selections. It is supported for the above
"simple" (integer, slice and ellipsis) slicing only.
.. warning::
Currently h5py does not support nested compound types, see :issue:`1197` for
more information.
.. _dataset_fancy:
......@@ -280,7 +284,6 @@ dataset with shape (10, 10)::
The following restrictions exist:
* List selections may not be empty
* Selection coordinates must be given in increasing order
* Duplicate selections are ignored
* Very long lists (> 1000 elements) may produce poor performance
......@@ -296,6 +299,9 @@ list of points to select, so be careful when using it with large masks::
>>> result.shape
(49,)
.. versionchanged:: 2.10
Selecting using an empty list is now allowed.
This returns an array with length 0 in the relevant dimension.
.. _dataset_iter:
......@@ -414,6 +420,24 @@ Reference
Return the size of the first axis.
.. method:: make_scale(name='')
Make this dataset an HDF5 :ref:`dimension scale <dimension_scales>`.
You can then attach it to dimensions of other datasets like this::
other_ds.dims[0].attach_scale(ds)
You can optionally pass a name to associate with this scale.
.. method:: virtual_sources
If this dataset is a :doc:`virtual dataset </vds>`, return a list of
named tuples: ``(vspace, file_name, dset_name, src_space)``,
describing which parts of the dataset map to which source datasets.
The two 'space' members are low-level
:class:`SpaceID <low:h5py.h5s.SpaceID>` objects.
.. attribute:: shape
NumPy-style shape tuple giving dataset dimensions.
......@@ -466,6 +490,16 @@ Reference
type-appropriate default value. Can't be changed after the dataset is
created.
.. attribute:: external
If this dataset is stored in one or more external files, this is a list
of 3-tuples, like the ``external=`` parameter to
:meth:`Group.create_dataset`. Otherwise, it is ``None``.
.. attribute:: is_virtual
True if this dataset is a :doc:`virtual dataset </vds>`, otherwise False.
.. attribute:: dims
Access to :ref:`dimension_scales`.
......
......@@ -4,7 +4,7 @@ Dimension Scales
================
Datasets are multidimensional arrays. HDF5 provides support for labeling the
dimensions and associating one or "dimension scales" with each dimension. A
dimensions and associating one or more "dimension scales" with each dimension. A
dimension scale is simply another HDF5 dataset. In principle, the length of the
multidimensional array along the dimension of interest should be equal to the
length of the dimension scale, but HDF5 does not enforce this property.
......@@ -36,10 +36,10 @@ We can also use HDF5 datasets as dimension scales. For example, if we have::
We are going to treat the ``x1``, ``x2``, ``y1``, and ``z1`` datasets as
dimension scales::
f['data'].dims.create_scale(f['x1'])
f['data'].dims.create_scale(f['x2'], 'x2 name')
f['data'].dims.create_scale(f['y1'], 'y1 name')
f['data'].dims.create_scale(f['z1'], 'z1 name')
f['x1'].make_scale()
f['x2'].make_scale('x2 name')
f['y1'].make_scale('y1 name')
f['z1'].make_scale('z1 name')
When you create a dimension scale, you may provide a name for that scale. In
this case, the ``x1`` scale was not given a name, but the others were. Now we
......
......@@ -57,15 +57,17 @@ of supported drivers and their options:
Buffered I/O using functions from stdio.h.
'core'
Memory-map the entire file; all operations are performed in
memory and written back out when the file is closed. Keywords:
Store and manipulate the data in memory, and optionally write it
back out when the file is closed. Using this with an existing file
and a reading mode will read the entire file into memory. Keywords:
backing_store: If True (default), save changes to a real file
when closing. If False, the file exists purely
in memory and is discarded when closed.
backing_store:
If True (default), save changes to the real file at the specified
path on :meth:`~.File.close` or :meth:`~.File.flush`.
If False, any changes are discarded when the file is closed.
block_size: Increment (in bytes) by which memory is extended.
Default is 64k.
block_size:
Increment (in bytes) by which memory is extended. Default is 64k.
'family'
Store the file on disk as a series of fixed-length chunks. Useful
......@@ -115,11 +117,11 @@ a better option may be to store temporary data on disk using the functions in
Version bounding
----------------
HDF5 has been evolving for many years now. By default, the library will
write objects in the most compatible fashion possible, so that older versions
will still be able to read files generated by modern programs. However, there
can be performance advantages if you are willing to forgo a certain level
of backwards compatibility. By using the "libver" option to File, you can
HDF5 has been evolving for many years now. By default, the library will write
objects in the most compatible fashion possible, so that older versions will
still be able to read files generated by modern programs. However, there can be
feature or performance advantages if you are willing to forgo a certain level of
backwards compatibility. By using the "libver" option to :class:`File`, you can
specify the minimum and maximum sophistication of these structures:
>>> f = h5py.File('name.hdf5', libver='earliest') # most compatible
......@@ -132,6 +134,19 @@ compatible.
The default is "earliest".
Specifying version bounds has changed from HDF5 version 1.10.2. There are two new
compatibility levels: `v108` (for HDF5 1.8) and `v110` (for HDF5 1.10). This
change enables, for example, something like this:
>>> f = h5py.File('name.hdf5', libver=('earliest', 'v108'))
which enforces full backward compatibility up to HDF5 1.8. Using any HDF5
feature that requires a newer format will raise an error.
`latest` is now an alias to another bound label that represents the latest
version. Because of this, the `File.libver` property will not use `latest` in
its output for HDF5 1.10.2 or later.
.. _file_userblock:
......
......@@ -369,6 +369,11 @@ Reference
``True``. Default is
``h5.get_config().track_order``.
:keyword external: Store the dataset in one or more external, non-HDF5
files. This should be a list of tuples of
``(filename[, offset[, size]])``, to store data from ``offset`` to
``offset + size`` in the specified file. The last file in the list
may have size ``h5py.h5s.UNLIMITED`` to let it grow as needed.
.. method:: require_dataset(name, shape=None, dtype=None, exact=None, **kwds)
......@@ -403,6 +408,18 @@ Reference
shape and dtype, in which case the provided values take precedence over
those from `other`.
.. method:: create_virtual_dataset(name, layout, fillvalue=None)
Create a new virtual dataset in this group. See :doc:`/vds` for more
details.
:param str name:
Name of the dataset (absolute or relative).
:param VirtualLayout layout:
Defines what source data fills which parts of the virtual dataset.
:param fillvalue:
The value to use where there is no data.
.. attribute:: attrs
:ref:`attributes` for this group.
......
......@@ -7,30 +7,29 @@ Copyright Notice and Statement for the h5py Project
::
Copyright (c) 2008 Andrew Collette and contributors
http://h5py.alfven.org
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
a. Redistributions of source code must retain the above copyright
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
b. Redistributions in binary form must reproduce the above copyright
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.
c. Neither the name of the author nor the names of contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
......
......@@ -77,20 +77,21 @@ are represented with the "object" dtype (kind 'O'). A small amount of
metadata attached to the dtype tells h5py to interpret the data as containing
reference objects.
H5py contains a convenience function to create these "hinted dtypes" for you:
These dtypes are available from h5py for references and region references:
>>> ref_dtype = h5py.special_dtype(ref=h5py.Reference)
>>> type(ref_dtype)
>>> type(h5py.ref_dtype)
<type 'numpy.dtype'>
>>> ref_dtype.kind
'O'
>>> type(h5py.regionref_dtype)
<type 'numpy.dtype'>
The types accepted by this "ref=" keyword argument are h5py.Reference (for
object references) and h5py.RegionReference (for region references).
To create an array of references, use this dtype as you normally would:
>>> ref_dataset = myfile.create_dataset("MyRefs", (100,), dtype=ref_dtype)
>>> ref_dataset = myfile.create_dataset("MyRefs", (100,), dtype=h5py.ref_dtype)
You can read from and write to the array as normal:
......
......@@ -16,41 +16,15 @@ store these types. Each type is represented by a native NumPy dtype, with a
small amount of metadata attached. NumPy routines ignore the metadata, but
h5py can use it to determine how to store the data.
There are two functions for creating these "hinted" dtypes:
.. function:: special_dtype(**kwds)
Create a NumPy dtype object containing type hints. Only one keyword
may be specified.
:param vlen: Base type for HDF5 variable-length datatype.
:param enum: 2-tuple ``(basetype, values_dict)``. ``basetype`` must be
an integer dtype; ``values_dict`` is a dictionary mapping
string names to integer values.
:param ref: Provide class ``h5py.Reference`` or ``h5py.RegionReference``
to create a type representing object or region references
respectively.
.. function:: check_dtype(**kwds)
Determine if the given dtype object is a special type. Example::
>>> out = h5py.check_dtype(vlen=mydtype)
>>> if out is not None:
... print "Vlen of type %s" % out
str
:param vlen: Check for an HDF5 variable-length type; returns base class
:param enum: Check for an enumerated type; returns 2-tuple ``(basetype, values_dict)``.
:param ref: Check for an HDF5 object or region reference; returns
either ``h5py.Reference`` or ``h5py.RegionReference``.
The metadata h5py attaches to dtypes is not part of the public API,
so it may change between versions.
Use the functions described below to create and check for these types.
Variable-length strings
-----------------------
.. seealso:: :ref:`strings`
In HDF5, data in VL format is stored as arbitrary-length vectors of a base
type. In particular, strings are stored C-style in null-terminated buffers.
NumPy has no native mechanism to support this. Unfortunately, this is the
......@@ -69,13 +43,43 @@ data and stored.
Here's an example showing how to create a VL array of strings::
>>> f = h5py.File('foo.hdf5')
>>> dt = h5py.special_dtype(vlen=str)
>>> dt = h5py.string_dtype(encoding='utf-8')
>>> ds = f.create_dataset('VLDS', (100,100), dtype=dt)
>>> ds.dtype.kind
'O'
>>> h5py.check_dtype(vlen=ds.dtype)
<type 'str'>
>>> h5py.check_string_dtype(ds.dtype)
string_info(encoding='utf-8', length=None)
.. function:: string_dtype(encoding='utf-8', length=None)
Make a numpy dtype for HDF5 strings
:param encoding: ``'utf-8'`` or ``'ascii'``.
:param length: ``None`` for variable-length, or an integer for fixed-length
string data, giving the length in bytes.
If ``encoding`` is ``'utf-8'``, the variable length strings should be passed as
Python ``str`` objects (``unicode`` in Python 2).
For ``'ascii'``, they should be passed as bytes.
.. function:: check_string_dtype(dt)
Check if ``dt`` is a string dtype.
Returns a *string_info* object if it is, or ``None`` if not.
.. class:: string_info
A named tuple type holding string encoding and length.
.. attribute:: encoding
The character encoding associated with the string dtype,
which can be ``'utf-8'`` or ``'ascii'``.
.. attribute:: length
For fixed-length string dtypes, the length in bytes.
``None`` for variable-length strings.
.. _vlen:
......@@ -85,7 +89,7 @@ Arbitrary vlen data
Starting with h5py 2.3, variable-length types are not restricted to strings.
For example, you can create a "ragged" array of integers::
>>> dt = h5py.special_dtype(vlen=np.dtype('int32'))
>>> dt = h5py.vlen_dtype(np.dtype('int32'))
>>> dset = f.create_dataset('vlen_int', (100,), dtype=dt)
>>> dset[0] = [1,2,3]
>>> dset[1] = [1,2,3,4,5]
......@@ -101,6 +105,16 @@ arrays::
>>> dset[0:2]
array([array([1, 2, 3], dtype=int32), array([1, 2, 3, 4, 5], dtype=int32)], dtype=object)
.. function:: vlen_dtype(basetype)
Make a numpy dtype for an HDF5 variable-length datatype.
:param basetype: The dtype of each element in the array.