用NumPy genfromtxt导入数据

用NumPy genfromtxt导入数据

NumPy provides several functions to create arrays from tabular data. We focus here on the genfromtxt function.

In a nutshell, genfromtxt runs two main loops. The first loop converts each line of the file in a sequence of strings. The second loop converts each string to the appropriate data type. This mechanism is slower than a single loop, but gives more flexibility. In particular, genfromtxt is able to take missing data into account, when other faster and simpler functions like loadtxt cannot.

NumPy提供了几种从表格数据创建数组的功能。这里专注genfromtxt功能。

genfromtxt运行两个主循环。第一个循环以字符串序列转换文件的每一行。第二个循环将每个字符串转换为适当的数据类型。这种机制比单循环慢，但具有更大的灵活性。特别是，当其他更快，更简单的功能（如loadtxt不能）无法处理时， genfromtxt能够考虑丢失的数据。

Note

When giving examples, we will use the following conventions: 在给出示例时，将使用以下约定：

>>> import numpy as np

>>> from io import StringIO

Defining the input

The only mandatory argument of genfromtxt is the source of the data. It can be a string, a list of strings, a generator or an open file-like object with a read method, for example, a file or io.StringIO object. If a single string is provided, it is assumed to be the name of a local or remote file. If a list of strings or a generator returning strings is provided, each string is treated as one line in a file. When the URL of a remote file is passed, the file is automatically downloaded to the current directory and opened.

Recognized file types are text files and archives. Currently, the function recognizes gzip and bz2 (bzip2) archives. The type of the archive is determined from the extension of the file: if the filename ends with '.gz', a gzip archive is expected; if it ends with 'bz2', a bzip2 archive is assumed.

唯一强制性参数genfromtxt是数据源。可以是字符串，字符串列表，生成器或带有read方法的打开的类似文件的对象，例如文件或 io.StringIO对象。如果提供单个字符串，假定是本地文件或远程文件的名称。如果提供了字符串列表或返回字符串的生成器，将每个字符串视为文件中的一行。传递远程文件的URL后，该文件将自动下载到当前目录并打开。

公认的文件类型是文本文件和存档。当前，该功能可识别gzip和bz2（bzip2）存档。存档的类型由文件的扩展名决定：如果文件名以'.gz'结尾，则应使用gzip存档；否则，将使用默认的存档。如果结尾为 'bz2'，bzip2则假定为存档。

Splitting the lines into columns

将行拆分为列

The delimiter argument

delimiter参数

Once the file is defined and open for reading, genfromtxt splits each non-empty line into a sequence of strings. Empty or commented lines are just skipped. The delimiter keyword is used to define how the splitting should take place.

Quite often, a single character marks the separation between columns. For example, comma-separated files (CSV) use a comma (,) or a semicolon (;) as delimiter:

定义文件并打开以供读取后，genfromtxt 将每条非空行拆分为一系列字符串。空行或注释行被跳过。关键字delimiter用来定义分割应该如何发生。

通常，单个字符标记列之间的分隔。例如，逗号分隔文件（CSV）使用逗号（,）或分号（;）作为分隔符：

>>> data = u"1, 2, 3 4, 5, 6"

>>> np.genfromtxt(StringIO(data), delimiter=",")

array([[ 1.,  2.,  3.],

       [ 4.,  5.,  6.]])

Another common separator is " ", the tabulation character. However, we are not limited to a single character, any string will do. By default, genfromtxt assumes delimiter=None, meaning that the line is split along white spaces (including tabs) and that consecutive white spaces are considered as a single white space.

Alternatively, we may be dealing with a fixed-width file, where columns are defined as a given number of characters. In that case, we need to set delimiter to a single integer (if all the columns have the same size) or to a sequence of integers (if columns can have different sizes):

另一个常见的分隔符是" "制表符。不仅限于单个字符，任何字符串都可以。默认情况下， genfromtxt假设delimiter=None，则表示该行沿空白（包括制表符）分隔，连续的空白被视为单个空白。

可能要处理一个固定宽度的文件，将列定义为给定数量的字符。在这种情况下，需要设置 delimiter为单个整数（如果所有列的大小都相同）或整数序列（如果列的大小可以不同）：

>>> data = u"  1  2  3   4  5 67 890123  4"

>>> np.genfromtxt(StringIO(data), delimiter=3)

array([[   1.,    2.,    3.],

       [   4.,    5.,   67.],

       [ 890.,  123.,    4.]])

>>> data = u"123456789    4  7 9    4567 9"

>>> np.genfromtxt(StringIO(data), delimiter=(4, 3, 2))

array([[ 1234.,   567.,    89.],

       [    4.,     7.,     9.],

       [    4.,   567.,     9.]])

The autostrip argument

By default, when a line is decomposed into a series of strings, the individual entries are not stripped of leading nor trailing white spaces. This behavior can be overwritten by setting the optional argument autostrip to a value of True:

autostrip参数

默认情况下，当将一行分解为一系列字符串时，不会删除各个条目的前导或尾随空格。通过将可选参数autostrip设置为值True，可以覆盖此行为 ：

>>> data = u"1, abc , 2 3, xxx, 4"

>>> # Without autostrip

>>> np.genfromtxt(StringIO(data), delimiter=",", dtype="|U5")

array([['1', ' abc ', ' 2'],

       ['3', ' xxx', ' 4']], dtype='<U5')

>>> # With autostrip

>>> np.genfromtxt(StringIO(data), delimiter=",", dtype="|U5", autostrip=True)

array([['1', 'abc', '2'],

       ['3', 'xxx', '4']], dtype='<U5')

The comments argument

The optional argument comments is used to define a character string that marks the beginning of a comment. By default, genfromtxt assumes comments='#'. The comment marker may occur anywhere on the line. Any character present after the comment marker(s) is simply ignored:

comments参数

可选参数comments用于定义标记注释开始的字符串。默认情况下， genfromtxt假设为comments='#'。注释标记可以出现在行中的任何位置。注释标记后面的任何字符都将被忽略：

>>> data = u"""#

... # Skip me !

... # Skip me too !

... 1, 2

... 3, 4

... 5, 6 #This is the third line of the data

... 7, 8

... # And here comes the last line

... 9, 0

... """

>>> np.genfromtxt(StringIO(data), comments="#", delimiter=",")

array([[1., 2.],

       [3., 4.],

       [5., 6.],

       [7., 8.],

       [9., 0.]])

New in version 1.7.0: When comments is set to None, no lines are treated as comments.在1.7.0版本的新功能：当comments设置为None，没有行被视为注释。

Note

There is one notable exception to this behavior: if the optional argument names=True, the first commented line will be examined for names. 此行为有一个明显的例外：如果可选参数 names=True，则将检查第一条注释行的名称。

Skipping lines and choosing columns

The skip_header and skip_footer arguments

The presence of a header in the file can hinder data processing. In that case, we need to use the skip_header optional argument. The values of this argument must be an integer which corresponds to the number of lines to skip at the beginning of the file, before any other action is performed. Similarly, we can skip the last n lines of the file by using the skip_footer attribute and giving it a value of n:

跳过行并选择列

skip_header和skip_footer参数

文件中标头的存在会阻碍数据处理。在这种情况下，需要使用skip_header可选参数。此参数的值必须是整数，该整数与执行任何其它操作之前，在文件开头要跳过的行数相对应。可以n通过使用skip_footer属性，将其值设置为n，跳过文件的最后几行n：

>>> data = u" ".join(str(i) for i in range(10))

>>> np.genfromtxt(StringIO(data),)

array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

>>> np.genfromtxt(StringIO(data),

...               skip_header=3, skip_footer=5)

array([ 3.,  4.])

By default, skip_header=0 and skip_footer=0, meaning that no lines are skipped.

The usecols argument

In some cases, we are not interested in all the columns of the data but only a few of them. We can select which columns to import with the usecols argument. This argument accepts a single integer or a sequence of integers corresponding to the indices of the columns to import. Remember that by convention, the first column has an index of 0. Negative integers behave the same as regular Python negative indexes.

For example, if we want to import only the first and the last columns, we can use usecols=(0, -1):

默认情况下，skip_header=0和skip_footer=0，表示不跳过任何行。

usecols参数

在某些情况下，对数据的所有列都不感兴趣，但仅对其中的一些感兴趣。可以选择使用usecols参数导入的列 。此参数接受与要导入的列的索引相对应的单个整数或整数序列。按照惯例，第一列的索引为0。负整数的行为与常规Python负索引相同。

例如，如果只想导入第一列和最后一列，则可以使用：usecols=(0, -1)

>>> data = u"1 2 3 4 5 6"

>>> np.genfromtxt(StringIO(data), usecols=(0, -1))

array([[ 1.,  3.],

       [ 4.,  6.]])

If the columns have names, we can also select which columns to import by giving their name to the usecols argument, either as a sequence of strings or a comma-separated string: 如果这些列具有名称，可以通过将其名称作为usecols参数，以字符串序列或逗号分隔的字符串作为参数来选择要导入的列：

>>> data = u"1 2 3 4 5 6"

>>> np.genfromtxt(StringIO(data),

...               names="a, b, c", usecols=("a", "c"))

array([(1.0, 3.0), (4.0, 6.0)],

      dtype=[('a', '<f8'), ('c', '<f8')])

>>> np.genfromtxt(StringIO(data),

...               names="a, b, c", usecols=("a, c"))

    array([(1.0, 3.0), (4.0, 6.0)],

          dtype=[('a', '<f8'), ('c', '<f8')])

Choosing the data type

The main way to control how the sequences of strings we have read from the file are converted to other types is to set the dtype argument. Acceptable values for this argument are:

• a single type, such as dtype=float. The output will be 2D with the given dtype, unless a name has been associated with each column with the use of the names argument (see below). Note that dtype=float is the default for genfromtxt.
• a sequence of types, such as dtype=(int, float, float).
• a comma-separated string, such as dtype="i4,f8,|U3".
• a dictionary with two keys 'names' and 'formats'.
• a sequence of tuples (name, type), such as dtype=[('A', int), ('B', float)].
• an existing numpy.dtype object.
• the special value None. In that case, the type of the columns will be determined from the data itself (see below).

In all the cases but the first one, the output will be a 1D array with a structured dtype. This dtype has as many fields as items in the sequence. The field names are defined with the names keyword.

When dtype=None, the type of each column is determined iteratively from its data. We start by checking whether a string can be converted to a boolean (that is, if the string matches true or false in lower cases); then whether it can be converted to an integer, then to a float, then to a complex and eventually to a string. This behavior may be changed by modifying the default mapper of the StringConverter class.

The option dtype=None is provided for convenience. However, it is significantly slower than setting the dtype explicitly.

选择数据类型

控制从文件中读取的字符串序列，如何转换为其它类型的主要方法是，设置dtype参数。此参数可接受的值为：

• 单一类型，例如dtype=float。输出将是具有给定dtype的2D，除非使用names参数，将名称与每个列相关联（请参见下文）。dtype=float是genfromtxt的默认设置 。
• 类型序列，例如。dtype=(int, float, float)
• 逗号分隔的字符串，例如dtype="i4,f8,|U3"。
• 有两个键'names'和'formats'的字典。
• 一组元组，如 ，(name, type)dtype=[('A', int), ('B', float)]
• 现有numpy.dtype对象。
• 特殊值None。列的类型将由数据本身确定（请参见下文）。

在除第一种情况以外的所有情况下，输出都是具有结构化dtype的一维数组。此dtype具有与序列中的条目一样多的字段。字段名称是用names关键字定义的。

如果dtype=None，根据数据迭代确定每个列的类型。首先检查一个字符串是否可以转换为布尔值（即，如果字符串匹配true或false小写）；否则，转换为布尔值。是否可以将其转换为整数，转换为浮点数，转换为复数，最后转换为字符串。通过修改类的默认映射，可以更改此行为 StringConverter。

dtype=None提供此选项是为了方便。但是，比dtype显式设置慢得多。

设置名称

Setting the names

The names argument

A natural approach when dealing with tabular data is to allocate a name to each column. A first possibility is to use an explicit structured dtype, as mentioned previously:

names参数

处理表格数据时，一种自然的方法是为每个列分配一个名称。如前所述，第一种可能性是使用显式结构化dtype：

>>> data = StringIO("1 2 3 4 5 6")

>>> np.genfromtxt(data, dtype=[(_, int) for _ in "abc"])

array([(1, 2, 3), (4, 5, 6)],

      dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')])

Another simpler possibility is to use the names keyword with a sequence of strings or a comma-separated string: 另一个更简单的可能性是，将names关键字与字符串序列，或逗号分隔的字符串，一起使用：

>>> data = StringIO("1 2 3 4 5 6")

>>> np.genfromtxt(data, names="A, B, C")

array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)],

      dtype=[('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

In the example above, we used the fact that by default, dtype=float. By giving a sequence of names, we are forcing the output to a structured dtype.

We may sometimes need to define the column names from the data itself. In that case, we must use the names keyword with a value of True. The names will then be read from the first line (after the skip_header ones), even if the line is commented out:

在上面的示例中，使用了默认情况下的事实dtype=float。通过提供一系列名称，将输出强制为结构化dtype。

有时可能需要根据数据本身定义列名称。在这种情况下，必须使用names值为 True的关键字。即使从第一行中删除了注释，也将从第一行中读取名称 skip_header：

>>> data = StringIO("So it goes #a b c 1 2 3 4 5 6")

>>> np.genfromtxt(data, skip_header=1, names=True)

array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)],

      dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])

The default value of names is None. If we give any other value to the keyword, the new names will overwrite the field names we may have defined with the dtype: 默认值names是None。如果给关键字赋予其它任何值，则新名称将覆盖可能已经用dtype定义的字段名称：

>>> data = StringIO("1 2 3 4 5 6")

>>> ndtype=[('a',int), ('b', float), ('c', int)]

>>> names = ["A", "B", "C"]

>>> np.genfromtxt(data, names=names, dtype=ndtype)

array([(1, 2.0, 3), (4, 5.0, 6)],

      dtype=[('A', '<i8'), ('B', '<f8'), ('C', '<i8')])

The defaultfmt argument

If names=None but a structured dtype is expected, names are defined with the standard NumPy default of "f%i", yielding names like f0, f1 and so forth:

如果names=None，只希望使用结构化dtype，则使用标准NumPy默认值定义"f%i"，产生f0，f1类似的名称， 依此类推：

>>> data = StringIO("1 2 3 4 5 6")

>>> np.genfromtxt(data, dtype=(int, float, int))

array([(1, 2.0, 3), (4, 5.0, 6)],

      dtype=[('f0', '<i8'), ('f1', '<f8'), ('f2', '<i8')])

In the same way, if we don’t give enough names to match the length of the dtype, the missing names will be defined with this default template: 同样，如果没有提供足够的名称来匹配dtype的长度，则会使用此默认模板来定义缺少的名称：

>>> data = StringIO("1 2 3 4 5 6")

>>> np.genfromtxt(data, dtype=(int, float, int), names="a")

array([(1, 2.0, 3), (4, 5.0, 6)],

      dtype=[('a', '<i8'), ('f0', '<f8'), ('f1', '<i8')])

We can overwrite this default with the defaultfmt argument, that takes any format string: 可以使用defaultfmt任何格式字符串的参数，覆盖此默认值：

>>> data = StringIO("1 2 3 4 5 6")

>>> np.genfromtxt(data, dtype=(int, float, int), defaultfmt="var_%02i")

array([(1, 2.0, 3), (4, 5.0, 6)],

      dtype=[('var_00', '<i8'), ('var_01', '<f8'), ('var_02', '<i8')])

Note

We need to keep in mind that defaultfmt is used only if some names are expected but not defined. 需要记住，defaultfmt仅当需要某些名称，但未定义某些名称时，才使用。

 

Validating names

NumPy arrays with a structured dtype can also be viewed as recarray, where a field can be accessed as if it were an attribute. For that reason, we may need to make sure that the field name doesn’t contain any space or invalid character, or that it does not correspond to the name of a standard attribute (like size or shape), which would confuse the interpreter. genfromtxt accepts three optional arguments that provide a finer control on the names:

验证名称

具有结构化dtype的NumPy数组，可以视为recarray，在其中可以像对待 字段一样，访问字段。可能需要确保字段名称，不包含任何空格或无效字符，或者不与标准属性（如size或 shape）的名称相对应，这会使解释器感到困惑。 genfromtxt 接受三个可选参数，对名称提供了更好的控制：

deletechars

Gives a string combining all the characters that must be deleted from the name. By default, invalid characters are ~!@#$%^&*()-=+~|]}[{';: /?.>,<.

excludelist

Gives a list of the names to exclude, such as return, file, print… If one of the input name is part of this list, an underscore character ('_') will be appended to it.

case_sensitive

Whether the names should be case-sensitive (case_sensitive=True), converted to upper case (case_sensitive=False or case_sensitive='upper') or to lower case (case_sensitive='lower').

Tweaking the conversion

The converters argument

Usually, defining a dtype is sufficient to define how the sequence of strings must be converted. However, some additional control may sometimes be required. For example, we may want to make sure that a date in a format YYYY/MM/DD is converted to a datetime object, or that a string like xx% is properly converted to a float between 0 and 1. In such cases, we should define conversion functions with the converters arguments.

The value of this argument is typically a dictionary with column indices or column names as keys and a conversion functions as values. These conversion functions can either be actual functions or lambda functions. In any case, they should accept only a string as input and output only a single element of the wanted type.

In the following example, the second column is converted from as string representing a percentage to a float between 0 and 1:

converters参数

通常，dtype足以定义必须如何转换字符串序列。有时可能需要一些其它控制。例如，可能要确保将格式中的日期 YYYY/MM/DD转换为datetime对象，或者将类似的字符串xx%正确转换为0到1之间的浮点数。在这种情况下，应该使用converters 参数定义转换函数。

该参数的值通常是一个字典，其中以列索引或列名作为键，而转换函数作为值。这些转换函数可以是实际函数，也可以是lambda函数。在任何情况下，都应仅接受字符串作为输入，并仅输出所需类型的单个元素。

在下面的示例中，第二列从表示百分比的字符串转换为0到1之间的浮点数。

>>> convertfunc = lambda x: float(x.strip(b"%"))/100.

>>> data = u"1, 2.3%, 45. 6, 78.9%, 0"

>>> names = ("i", "p", "n")

>>> # General case .....

>>> np.genfromtxt(StringIO(data), delimiter=",", names=names)

array([(1., nan, 45.), (6., nan, 0.)],

      dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')])

We need to keep in mind that by default, dtype=float. A float is therefore expected for the second column. However, the strings ' 2.3%' and ' 78.9%' cannot be converted to float and we end up having np.nan instead. Let’s now use a converter:

>>> # Converted case ...

>>> np.genfromtxt(StringIO(data), delimiter=",", names=names,

...               converters={1: convertfunc})

array([(1.0, 0.023, 45.0), (6.0, 0.78900000000000003, 0.0)],

      dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')])

The same results can be obtained by using the name of the second column ("p") as key instead of its index (1):

>>> # Using a name for the converter ...

>>> np.genfromtxt(StringIO(data), delimiter=",", names=names,

...               converters={"p": convertfunc})

array([(1.0, 0.023, 45.0), (6.0, 0.78900000000000003, 0.0)],

      dtype=[('i', '<f8'), ('p', '<f8'), ('n', '<f8')])

Converters can also be used to provide a default for missing entries. In the following example, the converter convert transforms a stripped string into the corresponding float or into -999 if the string is empty. We need to explicitly strip the string from white spaces as it is not done by default:

>>> data = u"1, , 3 4, 5, 6"

>>> convert = lambda x: float(x.strip() or -999)

>>> np.genfromtxt(StringIO(data), delimiter=",",

...               converters={1: convert})

array([[   1., -999.,    3.],

       [   4.,    5.,    6.]])

Using missing and filling values

Some entries may be missing in the dataset we are trying to import. In a previous example, we used a converter to transform an empty string into a float. However, user-defined converters may rapidly become cumbersome to manage.

The genfromtxt function provides two other complementary mechanisms: the missing_values argument is used to recognize missing data and a second argument, filling_values, is used to process these missing data.

missing_values

By default, any empty string is marked as missing. We can also consider more complex strings, such as "N/A" or "???" to represent missing or invalid data. The missing_values argument accepts three kind of values:

a string or a comma-separated string

This string will be used as the marker for missing data for all the columns

a sequence of strings

In that case, each item is associated to a column, in order.

a dictionary

Values of the dictionary are strings or sequence of strings. The corresponding keys can be column indices (integers) or column names (strings). In addition, the special key None can be used to define a default applicable to all columns.

filling_values

We know how to recognize missing data, but we still need to provide a value for these missing entries. By default, this value is determined from the expected dtype according to this table:

Expected type	Default
bool	False
int	-1
float	np.nan
complex	np.nan+0j
string	'???'

We can get a finer control on the conversion of missing values with the filling_values optional argument. Like missing_values, this argument accepts different kind of values:

a single value

This will be the default for all columns

a sequence of values

Each entry will be the default for the corresponding column

a dictionary

Each key can be a column index or a column name, and the corresponding value should be a single object. We can use the special key None to define a default for all columns.

In the following example, we suppose that the missing values are flagged with "N/A" in the first column and by "???" in the third column. We wish to transform these missing values to 0 if they occur in the first and second column, and to -999 if they occur in the last column:

>>> data = u"N/A, 2, 3 4, ,???"

>>> kwargs = dict(delimiter=",",

...               dtype=int,

...               names="a,b,c",

...               missing_values={0:"N/A", 'b':" ", 2:"???"},

...               filling_values={0:0, 'b':0, 2:-999})

>>> np.genfromtxt(StringIO(data), **kwargs)

array([(0, 2, 3), (4, 0, -999)],

      dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')])

usemask

We may also want to keep track of the occurrence of missing data by constructing a boolean mask, with True entries where data was missing and False otherwise. To do that, we just have to set the optional argument usemask to True (the default is False). The output array will then be a MaskedArray.

Shortcut functions

In addition to genfromtxt, the numpy.lib.npyio module provides several convenience functions derived from genfromtxt. These functions work the same way as the original, but they have different default values.

recfromtxt

Returns a standard numpy.recarray (if usemask=False) or a MaskedRecords array (if usemaske=True). The default dtype is dtype=None, meaning that the types of each column will be automatically determined.

recfromcsv

Like recfromtxt, but with a default delimiter=",".