如何使用基于文本的标题在 R 中读取面向表的二进制文件

我要处理的文件首先包含可变长度的文本数据，然后是二进制数据.具体地说，它是所谓的"面向表格的二进制文件".

简单地将文本数据函数作为二进制表的头，包含描述性列名以及定义列中每个值的长度的数据类型.

因为二进制数据没有分隔符，所以这个长度用于一个接一个地读取每个数据点.(更多:https://s.campbellsci.com/documents/us/manuals/loggernet34.pdfTob1小节也是下面的小示例)

我遇到了一个实现导入的旧存储库(https://github.com/mlt/csdf.git)，并从那里开始了我的方法，因为我还想读取类似但没有在那里实现的TOB3文件.

导入首先以文本模式加载文件

file.text <- file(file,"r")

要使用类似以下内容来读取元信息和标题

header <- read.csv(nrow = 4)

在读取文件时，每次都会转发字节位置.在读取头之后，使用

pos <- seek(file.text)

然后，该位置被假定为二进制数据的头的结束和开始.然后将加载二进制文件，并将其设置为以前的位置

file.bin <- file(file,"rb")
seek(file.bin,pos)

并且将使用报头中定义的每个数据类型的专用函数从该位置读取.例如，第一行来自类型ULONG，它有4个字节长，因此将读取下一个4个字节，同时自动转发位置.

一切都很容易和好……但是

由于某些原因，当使用read.csv()读取时，字节位置被转发的次数似乎超过了行的长度.

如果我读取文本文件的前4行，我将在字节位置3654结束，但是当我使用readLines()从二进制文件中读取相同的行时，它将在3152结束.似乎大多数偏移量发生在读取第一行时，并且每行的偏移量变小一个字节(参见下面的转储).通过十六进制编辑器和手动查找文件中的位置，我知道在二进制文件中开始的正确位置实际上是3152.

这种差异来自哪里？有没有其他方法可以找到二进制数据的起点？(我使用的是Windows 10)

> seek(testfile.text,0)
[1] 4096
> seek(testfile.bin,0)
[1] 95
> read.csv(testfile.text,header = FALSE,nrows = 1)
    V1    V2  V3   V4            V5                     V6    V7             V8
1 TOB1 Tower CR6 7562 CR6.Std.10.02 CPU:EasyFlux_Tower.cr6 22445 Flux_CSIFormat
> readLines(testfile.bin,n=1)
[1] "\"TOB1\",\"Tower\",\"CR6\",\"7562\",\"CR6.Std.10.02\",\"CPU:EasyFlux_Tower.cr6\",\"22445\",\"Flux_CSIFormat\""
> seek(testfile.text)
[1] 601
> seek(testfile.bin)
[1] 95
> 601-95
[1] 506
> read.csv(testfile.text,header = FALSE,nrows = 1)
       V1          V2     V3      V4    V5         V6 V7    V8         V9 V10  V11       V12
1 SECONDS NANOSECONDS RECORD FC_mass FC_QC FC_samples LE LE_QC LE_samples   H H_QC H_samples
     V13 V14 V15            V16                     V17         V18 V19    V20   V21   V22
1 NETRAD   G  SG energy_closure poor_energy_closure_flg Bowen_ratio TAU TAU_QC USTAR TSTAR
  V23      V24      V25        V26   V27       V28      V29      V30        V31 V32   V33
1 TKE TA_1_1_1 RH_1_1_1 T_DP_1_1_1 amb_e amb_e_sat TA_2_1_1 RH_2_1_1 T_DP_2_1_1   e e_sat
       V34      V35        V36     V37         V38       V39 V40 V41 V42      V43 V44
1 TA_3_1_1 RH_3_1_1 T_DP_3_1_1 e_probe e_sat_probe H2O_probe  PA VPD  Ux Ux_SIGMA  Uy
       V45 V46      V47     V48           V49           V50 V51     V52      V53      V54
1 Uy_SIGMA  Uz Uz_SIGMA T_SONIC T_SONIC_SIGMA sonic_azimuth  WS WS_RSLT WD_SONIC WD_SIGMA
  V55    V56         V57               V58         V59               V60                V61
1  WD WS_MAX CO2_density CO2_density_SIGMA H2O_density H2O_density_SIGMA CO2_sig_strgth_Min
                 V62 V63 V64   V65    V66   V67    V68      V69        V70         V71
1 H2O_sig_strgth_Min   P ALB SW_IN SW_OUT LW_IN LW_OUT T_nr_Avg LW_IN_meas LW_OUT_meas
      V72         V73           V74        V75             V76            V77     V78
1 PPFD_IN sun_azimuth sun_elevation hour_angle sun_declination air_mass_coeff daytime
       V79       V80            V81           V82            V83       V84      V85      V86
1 TS_1_1_1 SWC_1_1_1 cs65x_ec_1_1_1 G_plate_1_1_1 shfp_cal_1_1_1 FETCH_MAX FETCH_90 FETCH_55
       V87               V88            V89         V90
1 FETCH_40 UPWND_DIST_INTRST FP_DIST_INTRST FP_EQUATION
> readLines(testfile.bin,n=1)
[1] "\"SECONDS\",\"NANOSECONDS\",\"RECORD\",\"FC_mass\",\"FC_QC\",\"FC_samples\",\"LE\",\"LE_QC\",\"LE_samples\",\"H\",\"H_QC\",\"H_samples\",\"NETRAD\",\"G\",\"SG\",\"energy_closure\",\"poor_energy_closure_flg\",\"Bowen_ratio\",\"TAU\",\"TAU_QC\",\"USTAR\",\"TSTAR\",\"TKE\",\"TA_1_1_1\",\"RH_1_1_1\",\"T_DP_1_1_1\",\"amb_e\",\"amb_e_sat\",\"TA_2_1_1\",\"RH_2_1_1\",\"T_DP_2_1_1\",\"e\",\"e_sat\",\"TA_3_1_1\",\"RH_3_1_1\",\"T_DP_3_1_1\",\"e_probe\",\"e_sat_probe\",\"H2O_probe\",\"PA\",\"VPD\",\"Ux\",\"Ux_SIGMA\",\"Uy\",\"Uy_SIGMA\",\"Uz\",\"Uz_SIGMA\",\"T_SONIC\",\"T_SONIC_SIGMA\",\"sonic_azimuth\",\"WS\",\"WS_RSLT\",\"WD_SONIC\",\"WD_SIGMA\",\"WD\",\"WS_MAX\",\"CO2_density\",\"CO2_density_SIGMA\",\"H2O_density\",\"H2O_density_SIGMA\",\"CO2_sig_strgth_Min\",\"H2O_sig_strgth_Min\",\"P\",\"ALB\",\"SW_IN\",\"SW_OUT\",\"LW_IN\",\"LW_OUT\",\"T_nr_Avg\",\"LW_IN_meas\",\"LW_OUT_meas\",\"PPFD_IN\",\"sun_azimuth\",\"sun_elevation\",\"hour_angle\",\"sun_declination\",\"air_mass_coeff\",\"daytime\",\"TS_1_1_1\",\"SWC_1_1_1\",\"cs65x_ec_1_1_1\",\"G_plate_1_1_1\",\"shfp_cal_1_1_1\",\"FETCH_MAX\",\"FETCH_90\",\"FETCH_55\",\"FETCH_40\",\"UPWND_DIST_INTRST\",\"FP_DIST_INTRST\",\"FP_EQUATION\""
> seek(testfile.text)
[1] 1610
> seek(testfile.bin)
[1] 1105
> 1610-1105
[1] 505
> read.csv(testfile.text,header = FALSE,nrows = 1)
       V1          V2 V3         V4    V5      V6    V7    V8      V9   V10   V11     V12
1 SECONDS NANOSECONDS RN mg m-2 s-1 Grade samples W m-2 Grade samples W m-2 grade samples
    V13   V14   V15      V16 V17      V18        V19   V20   V21   V22    V23   V24 V25
1 W m-2 W m-2 W m-2 fraction  NA fraction kg m-1 s-2 grade m s-1 deg C m2 s-2 deg C   %
    V26 V27 V28   V29 V30   V31 V32 V33   V34 V35   V36 V37 V38   V39 V40 V41   V42   V43
1 deg C kPa kPa deg C   % deg C kPa kPa deg C   % deg C kPa kPa g/m^3 kPa hPa m s-1 m s-1
    V44   V45   V46   V47   V48   V49             V50   V51   V52             V53
1 m s-1 m s-1 m s-1 m s-1 deg C deg C Decimal degrees m s-1 m s-1 decimal degrees
              V54             V55   V56    V57    V58   V59   V60      V61      V62 V63 V64
1 decimal degrees decimal degrees m s-1 mg m-3 mg m-3 g m-3 g m-3 fraction fraction  mm   %
    V65   V66   V67   V68   V69   V70    V71                V72             V73
1 W m-2 W m-2 W m-2 W m-2 Klvin W m-2 W m-22 umolPhoton m-2 s-1 decimal degrees
              V74             V75             V76          V77      V78   V79 V80    V81
1 decimal degrees decimal degrees decimal degrees adimensional fraction deg C   % dS m-1
    V82 V83 V84 V85 V86 V87 V88 V89     V90
1 W m-2  NA   m   m   m   m   m   % authors
> readLines(testfile.bin,n=1)
[1] "\"SECONDS\",\"NANOSECONDS\",\"RN\",\"mg m-2 s-1\",\"Grade\",\"samples\",\"W m-2\",\"Grade\",\"samples\",\"W m-2\",\"grade\",\"samples\",\"W m-2\",\"W m-2\",\"W m-2\",\"fraction\",\"\",\"fraction\",\"kg m-1 s-2\",\"grade\",\"m s-1\",\"deg C\",\"m2 s-2\",\"deg C\",\"%\",\"deg C\",\"kPa\",\"kPa\",\"deg C\",\"%\",\"deg C\",\"kPa\",\"kPa\",\"deg C\",\"%\",\"deg C\",\"kPa\",\"kPa\",\"g/m^3\",\"kPa\",\"hPa\",\"m s-1\",\"m s-1\",\"m s-1\",\"m s-1\",\"m s-1\",\"m s-1\",\"deg C\",\"deg C\",\"Decimal degrees\",\"m s-1\",\"m s-1\",\"decimal degrees\",\"decimal degrees\",\"decimal degrees\",\"m s-1\",\"mg m-3\",\"mg m-3\",\"g m-3\",\"g m-3\",\"fraction\",\"fraction\",\"mm\",\"%\",\"W m-2\",\"W m-2\",\"W m-2\",\"W m-2\",\"Klvin\",\"W m-2\",\"W m-22\",\"umolPhoton m-2 s-1\",\"decimal degrees\",\"decimal degrees\",\"decimal degrees\",\"decimal degrees\",\"adimensional\",\"fraction\",\"deg C\",\"%\",\"dS m-1\",\"W m-2\",\"\",\"m\",\"m\",\"m\",\"m\",\"m\",\"%\",\"authors\""
> seek(testfile.text)
[1] 2400
> seek(testfile.bin)
[1] 1896
> 2400-1896
[1] 504
> read.csv(testfile.text,header = FALSE,nrows = 1)
  V1 V2 V3  V4  V5  V6  V7  V8  V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20 V21 V22 V23
1 NA NA NA Smp Smp Tot Smp Smp Tot Smp Smp Tot Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp
  V24 V25 V26 V27 V28 V29 V30 V31 V32 V33 V34 V35 V36 V37 V38 V39 V40 V41 V42 V43 V44 V45
1 Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp
  V46 V47 V48 V49 V50 V51 V52 V53 V54 V55 V56 V57 V58 V59 V60 V61 V62 V63 V64 V65 V66 V67
1 Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Smp Min Min Tot Smp Avg Smp Smp
  V68 V69 V70 V71 V72 V73 V74 V75 V76 V77 V78 V79 V80 V81 V82 V83 V84 V85 V86 V87 V88 V89
1 Smp Smp Avg Avg Avg Avg Avg Avg Avg Avg Tot Avg Avg Avg Smp Smp Smp Smp Smp Smp Smp Smp
  V90
1 Smp
> readLines(testfile.bin,n=1)
[1] "\"\",\"\",\"\",\"Smp\",\"Smp\",\"Tot\",\"Smp\",\"Smp\",\"Tot\",\"Smp\",\"Smp\",\"Tot\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Min\",\"Min\",\"Tot\",\"Smp\",\"Avg\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Avg\",\"Avg\",\"Avg\",\"Avg\",\"Avg\",\"Avg\",\"Avg\",\"Avg\",\"Tot\",\"Avg\",\"Avg\",\"Avg\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\",\"Smp\""
> seek(testfile.text)
[1] 2931
> seek(testfile.bin)
[1] 2428
> 2931-2428
[1] 503
> read.csv(testfile.text,header = FALSE,nrows = 1)
     V1    V2    V3    V4    V5    V6    V7    V8    V9   V10   V11   V12   V13   V14   V15
1 ULONG ULONG ULONG IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
    V16  V17   V18   V19   V20   V21   V22   V23   V24   V25   V26   V27   V28   V29   V30
1 IEEE4 BOOL IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
    V31   V32   V33   V34   V35   V36   V37   V38   V39   V40   V41   V42   V43   V44   V45
1 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
    V46   V47   V48   V49   V50   V51   V52   V53   V54   V55   V56   V57   V58   V59   V60
1 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
    V61   V62   V63   V64   V65   V66   V67   V68   V69   V70   V71   V72   V73   V74   V75
1 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
    V76   V77   V78   V79   V80   V81   V82   V83   V84   V85   V86   V87   V88   V89
1 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4 IEEE4
        V90
1 ASCII(16)

> readLines(testfile.bin,n=1)
[1] "\"ULONG\",\"ULONG\",\"ULONG\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"BOOL\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"IEEE4\",\"ASCII(16)\""
> seek(testfile.text)
[1] 3654
> seek(testfile.bin)
[1] 3152
> 3654-3152
[1] 502

将ReadLines也用于文本数据时也会出现同样的问题:

> readLines(testfile.text,n=1)
[1] "\"TOB1\",\"Tower\",\"CR6\",\"7562\",\"CR6.Std.10.02\",\"CPU:EasyFlux_Tower.cr6\",\"22445\",\"Flux_CSIFormat\""
> readLines(testfile.bin,n=1)
[1] "\"TOB1\",\"Tower\",\"CR6\",\"7562\",\"CR6.Std.10.02\",\"CPU:EasyFlux_Tower.cr6\",\"22445\",\"Flux_CSIFormat\""
> seek(testfile.text)
[1] 601
> seek(testfile.bin)
[1] 95

在编辑器中打开时的标题和第一个数据行:

"TOB1","Tower","CR6","7562","CR6.Std.10.02","CPU:EasyFlux_Tower.cr6","22445","Flux_CSIFormat"
"SECONDS","NANOSECONDS","RECORD","FC_mass","FC_QC","FC_samples","LE","LE_QC","LE_samples","H","H_QC","H_samples","NETRAD","G","SG","energy_closure","poor_energy_closure_flg","Bowen_ratio","TAU","TAU_QC","USTAR","TSTAR","TKE","TA_1_1_1","RH_1_1_1","T_DP_1_1_1","amb_e","amb_e_sat","TA_2_1_1","RH_2_1_1","T_DP_2_1_1","e","e_sat","TA_3_1_1","RH_3_1_1","T_DP_3_1_1","e_probe","e_sat_probe","H2O_probe","PA","VPD","Ux","Ux_SIGMA","Uy","Uy_SIGMA","Uz","Uz_SIGMA","T_SONIC","T_SONIC_SIGMA","sonic_azimuth","WS","WS_RSLT","WD_SONIC","WD_SIGMA","WD","WS_MAX","CO2_density","CO2_density_SIGMA","H2O_density","H2O_density_SIGMA","CO2_sig_strgth_Min","H2O_sig_strgth_Min","P","ALB","SW_IN","SW_OUT","LW_IN","LW_OUT","T_nr_Avg","LW_IN_meas","LW_OUT_meas","PPFD_IN","sun_azimuth","sun_elevation","hour_angle","sun_declination","air_mass_coeff","daytime","TS_1_1_1","SWC_1_1_1","cs65x_ec_1_1_1","G_plate_1_1_1","shfp_cal_1_1_1","FETCH_MAX","FETCH_90","FETCH_55","FETCH_40","UPWND_DIST_INTRST","FP_DIST_INTRST","FP_EQUATION"
"SECONDS","NANOSECONDS","RN","mg m-2 s-1","Grade","samples","W m-2","Grade","samples","W m-2","grade","samples","W m-2","W m-2","W m-2","fraction","","fraction","kg m-1 s-2","grade","m s-1","deg C","m2 s-2","deg C","%","deg C","kPa","kPa","deg C","%","deg C","kPa","kPa","deg C","%","deg C","kPa","kPa","g/m^3","kPa","hPa","m s-1","m s-1","m s-1","m s-1","m s-1","m s-1","deg C","deg C","Decimal degrees","m s-1","m s-1","decimal degrees","decimal degrees","decimal degrees","m s-1","mg m-3","mg m-3","g m-3","g m-3","fraction","fraction","mm","%","W m-2","W m-2","W m-2","W m-2","Klvin","W m-2","W m-22","umolPhoton m-2 s-1","decimal degrees","decimal degrees","decimal degrees","decimal degrees","adimensional","fraction","deg C","%","dS m-1","W m-2","","m","m","m","m","m","%","authors"
"","","","Smp","Smp","Tot","Smp","Smp","Tot","Smp","Smp","Tot","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Min","Min","Tot","Smp","Avg","Smp","Smp","Smp","Smp","Avg","Avg","Avg","Avg","Avg","Avg","Avg","Avg","Tot","Avg","Avg","Avg","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp","Smp"
"ULONG","ULONG","ULONG","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","BOOL","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","IEEE4","ASCII(16)"
|·<    ç  Î“f¾  A  ŒFÓÁ@À  A  ŒFL>  A  ŒFi3]Â=dÖ¿Ýè¿[øP= 𧻽ǩV=  À@o>ŽzŸºoçD?\ÇqAõ.CB…•@¢JW?r¢Ü?ûÞ’Aõš BƳ’@‚·Y?ÿ™@¨pAdÁB"ÒgApîÓ?:§Û?"/GA䔡BLîJA§`á>ŒW??V1¿è‹]?­¾Œ½t‘>¶šA!¨²>    ‹À¢?ÀR?‹7—Cíè@B¨CfBñ%u@œm   D9Ì@åJÊ@nä“=µ9q?ÉF^?        ’X`À¼‹V?™©CÂC
CÑäRÂÚ#â¿    ^H
ýŠBìd•Âl|%ÃääAÿÿÿ    ׂAyž@Bh¡÷<
>¨3«A„œÆBòC5ßC?í×BVÕªD:BÅBKljun et al