我是Bash脚本编程的新手,我想测试提取数据,我的计划是基于AWS从.sh文件(Bashscript)的查询将数据提取到一个.txt文件.我有一个更大的目标,但我想在输入我的主要AWS查询之前先完成这项工作.

I have a .sh script here below.

NOTE: I put all values to -confidential- without any quotations. Just the value

#!/bin/bash

variables () {
lambdas="confidential"
}
variables

queryId=$(aws logs start-query --log-group-name $lambdas --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: '*'," as API | parse @message "status: *," as Status | parse @message "channel: '*'," as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc' | jq '.queryId' | awk -F '"' '{print $2}')


results=$(aws logs get-query-results --query-id $queryId | jq .)


echo "$results" > testextract5.txt

Whenever I run that .sh file, this is the result .txt below:

{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}

BUT when I manually copy paste each command from .sh file all the bash script in the git-bash command line, it works! See below:

jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ queryId=$(aws logs start-query --log-group-name $lambdas --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: '*'," as API | parse @message "status: *," as Status | parse @message "channel: '*'" as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc' | jq '.queryId' | awk -F '"' '{print $2}')
jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ results=$(aws logs get-query-results --query-id $queryId | jq .)
jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ echo "$results" > testextract5.txt

RESULT extracted .txt below,

     {
        "field": "API",
        "value": "'Confidential'"
      },
      {
        "field": "Channel",
        "value": "'Confidential'"
      },
      {
        "field": "Status",
        "value": "Confidential"
      },
      {
        "field": "Total",
        "value": "1"
      }
    ]
  ],
  "statistics": {
    "recordsMatched": 16507,
    "recordsScanned": 69000,
    "bytesScanned": 29835564
  },
  "status": "Complete"
}

采取的行动/future 计划:

我try 进行故障排除,并使用了bash-x脚本.sh,但我根本看不到任何问题.我try 回滚到WSL版本1,安装了AWS CLI的最新更新.这些都不管用.我的 playbook 有什么问题吗?

我希望当我运行.sh文件时,应该会有与上面的.txt结果类似的预期数据.我创建此bash脚本的主要目的是因为AWS Cloudwatch Log Insights具有与lambda相关的限制,因此我计划创建一个bash脚本,在$lambda中使用array.这篇文章仅供我试错之用.我需要让这件事继续下go .

=

try 对$lambda和$queryID使用双引号,结果相同,没有数据.在下面

jomasangkay@DESKTOP-R1DM530:/mnt/c/Users/personal pc/Desktop/company/Bashscript/test$ bash -x testing2.sh
+ variables
+ lambdas=confidential
++ jq .queryId
++ awk -F '"' '{print $2}'
++ aws logs start-query --log-group-name confidential --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: *," as API | parse @message "status: *," as Status | parse @message "channel: *," as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc'
+ queryId=0f6c6fd6-da5e-40ed-9c74-ad22791455be
++ aws logs get-query-results --query-id 0f6c6fd6-da5e-40ed-9c74-ad22791455be
++ jq .
+ results='{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}'
+ echo '{
  "results": [],
  "statistics": {
    "recordsMatched": 0,
    "recordsScanned": 0,
    "bytesScanned": 0
  },
  "status": "Running"
}'

======

更新日期:2023-05:15新加坡时间10月3日

我通过创建一个名为test1.txt和test2.txt的2.txt文件隔离了这个问题,它们的内容如下所示:

test1.txt
c86c7f6b-95e4-4d17-9417-2f6987989316
test2.txt
c86c7f6b-95e4-4d17-9417-2f6987989316

正如您在上面看到的,它们具有完全相同的值(没有空格,没有不寻常的文本或其他任何东西).

See may new modified code below,

#!/bin/bash

variables () {
lambdas="confidential"

}
variables

queryId=$(aws logs start-query --log-group-name "$lambdas" --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId"' | jq '.queryId' | awk -F '"' '{print $2}')

#NOT WORKING
echo $queryId > test1.txt

#WORKING
echo c86c7f6b-95e4-4d17-9417-2f6987989316 > test2.txt


queryIds=$(awk '{print $1}' test2.txt)

results=$(aws logs get-query-results --query-id $queryIds | jq .)

echo "$results" > testextract.txt

当我从test1.txt中赋值变量时,它不工作,没有提取数据.

当我从test2.txt赋值变量时,它按预期工作.

由于某种原因,看起来像是来自test1.txt查询id的值被转换为某种特殊字符,我不确定.

推荐答案

Repeat aws logs get-query-results until results are available

正如我们在问题的更新输出中看到的,对aws logs get-query-results的响应为"status": "Running".查询仍然在AWS上运行,结果还没有准备好,无法下载.所以我们只需要稍后再打aws logs get-query-results就可以了.

根据documentation,有效状态值为Scheduled | Running | Complete | Failed | Cancelled | Timeout | Unknown.您可以try 在循环中重复获取结果,直到状态不是ScheduledRunning之一:

while true; do
    results=$(aws logs get-query-results --query-id "$queryId" | jq .)
    status=$(jq -r .status <<< "$results")
    echo "status=$status"
    [ "$status" = "Scheduled" -o "$status" = "Running" ] || break
    sleep 1
done

转移注意力的鲱鱼

最初,这部分看起来很可疑,巴什:

queryId=$(aws ... 'filter ... operationName: '*', ... channel: '*', ...' | ...)
                                             ^^^               ^^^

也就是说,在单引号的表达式之外,是一个"裸露的"*,这看起来不是故意的,而且容易出错.

我们的意图似乎是在单引号的表达式中包括'*',我们可以这样做:通过添加单引号结束从'filter ...开始的单引号的表达式,然后在双引号的表达式中添加'*',然后开始另一个单引号的表达式以继续我们中断的表达式:

queryId=$(aws ... 'filter ... operationName: '"'*'"', ... channel: '"'*'"', ...' | ...)
                                             ^ ending the previous single-quoted expression
                                              ^^^^^ a double-quoted expression
                                                   ^ starting a new single-quoted expression

修复发布的代码中的行:

queryId=$(aws logs start-query --log-group-name "$lambdas" --start-time 1695704400 --end-time 1695707999 --query-string 'filter @message like "requestId" | parse @message "operationName: '"'*'"'," as API | parse @message "status: *," as Status | parse @message "channel: '"'*'"'," as Channel | stats count(*) as Total by API,Channel,Status | sort API,Total desc' | jq '.queryId' | awk -F '"' '{print $2}')

尽管这不是真正的问题,但修复它仍然很好.

Json相关问答推荐

2020-12年草案中的Json列举模式

JOLT拉平数组

Vega通孔信号中的动态梯度

将json数组反序列化为选项 struct

最新版本的Deneb在数据溢出时不支持滚动

如何循环访问多个子数组并在单个JSON中检索数据点

GO KaZaam转换返回意外结果

如何在JSONata对象中迭代并向数组添加新字段?

震动范围改善

如何使用nifi从json文件中过滤属性

如果值不存在,则将值插入 JSON 数组

向 JSON 文件添加注释

如何将动态复杂 json 解析为dart 对象或模型

如何在生产环境中更改 Flutter 应用程序中的数据模型?

UTF-8 字符编码之战 json_encode()

编写 JSON 日志(log)文件的格式?

JSON对象中的JavaScript递归搜索

有必要清理 JSON 吗?

C#扁平化json struct

Spring restTemplate 获取原始 json 字符串