case 描述:
- 我有一个连接到mongo DB的非常简单的应用程序(由Spring Initializr生成).
- 应用程序的任务是执行
2
个查询(count
和find
)并返回响应.(Mongo查询复制自[DEBUG]登录"MongoTemplate",这就是它显示$regularExpression
而不是$regex
的原因)
前query名:
{
"$and": [
{"domain": "test"},
{"deleted": {"$ne": true}},
{"specialItem": {"$ne": true}},
{"$or": [
{"textSearch": {"$regularExpression": {"pattern": "\\QHANDLE\\E", "options": ""}}}
]
},
{
"$or": [
{"textSearch": {"$regularExpression": {"pattern": "\\QMOUNT\\E", "options": ""}}}
]
}
]
}
生成上述查询的code:
@GetMapping("/demo") public ResponseEntity<String> demo() {
final List<org.springframework.data.mongodb.core.query.Criteria> criterias = new ArrayList<>();
criterias.add(Criteria.where("domain").is("test"));
criterias.add(Criteria.where("deleted").ne(true));
criterias.add(Criteria.where("specialItem").ne(true));
for (String word : List.of("handle", "mount")) {
criterias.add(new Criteria().orOperator(
Criteria.where("textSearch").regex("^.*" + Pattern.quote(word.toUpperCase()) + ".*")
));
}
final List<Criteria> orCriterias = new ArrayList<>();
if(!orCriterias.isEmpty())
criterias.add(new Criteria().orOperator(orCriterias));
Query q = Query.query(new Criteria().andOperator(criterias.toArray(new Criteria[]{}))).with(Sort.by("domain", "deleted", "itemNumber"));
q = q.skip(0).limit(20);
long total = mongoOperations.count(q, DaoItem.class);
final List<DaoItem> results = new ArrayList<>();
List<DaoItem> daoItems = mongoOperations.find(q, DaoItem.class);
results.addAll(daoItems);
return ResponseEntity.ok("total: " + total + ", result: " + results);
}
-
textSearch
是字符串数组,F.E.:["00000001", "SMOUNT.", "TER.", "YYY", "1", "GLASS"]
问题描述:
前面提到的query takes long time,取决于执行它的环境.
环境描述:
-
mongo-java-driver|sync
,版本:4.7.1
-
mongodb
个版本因环境不同而不同(主要版本相同) -
spring-boot
版本:3.1.3
我在循环中使用JMeter for each 环境打印基本响应时间图.测试针对的是同一个域:test
个,在每个环境中.对于每个非本地环境.ping <server address>
等于时间~160 ms
我有四个环境:
- Local,
-
DB:Mongo(
v3.6.19
)在被告席上有747259
份文件要看, -
Resources:
m1 max
+64 GB
, -
Space:
2 TB
中有530 GB
个可用, -
Response time:~
200
-300
ms
-
DB:Mongo(
- Dev,
-
DB:Mongo安装在机器(
v3.6.8
)上,有747259
个文件要查看, -
Resources:
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
+5,78 GB
, -
Space:
/dev/sda1 146G 9.3G 129G 7% /
, -
Response time:
3
-8
s
-
DB:Mongo安装在机器(
- QA,
-
DB:Mongo(
v3.6.8
)安装在机器上,有42491
个文件要查看, -
Resources:
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
+5,78 GB
, -
Space:
/dev/sda1 146G 6.3G 132G 5% /
. -
Response time:~
2
-3
s
.
-
DB:Mongo(
我的观察结果:
- 我在一台远程服务器上设置了一个新的MongoDB实例 (docker ).从"Dev"中导出文档,然后导入其中.无名之辈 使用该数据库.当我连接到它时,我有响应时间 类似于"Dev"env.
- 由于在查询中使用了部分匹配,因此不能使用索引.获胜的计划是:
IXSCAN
(当然不包括textSearch
) - 需要部分匹配,因此我不能使用Mongo
text
索引. - 每个环境上的MongoDB配置是相同的(默认)
-
Dev
环境的查询explain
返回:
Documents Returned: 138 Index Keys Examined: 62448 Documents Examined: 62446 Actual Query Execution Time (ms): 2479 Sorted in Memory: no Query used the following index: DOMAIN, DELETED SPECIALITEM
我更改了代码,以生成以下查询,但没有发现响应时间发生变化:
{
"$and": [
{"domain": "egtest-adbegt0000"},
{"deleted": {"$ne": true}},
{"supplyArticle": {"$ne": true}},
{"$or": [
{"searchWords": {"$regularExpression": {"pattern": "\\QGIPS\\E","options": ""}}},
{"searchWords": {"$regularExpression": {"pattern": "\\Q1200\\E","options": ""}}}
]
}
]
}
{
"$and": [
{"domain": "egtest-adbegt0000"},
{"deleted": {"$ne": true}},
{"supplyArticle": {"$ne": true}},
{"searchWords": {"$regularExpression": {"pattern": "\\QGIPS\\E","options": ""}}},
{"searchWords": {"$regularExpression": {"pattern": "\\Q1200\\E","options": ""}}}
}
]
}
以下是explain
强的结果:
-你在说什么?
{
"domain": "egtest-adbegt0000",
"deleted": {"$ne": true},
"supplyArticle": {"$ne": true},
"searchWords": {"$regex": "(GIPS|1200)"}
}
explain
结果(完整):
{
"explainVersion": "2",
"queryPlanner": {
"namespace": "test.daoArticles",
"indexFilterSet": false,
"parsedQuery": {
"$and": [
{
"domain": {
"$eq": "egtest-adbegt0000"
}
},
{
"searchWords": {
"$regex": "(GIPS|1200)"
}
},
{
"deleted": {
"$not": {
"$eq": true
}
}
},
{
"supplyArticle": {
"$not": {
"$eq": true
}
}
}
]
},
"queryHash": "077462CA",
"planCacheKey": "1BD290D3",
"maxIndexedOrSolutionsReached": false,
"maxIndexedAndSolutionsReached": false,
"maxScansToExplodeReached": false,
"winningPlan": {
"queryPlan": {
"stage": "FETCH",
"planNodeId": 2,
"filter": {
"$and": [
{
"deleted": {
"$not": {
"$eq": true
}
}
},
{
"supplyArticle": {
"$not": {
"$eq": true
}
}
},
{
"searchWords": {
"$regex": "(GIPS|1200)"
}
}
]
},
"inputStage": {
"stage": "IXSCAN",
"planNodeId": 1,
"keyPattern": {
"domain": -1,
"articleNo": -1
},
"indexName": "domain_-1_articleNo_-1",
"isMultiKey": false,
"multiKeyPaths": {
"domain": [],
"articleNo": []
},
"isUnique": false,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"domain": [
"[\"egtest-adbegt0000\", \"egtest-adbegt0000\"]"
],
"articleNo": [
"[MaxKey, MinKey]"
]
}
}
},
"slotBasedPlan": {
"slots": "$$RESULT=s11 env: { s3 = 1694351541263 (NOW), s6 = KS(C39A988B9A8C8BD29E9B9D9A988BCFCFCFCFFFF5FE04), s18 = /(GIPS|1200)/, s2 = Nothing (SEARCH_META), s5 = KS(C39A988B9A8C8BD29E9B9D9A988BCFCFCFCFFF0F0104), s17 = true, s10 = {\"domain\" : -1, \"articleNo\" : -1}, s1 = TimeZoneDatabase(Europe/Monaco...Etc/GMT-10) (timeZoneDB), s19 = PcreRegex(/(GIPS|1200)/), s16 = true }",
"stages": "[2] filter {(!(traverseF(s13, lambda(l1.0) { ((l1.0 == s16) ?: false) }, false)) && (!(traverseF(s15, lambda(l2.0) { ((l2.0 == s17) ?: false) }, false)) && traverseF(s14, lambda(l3.0) { (((l3.0 == s18) ?: false) || (regexMatch(s19, l3.0) ?: false)) }, false)))} \n[2] nlj inner [] [s4, s7, s8, s9, s10] \n left \n [1] cfilter {(exists(s5) && exists(s6))} \n [1] ixseek s5 s6 s9 s4 s7 s8 [] @\"222538cb-3883-4079-bc0f-038b10a80b5f\" @\"domain_-1_articleNo_-1\" true \n right \n [2] limit 1 \n [2] seek s4 s11 s12 s7 s8 s9 s10 [s13 = deleted, s14 = searchWords, s15 = supplyArticle] @\"222538cb-3883-4079-bc0f-038b10a80b5f\" true false \n"
}
},
"rejectedPlans": [
{
"queryPlan": {
"stage": "FETCH",
"planNodeId": 2,
"filter": {
"$and": [
{
"deleted": {
"$not": {
"$eq": true
}
}
},
{
"supplyArticle": {
"$not": {
"$eq": true
}
}
},
{
"searchWords": {
"$regex": "(GIPS|1200)"
}
}
]
},
"inputStage": {
"stage": "IXSCAN",
"planNodeId": 1,
"keyPattern": {
"domain": 1,
"articleNo": 1
},
"indexName": "domain_1_articleNo_1",
"isMultiKey": false,
"multiKeyPaths": {
"domain": [],
"articleNo": []
},
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"domain": [
"[\"egtest-adbegt0000\", \"egtest-adbegt0000\"]"
],
"articleNo": [
"[MinKey, MaxKey]"
]
}
}
},
"slotBasedPlan": {
"slots": "$$RESULT=s11 env: { s16 = true, s18 = /(GIPS|1200)/, s3 = 1694351541263 (NOW), s6 = KS(3C6567746573742D6164626567743030303000F0FE04), s17 = true, s5 = KS(3C6567746573742D61646265677430303030000A0104), s2 = Nothing (SEARCH_META), s10 = {\"domain\" : 1, \"articleNo\" : 1}, s1 = TimeZoneDatabase(Europe/Monaco...Etc/GMT-10) (timeZoneDB), s19 = PcreRegex(/(GIPS|1200)/) }",
"stages": "[2] filter {(!(traverseF(s13, lambda(l1.0) { ((l1.0 == s16) ?: false) }, false)) && (!(traverseF(s15, lambda(l2.0) { ((l2.0 == s17) ?: false) }, false)) && traverseF(s14, lambda(l3.0) { (((l3.0 == s18) ?: false) || (regexMatch(s19, l3.0) ?: false)) }, false)))} \n[2] nlj inner [] [s4, s7, s8, s9, s10] \n left \n [1] cfilter {(exists(s5) && exists(s6))} \n [1] ixseek s5 s6 s9 s4 s7 s8 [] @\"222538cb-3883-4079-bc0f-038b10a80b5f\" @\"domain_1_articleNo_1\" true \n right \n [2] limit 1 \n [2] seek s4 s11 s12 s7 s8 s9 s10 [s13 = deleted, s14 = searchWords, s15 = supplyArticle] @\"222538cb-3883-4079-bc0f-038b10a80b5f\" true false \n"
}
}
]
},
"executionStats": {
"executionSuccess": true,
"nReturned": 2853,
"executionTimeMillis": 1573,
"totalKeysExamined": 76606,
"totalDocsExamined": 76606,
"executionStages": {
"stage": "filter",
"planNodeId": 2,
"nReturned": 2853,
"executionTimeMillisEstimate": 1530,
"opens": 1,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 1,
"numTested": 76606,
"filter": "(!(traverseF(s13, lambda(l1.0) { ((l1.0 == s16) ?: false) }, false)) && (!(traverseF(s15, lambda(l2.0) { ((l2.0 == s17) ?: false) }, false)) && traverseF(s14, lambda(l3.0) { (((l3.0 == s18) ?: false) || (regexMatch(s19, l3.0) ?: false)) }, false))) ",
"inputStage": {
"stage": "nlj",
"planNodeId": 2,
"nReturned": 76606,
"executionTimeMillisEstimate": 719,
"opens": 1,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 1,
"totalDocsExamined": 76606,
"totalKeysExamined": 76606,
"collectionScans": 0,
"collectionSeeks": 76606,
"indexScans": 0,
"indexSeeks": 1,
"indexesUsed": [
"domain_-1_articleNo_-1"
],
"innerOpens": 76606,
"innerCloses": 1,
"outerProjects": [],
"outerCorrelated": [
4,
7,
8,
9,
10
],
"outerStage": {
"stage": "cfilter",
"planNodeId": 1,
"nReturned": 76606,
"executionTimeMillisEstimate": 311,
"opens": 1,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 1,
"numTested": 1,
"filter": "(exists(s5) && exists(s6)) ",
"inputStage": {
"stage": "ixseek",
"planNodeId": 1,
"nReturned": 76606,
"executionTimeMillisEstimate": 301,
"opens": 1,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 1,
"indexName": "domain_-1_articleNo_-1",
"keysExamined": 76606,
"seeks": 1,
"numReads": 76607,
"indexKeySlot": 9,
"recordIdSlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"outputSlots": [],
"indexKeysToInclude": "00000000000000000000000000000000",
"seekKeyLow": "s5 ",
"seekKeyHigh": "s6 "
}
},
"innerStage": {
"stage": "limit",
"planNodeId": 2,
"nReturned": 76606,
"executionTimeMillisEstimate": 382,
"opens": 76606,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 1,
"limit": 1,
"inputStage": {
"stage": "seek",
"planNodeId": 2,
"nReturned": 76606,
"executionTimeMillisEstimate": 378,
"opens": 76606,
"closes": 1,
"saveState": 96,
"restoreState": 96,
"isEOF": 0,
"numReads": 76606,
"recordSlot": 11,
"recordIdSlot": 12,
"seekKeySlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"indexKeySlot": 9,
"indexKeyPatternSlot": 10,
"fields": [
"deleted",
"searchWords",
"supplyArticle"
],
"outputSlots": [
13,
14,
15
]
}
}
}
},
"allPlansExecution": [
{
"nReturned": 101,
"executionTimeMillisEstimate": 50,
"totalKeysExamined": 1705,
"totalDocsExamined": 1705,
"score": 1.0301032541776605,
"executionStages": {
"stage": "filter",
"planNodeId": 2,
"nReturned": 101,
"executionTimeMillisEstimate": 50,
"opens": 1,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numTested": 1705,
"filter": "(!(traverseF(s13, lambda(l1.0) { ((l1.0 == s16) ?: false) }, false)) && (!(traverseF(s15, lambda(l2.0) { ((l2.0 == s17) ?: false) }, false)) && traverseF(s14, lambda(l3.0) { (((l3.0 == s18) ?: false) || (regexMatch(s19, l3.0) ?: false)) }, false))) ",
"inputStage": {
"stage": "nlj",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 20,
"opens": 1,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"totalDocsExamined": 1705,
"totalKeysExamined": 1705,
"collectionScans": 0,
"collectionSeeks": 1705,
"indexScans": 0,
"indexSeeks": 1,
"indexesUsed": [
"domain_-1_articleNo_-1"
],
"innerOpens": 1705,
"innerCloses": 0,
"outerProjects": [],
"outerCorrelated": [
4,
7,
8,
9,
10
],
"outerStage": {
"stage": "cfilter",
"planNodeId": 1,
"nReturned": 1705,
"executionTimeMillisEstimate": 10,
"opens": 1,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numTested": 1,
"filter": "(exists(s5) && exists(s6)) ",
"inputStage": {
"stage": "ixseek",
"planNodeId": 1,
"nReturned": 1705,
"executionTimeMillisEstimate": 10,
"opens": 1,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"indexName": "domain_-1_articleNo_-1",
"keysExamined": 1705,
"seeks": 1,
"numReads": 1705,
"indexKeySlot": 9,
"recordIdSlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"outputSlots": [],
"indexKeysToInclude": "00000000000000000000000000000000",
"seekKeyLow": "s5 ",
"seekKeyHigh": "s6 "
}
},
"innerStage": {
"stage": "limit",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 10,
"opens": 1705,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 1,
"limit": 1,
"inputStage": {
"stage": "seek",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 10,
"opens": 1705,
"closes": 0,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numReads": 1705,
"recordSlot": 11,
"recordIdSlot": 12,
"seekKeySlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"indexKeySlot": 9,
"indexKeyPatternSlot": 10,
"fields": [
"deleted",
"searchWords",
"supplyArticle"
],
"outputSlots": [
13,
14,
15
]
}
}
}
}
},
{
"nReturned": 91,
"executionTimeMillisEstimate": 40,
"totalKeysExamined": 1705,
"totalDocsExamined": 1705,
"score": 1.0271636576787808,
"executionStages": {
"stage": "filter",
"planNodeId": 2,
"nReturned": 91,
"executionTimeMillisEstimate": 40,
"opens": 1,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numTested": 1705,
"filter": "(!(traverseF(s13, lambda(l1.0) { ((l1.0 == s16) ?: false) }, false)) && (!(traverseF(s15, lambda(l2.0) { ((l2.0 == s17) ?: false) }, false)) && traverseF(s14, lambda(l3.0) { (((l3.0 == s18) ?: false) || (regexMatch(s19, l3.0) ?: false)) }, false))) ",
"inputStage": {
"stage": "nlj",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 37,
"opens": 1,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"totalDocsExamined": 1705,
"totalKeysExamined": 1705,
"collectionScans": 0,
"collectionSeeks": 1705,
"indexScans": 0,
"indexSeeks": 1,
"indexesUsed": [
"domain_1_articleNo_1"
],
"innerOpens": 1705,
"innerCloses": 1,
"outerProjects": [],
"outerCorrelated": [
4,
7,
8,
9,
10
],
"outerStage": {
"stage": "cfilter",
"planNodeId": 1,
"nReturned": 1705,
"executionTimeMillisEstimate": 0,
"opens": 1,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numTested": 1,
"filter": "(exists(s5) && exists(s6)) ",
"inputStage": {
"stage": "ixseek",
"planNodeId": 1,
"nReturned": 1705,
"executionTimeMillisEstimate": 0,
"opens": 1,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"indexName": "domain_1_articleNo_1",
"keysExamined": 1705,
"seeks": 1,
"numReads": 1706,
"indexKeySlot": 9,
"recordIdSlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"outputSlots": [],
"indexKeysToInclude": "00000000000000000000000000000000",
"seekKeyLow": "s5 ",
"seekKeyHigh": "s6 "
}
},
"innerStage": {
"stage": "limit",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 37,
"opens": 1705,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 1,
"limit": 1,
"inputStage": {
"stage": "seek",
"planNodeId": 2,
"nReturned": 1705,
"executionTimeMillisEstimate": 37,
"opens": 1705,
"closes": 1,
"saveState": 4,
"restoreState": 4,
"isEOF": 0,
"numReads": 1705,
"recordSlot": 11,
"recordIdSlot": 12,
"seekKeySlot": 4,
"snapshotIdSlot": 7,
"indexIdentSlot": 8,
"indexKeySlot": 9,
"indexKeyPatternSlot": 10,
"fields": [
"deleted",
"searchWords",
"supplyArticle"
],
"outputSlots": [
13,
14,
15
]
}
}
}
}
}
]
},
"command": {
"find": "daoItem",
"filter": {
"domain": "egtest-adbegt0000",
"deleted": {
"$ne": true
},
"supplyArticle": {
"$ne": true
},
"searchWords": {
"$regex": "(GIPS|1200)"
}
},
"skip": 0,
"limit": 0,
"maxTimeMS": 60000,
"$db": "test"
},
"serverInfo": {
"host": "09c0c735d1b1",
"port": 27017,
"version": "7.0.1",
"gitVersion": "425a0454d12f2664f9e31002bbe4a386a25345b5"
},
"serverParameters": {
"internalQueryFacetBufferSizeBytes": 104857600,
"internalQueryFacetMaxOutputDocSizeBytes": 104857600,
"internalLookupStageIntermediateDocumentMaxSizeBytes": 104857600,
"internalDocumentSourceGroupMaxMemoryBytes": 104857600,
"internalQueryMaxBlockingSortMemoryUsageBytes": 104857600,
"internalQueryProhibitBlockingMergeOnMongoS": 0,
"internalQueryMaxAddToSetBytes": 104857600,
"internalDocumentSourceSetWindowFieldsMaxMemoryBytes": 104857600,
"internalQueryFrameworkControl": "trySbeEngine"
},
"ok": 1
}
explain
结果(简短):
Documents Returned: 2853 Index Keys Examined: 76606 Documents Examined: 76606 Actual Query Execution Time (ms): 1391 Sorted in Memory: no Query used the following index: DOMAIN ARTICLENO
---
FETCH nReturned: 2853 Execution Time: 974ms IXSCAN nReturned: 76606 Execution Time: 358ms Index Name: domain_-1_articleNo_-1 Multi Key Index: no
我的问题
- 说到Mongo,我还是个新手,所以我别无 Select .
- 当涉及到那个查询时,我还可以从什么方向进行调试,以调试为什么在"Dev"环境上花费了这么长时间?
- 我正在考虑将搜索机制转移到ElasticSearch,但这个特别的问题仍然是关于Mongo的,我可以做些什么来将响应时间减少到
~2s
F.E.在Dev
env上?