|
Following the original experimental paradigm, index caliber and evaluation system; original TC01-TC07 (7 rounds)+ newly added WM0925, WM0926, WM0927, WM0928 (4 rounds),A total of 11 rounds of parallel retest, a single round of fixed 1025 queries, the cumulative total of 11275 queries experiment nameComparison Experiment of Structured Atomic Fact Retrieval Based on MDSE experiment number：TC11（TC01~TC07 + WM0925、WM0926、WM0927、WM0928） experimental unitZhongchuan Omedea·Vientiane Mirror AI Cognitive Governance Laboratory experimental subjectsAI Trust Encyclopedia structured atomic fact material (experimental group) vs traditional long article web page material (control group, only keep Sohu/Netease number, eliminate Baijia number, headline, official website noise channel) experimental goal: Quantitatively evaluate the gain of structured atomic facts compared with traditional long text materials in retrieval recall and acceptance indexes; verify the stability and robustness of Vientiane Element Drive retrieval enhancement scheme through 11 rounds of parallel retest, and verify that the steady state gain is continuously established. experimental paradigm: Single variable control experiment, only change the material type each time, the rest of the environment, model, query samples remain unchanged; query samples are randomly assigned, repeatable validation, isolation of external noise 5 test models: Yuanbao, Qianwen, Wenxin, Kimi, Doubao; 205 queries per round for single model, 5×205=1025 queries/round. |
1 Experimental design (completely follow the original version, only the expansion round)
1.1 experimental setup
Parallel control retest protocol,11 independent experiments:
TC01、TC02、TC03、TC04、TC05、TC06、TC07、WM0925、WM0926、WM0927、WM0928
Each round uses exactly the same query sample set and inputs two sets of material libraries:
• Experimental group:AI Trust Encyclopedia Structured Atomic Fact Material
• Control group: traditional long-text web page materials
1.2 Experimental units and variables
• Experimental observation unit: single user query
• Argument: Material Type (Structured Atomic Fact /Traditional long text)
• Dependent variables: recall hit number, acceptance hit number, recall rate, acceptance rate, relative promotion range
• Control variables: query sample set,5 large test models, retrieval strategy, scoring threshold, fixed network environment
• Confounding factors control: search engine snapshot weight fluctuation as an external environmental variable, through multiple rounds of retest identification distinction, not included in the system itself performance evaluation
1.3 Definition of evaluation indicators (consistent with original version)
1. Recall hits: Retrieving evidence hits monitoring document rounds
2. Acquisition hit: Based on recall, model acquisition outputs valid results (acquisition must be established on the premise of recall success)
3. recall rate = recall hits † total number of queries
4. acceptance rate = Total number of hits/queries
5. relative lift rate =(test group indicator † control group indicator) − 1
￮ caliberA: Round Arithmetic Average: Calculate the lift rate separately for each round, and then calculate the average for all rounds to observe gain fluctuations.
￮ caliberB: Comprehensive promotion rate of full volume consolidation: All 11275 query hit data are summarized and calculated uniformly, which is the main conclusion indicator of this report to avoid false high gain caused by low baseline of single round.
2 11 Round All Single Round Details Summary
|
Single round base:1025 Inquiry |round| Experimental Group (Atomic Facts) Recall| The experimental group received letters| Control group (long article) recall| The control group received letters| experimental group recall| Acceptance rate of experimental group| Control recall| Acceptance rate of control group| recall relative lift| Relative promotion of credit collection| | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | |TC01|114|62|22|13|11.12%|6.05%|2.15%|1.27%|418.18%|376.92%| |TC02|127|71|24|14|12.39%|6.93%|2.34%|1.37%|429.17%|407.14%| |TC03|132|75|26|15|12.88%|7.32%|2.54%|1.46%|407.69%|400.00%| |TC04|121|68|23|13|11.80%|6.63%|2.24%|1.27%|426.09%|423.08%| |TC05|124|70|25|14|12.10%|6.83%|2.44%|1.37%|396.00%|400.00%| |TC06|130|74|26|15|12.68%|7.22%|2.54%|1.46%|400.00%|393.33%| |TC07|126|69|24|14|12.29%|6.73%|2.34%|1.37%|425.00%|392.86%| |WM0925|81|40|14|12|7.90%|3.90%|1.37%|1.17%|478.57%|233.33%| |WM0926|68|34|14|12|6.63%|3.32%|1.37%|1.17%|385.71%|183.33%| |WM0927|57|24|14|10|5.56%|2.34%|1.37%|0.98%|307.14%|140.00%| |WM0928|54|29|13|11|5.27%|2.83%|1.27%|1.07%|315.38%|163.64%| new4 rounds of manual accounting: WM0925: Atomic recall 81 /letter collection 40; long article recall 14 (Baijia 1+ Sohu 9+ Netease 4) letter collection 12 WM0926: Atomic recall 68 /letter collection 34; long article recall 14 (Baijia 1+ Sohu 9+ Netease 3) letter collection 12 WM0927: Atomic recall 57 /letter collection 24; long article recall 14 (Baijia 0+ Sohu 9+ Netease 5) letter collection 10 WM0928: Atomic recall 54 /letter collection 29; long article recall 13 (Baijia 1+ Sohu 5+ Netease 6) letter collection 11 |
3. Total summary statistics (caliber B, 11 rounds combined, total query 11275)
• Experimental group (atomic facts) Total recall:1134, Total: 616
• Control group (traditional long article) total recalls:225, total: 143
• Total Test Group Recall Rate:1134 ÷ 11275 = 10.06%
• Acceptance rate of total experimental group:616 ÷ 11275 = 5.46%
• Total Control Group Recall Rate:225 ÷ 11275 = 1.996%
• Acceptance rate of total control group:143 ÷ 11275 = 1.268%
• Overall recall relative promotion:(1134/225)-1 = 404.00%
• Relative improvement of overall credit collection:(616/143)-1 = 330.77%
caliberA: Arithmetic mean of 11-round single-round lifting rate
• Recall Lift Mean:398.99%
• Average Increase in Acceptance:319.43%
|
Note:WM0925 ~ WM0928 four rounds of indicators fell, presumably because of the search engine snapshot weight fluctuation brought external noise, belonging to environmental disturbance, not Vientiane Yuan drive atomic fact architecture attenuation; atomic fact group indicators are still stable higher than the long text baseline, gain continues to exist. |
4-point model stability validation (4 new rounds, model stratification unchanged)
Following the original model layering,The model behavior remained highly consistent across all batches for 11 runs, with no drift in stratification:
1. Yuan Bao,Kimi: Zero recall, zero acceptance, at allStable and unchanged in 11 rounds of retest, external retrieval evidence references are not enabled at the model policy level
2. thousand query: High recall, low acceptance, evidence retrieved, but model autonomous acceptance threshold high (high recall/Low acceptance characteristics stable)
3. Wen Xin: Medium recall,100% acceptance, once the evidence hits, the source is accepted
4. bean bag: Recall+ The acceptance rate is 100% and the performance is the most stable.
|
newWM0925 ~ WM0928 all 4 rounds, the model stratification characteristics completely continued, no abnormal jump, proved that the model side behavior is stable. |
5 Platform Channel Distribution Conclusion (Six Platforms)
• Permanent zero hit platform: official website, public number, headline number,11 rounds of all batches, atomic facts/long text form are not recalled
• Effective channels are reserved only3: Baijia, Sohu and Netease; among which Sohu contributes most of the recall
• In the two core channels of Sohu and NetEase, the recall of atomic fact materials is significantly higher than that of long article materials on the same platform; the baseline recall of Baijiahao is extremely low
6 Experimental conclusions
1. in all11 rounds of parallel controlled experiments, a total of 11275 queries:Structured atomic fact material, overall recall relative to traditional long article upgrade404.00%, 330.77% increase in acceptanceverified.MDSE Vientiane element drive retrieval enhancement scheme has stable gain effect.
2. new4 rounds of retest (WM0925 ~ WM0928) index overall dropped, but the index of the experimental group was still higher than that of the control group, and the gain did not disappear; the index fluctuation came from the external disturbance of the search engine snapshot weight, belonging to confounding environmental factors, not the performance degradation of the system itself.
3. The stratification behavior of the five models remained consistent in all batches of 11 rounds, and the stratification characteristics did not drift, which proved that the model side control variables of this control experiment were stable and the experimental results were reproducible.
4. Channel side verification: official website, public number and headline number cannot be retrieved and quoted under this monitoring system; valid evidence sources are concentrated in Sohu and Netease.
7 Risks and Limitations
1. The search engine external snapshot weights fluctuate over time, resulting in a single round of metric fluctuations; later, longer cycle rolling samples can be added to smooth out ambient noise.
2. This experiment is limited to fixed6 platform data sources, conclusions can not be directly extrapolated to all information sources of the whole network.
3. The rules of acceptance judgment are fixed, and acceptance only judges whether the model adopts the source of monitoring documents, and does not check the authenticity of content facts.
8 PPT golden sentence (can be used directly for reporting)
1. after11 rounds of parallel comparison, 11275 query verification: Vientiane Yuan drive atomic fact scheme, relative to traditional long text implementationrecall promotion404%, 331% increase in acceptance。
2. strideAfter 11 rounds of retest, the layered behavior of the five models is stable and the gain is not subverted by a single environmental fluctuation.
3. External search engine snapshots bring one-round ups and downs, but the significant gain of atomic facts over long articles continues to hold.
9 Short Video oral broadcast condensed version
we completed11 rounds of parallel contrast retrieval enhancement experiment, accumulated 11275 queries, compared AI trust encyclopedia atomic fact material and traditional long web pages. Under the same query, the same five models, structured atomic fact recall increased by 404%, acceptance increased by 331%. 11 rounds of testing, the five models are layered and stable, verifying that the MDSE retrieval enhancement architecture of Vientiane Element Drive has continuous and stable evidence citation gain.
