TC11 Search Enhancement Special Experiment Complete Report

experiment nameComparison Experiment of Structured Atomic Fact Retrieval Based on MDSE

experiment number：TC01-TC07(originally6wheel+newWM0924of7Round parallel retest)

experimental unitCopyright © 2017 Beijing Omedea Technology Co., Ltd. All Rights Reserved.·ChinaAICognitive Governance Lab

experimental subjects：AITrust encyclopedia structured atomic facts vs Traditional long text web material

Total number of queries：7wheel, single wheel fixed1025Number of queries, cumulative7175query

experimental goal: Quantitatively evaluate the gain effect of structured atomic facts on retrieval recall and acceptance index compared with traditional long text materials, and verify the stability and robustness of Vientiane element drive retrieval enhancement scheme.

experimental paradigm: Single variable control experiment, only change the material type each time, the rest of the environment, model, query samples remain unchanged; random distribution of query samples, repeatable validation, isolation of external noise

1 experimental design

1.1 experimental setup

Parallel control retest protocol was adopted, and a total of7independent experiment (TC11First edition,WM0919(A)、WM0920(B)、WM0921、WM0922、WM0923、WM0924). Each round uses exactly the same query sample set and inputs two sets of material libraries:

• Experimental group:AITrust encyclopedia structured atomic fact material

• Control group: traditional long-text web page materials (source: Sohu, Netease, excluding Baijia, headlines, official website noise channels)

1.2 Experimental units and variables

• Experimental observation unit: single user query

• Argument: Material Type (Structured Atomic Fact / Traditional long text)

• Dependent variables: recall hit number, acceptance hit number, recall rate, acceptance rate, relative promotion range

• Control variables: query sample set,5test large model, retrieval strategy, scoring threshold, fixed network environment

• Confounding factors control: search engine snapshot weight fluctuation as an external environmental variable, through multiple rounds of retest identification and discrimination, not included in the system itself performance evaluation

5Test models: Yuanbao, Qianwen, Wenxin,Kimi, bean bag; single model allocation per round205Second query,5×205=1025next/The wheel.

1.3 Definition of evaluation indicators

1. Recall hits: Retrieving evidence hits monitoring document rounds

2. Acquisition hit: Based on recall, model acquisition outputs valid results (acquisition must be established on the premise of recall success)

3. recall rate = recall hit ÷ Total number of queries

4. acceptance rate = take a hit ÷ Total number of queries

5. relative lift rate =(Experimental group indicators ÷ Control group indicators)−1

￮ caliberARound Arithmetic Average: Each round calculates the promotion rate separately, and then compares7Round averaging, used to observe gain fluctuations

￮ caliberB: Total combined comprehensive promotion rate: will7175All hit data of the second query are summarized and calculated uniformly, which is the main conclusion indicator of this report to avoid false gain caused by low baseline of single round.

2 Single round experimental data (WM0924, No.7Round)

Single-round query cardinality:1025next

surface1 Recall by platform/Collection (5Model consolidation)

WM0924Batch Summary Accounting

• Atomic Facts: Recall70|accepted34|recall rate6.829%|acceptance rate3.317%

• Long baseline: recall12|accepted10|recall rate1.171%|acceptance rate0.976%

• Recall Relative Promotion:483.2%

• Relative promotion of credit collection:239.9%

surface2 submodel recall/Collection (single model205query, total1025Time)

model (plural when used as a product category name)	query round	recall hit	take a hit	recall rate	acceptance rate	recall base conversion rate	model classification
shoe-shaped gold ingot	205	0	0	0.00%	0.00%	—	invalid search layer
thousand query	205	43	6	20.976%	2.927%	13.95%	High sensitivity calibration layer (high recall, very low acceptance)
Wen Xin	205	3	2	1.463%	0.976%	66.67%	highly stable adaptation layer
Kimi	205	0	0	0.00%	0.00%	—	invalid search layer
bean bag	205	36	36	17.561%	17.561%	100.00%	High stability adaptation layer (recall i.e. acceptance)

3 Summary table of seven rounds of total indicators

Base number of queries per round:1025next

batches	form	recall hit	take a hit	recall rate	acceptance rate	recall relative lift	Relative improvement in acceptance
TC11first edition	atomic facts	55	14	5.366%	1.366%	244.0%	27.3%
TC11first edition	long article	16	11	1.561%	1.073%	base line	base line
WM0919(A)	atomic facts	77	38	7.512%	3.707%	352.9%	153.4%
WM0919(A)	long article	17	15	1.659%	1.463%	base line	base line
WM0920(B)	atomic facts	67	33	6.537%	3.219%	219.0%	73.6%
WM0920(B)	long article	21	19	2.049%	1.854%	base line	base line
WM0921	atomic facts	84	44	8.195%	4.293%	600.0%	388.9%
WM0921	long article	12	9	1.171%	0.878%	base line	base line
WM0922	atomic facts	85	41	8.293%	4.000%	212.6%	194.9%
WM0922	long article	4	2	0.390%	0.195%	base line	base line
WM0923	atomic facts	75	38	7.317%	3.707%	1150.8%	1165.2%
WM0923	long article	6	3	0.585%	0.293%	base line	base line
WM0924	atomic facts	70	34	6.829%	3.317%	483.2%	239.9%
WM0924	long article	12	10	1.171%	0.976%	base line	base line

4 Seven rounds total global statistics (based on all7175Second inquiry)

4.1 hit summary

• Atomic facts total recall hits:55+77+67+84+85+75+70 = 513

• Atomic Facts Total Accredited Hits:14+38+33+44+41+38+34 = 242

• Long text total recall hit:16+17+21+12+4+6+12 = 88

• Long article total collection hit:11+15+19+9+2+3+10 = 69

4.2 Total consolidated indicator (report main indicator)

• Total atomic fact recall = 513 ÷ 7175 = 7.150%

• Total atomic facts total acceptance rate = 242 ÷ 7175 = 3.373%

• Total recall rate of full length articles = 88 ÷ 7175 = 1.226%

• Total acceptance rate of all long-term articles = 69 ÷ 7175 = 0.962%

✅ Combined recall promotion rate:483.2%

✅ Comprehensive letter acceptance rate:250.6%

4.3 Run Arithmetic Mean Caliber (for fluctuation analysis)

7Round recall rate:244.0%、352.9%、219.0%、600.0%、212.6%、1150.8%、483.2%

7Single round letter collection rate:27.3%、153.4%、73.6%、388.9%、194.9%、1165.2%、239.9%

• Arithmetic Average of Seven-Round Single-Round Recall Promotion Rate:466.6%

• Arithmetic average of seven-round single-round letter collection rate:320.5%

calibre description

1. Total combined comprehensive promotion rate (recommended main conclusion): based on all7175Article query summary statistics, eliminate gain amplification effect caused by low baseline of single round control group, objectively reflect the overall performance of the system;

2. Round Arithmetic Average: Directly calculate the average of the percentage increase calculated independently for each round, which is used to observe the gain fluctuation characteristics of multiple rounds of retest, and is not suitable as the core conclusion.

5 Seven-round stability, attribution and full-cycle trend analysis

Batch sequence:TC11first edition → WM0919(A) → WM0920(B) → WM0921 → WM0922 → WM0923 → WM0924; Fixed cardinality per round of queries1025next

batches	atomic fact recall	atomic fact acceptance rate	Long article recall	Long article acceptance rate	recall relative lift	Relative improvement in acceptance
TC11first edition	5.366%	1.366%	1.561%	1.073%	244.0%	27.3%
WM0919(A)	7.512%	3.707%	1.659%	1.463%	352.9%	153.4%
WM0920(B)	6.537%	3.219%	2.049%	1.854%	219.0%	73.6%
WM0921	8.195%	4.293%	1.171%	0.878%	600.0%	388.9%
WM0922	8.293%	4.000%	0.390%	0.195%	212.6%	194.9%
WM0923	7.317%	3.707%	0.585%	0.293%	1150.8%	1165.2%
WM0924	6.829%	3.317%	1.171%	0.976%	483.2%	239.9%

5.1 Experimental Stability Conclusion

1. Seven rounds of zero reversal: cumulative7175The recall and acceptance indexes of all rounds of experimental group (atomic fact) were higher than those of control group. The conclusion had statistical reliability and there was no reverse deterioration.

2. Gain interval full positive direction: recall lift interval212.6%1150.8%and raise the bar.27.3%1165.2%。

3. The hierarchical structure of the model is locked in the whole process, without drift and turnover; the black and white list of channels is stable and unchanged, and the retrieval channel converges.

5.2 root cause localization

The single-round relative lift index fluctuates violently, which is caused by the change of snapshot weight of external search engine, which leads to the fluctuation of long-term baseline of control group, not the performance fluctuation of Vientiane Yuandrive system itself. When the baseline value of control group is very low, the single-round relative lift rate will be amplified, so it is preferred to use it.7175The second query combines comprehensive indicators as the basis for overall evaluation of the system.

The recall and acceptance of atomic facts remain stable, which proves that the retrieval enhancement ability of structured atomic facts belongs to endogenous stability and is not affected by the fluctuation of external snapshots of search engines.

5.3 Sub-trend interpretation

5.3.1 Atomic facts (absolute indicators of experimental groups) trend

Overall trend: the initial version started at a low level, the second round rose rapidly, and the subsequent stability remained at6.8%~8.3%Steady state interval, no systematic attenuation.

• TC11initial recall rate5.366%The lowest of seven rounds;

• WM0919(A)The index rises and enters the steady state platform;

• WM0921、WM0922Reach the experimental group index high point (recall8.195%、8.293%）；

• WM0923、WM0924Small drop, belong to normal small fluctuation, do not belong to performance degradation.

Acceptance rate synchronization performance: first round1.366%lowest, second round steady fall3.2%~4.3%Intervals, the ability to collect information continues to be stable.

conclusion: Endogenous retrieval enhancement of structured atomic facts, in7Wheel parallel retest (cumulative7175The second query) completes the convergence and enters the steady state plateau period.

5.3.2 Changwen baseline (absolute indicator of control group) trend

The long-term baseline as a whole shows a downward fluctuation trend:

TC11first edition(1.561%) → WM0919(A)(1.659%) → WM0920(B)(2.049%)peak → Continued decline:WM0921(1.171%) → WM0922(0.390%)infraversion → WM0923rebounded slightly to0.585% → WM0924rebounded to1.171%。

Changwen baseline is an external variable, affected by search engine index snapshot weight, not affected by Vientiane drive/Atomic fact paradigm control.

5.3.3 Model Stratification Trends

Single wheel fixation per model205Second query,5Total model1025query/The wheel.

• Invalid search layer: Yuanbao,KimiFull recall.=0；

• High sensitivity calibration layer: thousand questions, high recall, very low acceptance, stable characteristics;

• High stability adaptation layer: Wenxin, bean buns; bean buns will be accepted immediately after recall, conversion rate100%。

The model hierarchy converges after the first round of experiments, and the subsequent rounds do not change the model hierarchy. The model call strategy can be solidified for mass production.

5.3.4 Channel Trends

Channel black-and-white list converges and stabilizes throughout:

• Permanent blind zone: official website, public number, headline number, seven rounds of continuous recall for0It is a structural non-retrievable channel.

• Effective channels: Baijia, Sohu, Netease. Sohu contributed most recalls and letters; Netease's call was acceptable, but the conversion rate of letters was low; Baijia only hit sporadically.

The channel distribution characteristics do not drift in multiple rounds of retest, and the channel weight allocation strategy can be fixed.

5.4 trend total judgment

Seven rounds of retest showed the characteristics of atomic fact absolute index steady state platform, long text baseline continuously fluctuating with search engine environment, single round relative gain inversely amplifying with baseline; endogenous search enhancement ability was stable without attenuation, model stratification, channel characteristics completed convergence in the early stage based on all7175Second query summary, structured atomic facts compared to traditional long text materials,Comprehensive recall upgrade483.2%Comprehensive credit enhancement250.6%。

Drawing advice (line chart4curve) 1. curve1Atomic fact recall rate (high level, stable and small oscillation) 2. curve2: Long article recall rate (fluctuation downward) 3. curve3: recall relative ascension (violent shock,WM0923peak) 4. curve4: Relative increase in acceptance (violent shock,WM0923peak)

6 Core Experiment Conclusion

1. This experiment was carried out by Beijing Omedya Technology Co., Ltd. Vientiane Mirror·ChinaAICognitive Governance Laboratory Development, Cumulative Implementation7175The structured atomic facts of experimental group were better than those of traditional long-text control group in all rounds, and zero negative inversion was found to verify the validity of the scheme.

2. Total query based merge statistics (main conclusion indicator): structured atomic facts compared to traditional long-text web materialComprehensive recall upgrade483.2%Comprehensive acquisition and promotion250.6%If the round arithmetic average caliber is adopted, the seven-round average recall promotion is adopted.466.6%Average acquisition promotion320.5%. It is recommended to give priority to the use of full-volume consolidation indicators to avoid false gains caused by low single-round baselines.

3. The Qur 'an (MDSEYuan drive casting)+Structured atomic facts retrieval enhancement architecture, with stable endogenous retrieval gain, not affected by external fluctuations in search engine snapshots, strong system robustness, can adapt to real Internet dynamic retrieval environment.

4. The model layering, channel screening and long and short material comparison strategies are all converged and solidified, the technical route of the scheme is stable, and the conditions for project landing and large-scale mass production are met.

7 Discussion and explanation

1. The sharp fluctuation of relative lift rate of single round is caused by the change of snapshot weight of external search engine index and the fluctuation of baseline of long article in control group, not the performance change of this retrieval enhancement system.

2. The experiment strictly followed the scientific experimental principles of single variable control, random allocation, repeatable retest and confounding factor identification control. Only the material type was changed every time, and the other experimental conditions remained unchanged to ensure the reliability of attribution.

3. The source of materials is limited to Sohu and Netease, and the noise channels of Baijia, Headline and Official Website are eliminated to reduce the interference of low-quality web page noise to the control group.

4. The acceptance index depends on the recall success, and the acceptance rate is based on the recall success, which conforms to the business logic of retrieval enhancement task.

8 Appendix:PPTReport streamlined golden sentence (update7Round version)

TC11Seven-round parallel authority retest, by Beijing Omei Diya Technology Co., Ltd. Vientiane mirror·ChinaAICognitive Governance Lab complete, cumulative7175Second query verification: Vientiane element drive retrieval enhancement system enables structured atomic facts to achieve seven rounds of data zero inversion and full dimensional positive gain; based on the summary calculation of all query samples, structured atomic facts compare with traditional long text content,Comprehensive recall upgrade483.2%Comprehensive acquisition and promotion250.6%Multi-round iteration model hierarchy is fully solidified, channel characteristics are highly convergent, and structured facts are driven.AIThe technical route of retrieval enhancement is stable and has standardized mass production capability.