Timber Blind Prediction Challenges / Methods
A prediction made before
the result is released.
Challenge records keep the experimental brief, submission timing, scoring definitions and source permissions inspectable. A challenge comparison describes performance on the recorded test; it does not certify a model or establish design suitability.
What makes a challenge blind
The organizer supplies a test whose measured response is withheld from participants, records its source and sharing permission, and describes prior access and potential conflicts. Opening the challenge freezes the public case brief, deadline, private reference and scoring rules.
The website owner and infrastructure administrators can access stored reference data. The system restricts participant access; it is not end-to-end encryption or proof that no participant has previously seen the result. Calibration and prior knowledge remain explicit contributor declarations.
Published experiments remain practice benchmarks. Hiding a published curve in this interface does not make it a new blind experiment. No laboratory partnership or unpublished dataset is implied by the availability of this software.
Phases and frozen records
| Phase | Allowed activity | Participant access |
|---|---|---|
| Draft | The owner prepares and checks the brief and private data. | The draft is private to the owner. |
| Open | Approved academic members lock one prediction per account and challenge. | The brief and their own frozen submission; no measured curve or scores. |
| Closed | The deadline prevents new predictions. The owner prepares the release. | Existing receipts stay available; measured results remain hidden. |
| Revealed | The owner explicitly releases the reference and freezes prediction reports. Replication submissions can begin. | The released reference; their own report; separately published results from others. |
| Cancelled | A public reason records why the challenge stopped. Frozen records are retained. | Access remains subject to the recorded release state. |
The server clock determines the deadline. A local preview is editable; a locked submission cannot be replaced. Check units, column mapping, metadata and publication consent before confirming. Approval and public result release are separate owner actions.
Frozen inputs and reports are retained. Later explanations use appended public notes; changes to the experiment or scoring require a new challenge. Cancellation does not erase its history.
Reference commitments and receipts
The public case fingerprint identifies the fixed brief. A salted reference commitment is published when the challenge opens; its random nonce and the reference record stay private until reveal. After reveal, the reference and nonce allow the commitment to be checked.
Fingerprint format version 1 uses UTF-8 compact JSON with object keys sorted in binary UTF-16 order, original array order and finite numbers serialized using JSON.stringify. The reference commitment hashes an object with the schema value “lingoscholar-private-reference-commitment/1”, the nonce and the reference. The released JSON record supplies those exact inputs. Submission receipts record their input fingerprint and server timestamp. Released reports retain the fixed case, reference, submitted inputs, method version and computed values. Use “Verify released record” on a revealed case to check its brief, reference and salted commitment locally. Fingerprints identify matching contents; they do not prove scientific accuracy, authorship or independent custody of the database.
Curve metrics
Version 1 supports strictly increasing displacement under explicitly monotonic loading. Displacement is stored in mm and force in kN. Cyclic histories, displacement reversals and repeated displacement coordinates are rejected; the system does not sort or collapse them into a monotonic curve.
Force and displacement definitions must match the fixed case. The case specifies whether peak force refers to the full supplied trace or only the fixed displacement interval. Secant stiffness uses the first 10–40% ascending-branch crossings where that definition is applicable; an ambiguous or unavailable value remains unavailable.
RMSE is computed by exact integration of the squared difference between piecewise-linear curves over the complete fixed comparison interval. Predictions must cover that interval; partial coverage withholds RMSE and interval assessments. No extrapolation is used. Relative force errors with a zero reference denominator remain unavailable.
Results retain calibration and prior-knowledge declarations. There is no universal combined ranking across different tests, scoring domains or nominal interval levels.
Prediction intervals
A contributor may supply lower and upper pointwise force bounds at the same displacement coordinates as the central prediction. The bounds must contain the central prediction and retain their stated nominal coverage. These are force intervals at prescribed displacement, rather than uncertainty in displacement.
For a nominal coverage of 1 − α, the pointwise interval score is:
The report integrates this score over the fixed displacement interval and divides by that interval's span. It also reports mean interval width in kN and the fraction of the displacement span where the observed curve lies inside the bounds. Piecewise-linear crossings are included when computing these quantities.
A smaller interval score favors useful narrow intervals while penalizing missed observations at the same nominal coverage. Coverage along one force–displacement curve is a descriptive geometric quantity; it is not a confidence level, evidence of statistical calibration, or a set of independent experimental trials.
Comparing intervals across specimens
The multi-case comparison uses explicitly selected, published prediction reports with the same nominal coverage and one selected result per case version. Its averages weight the selected cases equally; they are not a pooled material-property estimate or confidence statement.
Different specimens can share a campaign, material batch, laboratory or calibration data. Selection does not establish independence. Inspect case conditions and contributor declarations, preserve the selected report identifiers, and state why the cases were chosen. Unpublished and unrevealed results are excluded.
Replication rounds
After reveal, an approved academic member can submit a new observed curve with its campaign, material, test conditions, source, permission statement and differences from the original case. The contributor declares whether it came from a separate campaign. The original case supplies a common comparison interval and definitions.
A replication report describes differences between two recorded experiments. Review checks source metadata, permission and completeness. Approval alone does not establish that another laboratory independently reproduced the finding. Public release remains separate from approval, and every stated difference stays with the report.
Current scope
Each challenge supports up to 50 prediction entries and 50 replication entries, with one of each per academic account. Reference and submitted curves contain at most 1,000 points. These limits keep the frozen comparison packages bounded. A new experiment or protocol uses a new challenge record.
The existing published-data benchmarks are available for practice in the Timber Benchmark Exchange. Actual blind challenges require an authorized withheld dataset and an organizer willing to release it under the recorded terms.