AI Experiment Design

When putting your research into a paper, you need to show the effectiveness of your proposed method not only theoretically but also experimentally.

The experiment logic archetype

The experiment logic should be something like above.

(1) Problem interpretation

The problem is a central idea of your research. We first need to try to convert a set of experiment specifications which represent the problem very well.

Example:

• Problem: Acquiring chemical sensor data is costly; therefore the data points are extremely small. We need to make a robust, high-accuracy predictor based on such a small number of data.
• Interpretation: The problem frame is predictive modeling. The data-set must be quite similar to the actual chemical sensor outputs.

The problem interpretation should be as clear as possible (for others to do exactly the same experiment).

(2) Premise clarification

In the case above, the premise is as follows.

• The central premise is about data-set. If we want to say that it’s useful in chemical sensor data analysis, we should prepare at least several data sets to justify the methods. Describe the nature of the data well.
  e.g. Dimension. Time-series? Vector/Scalar?
• Some other premises should be listed.
  e.g. The computer spec should be limited, or GPU is usable.
• If it requires system implementation to fully simulate the sensor connection, we should have the system architecture and clarification of input/output, too.

(3) Criteria clarification

• We should have other criteria to evaluate. Single criterion, accuracy, is not always very good.
  e.g. Training Speed, Execution Speed, Coverage of words, etc.
• The measurement should follow some related works, STRICTLY, especially in the top conferences.

(4) Baseline

The baseline should be some well-known methods or the easy combination of well-known methods.

The description of the baseline should also be precise. For example, if there’re the needs of parameter tuning, the method of identifying the parameters should also be precisely written for the reproducability purpose.

(5) Comparison

If the proposed method is the combination of X, Y, and Z, the comparison could be as follows.

• Baseline
• Proposal: X + Y + Z
• X + Y
• X + Z
• Y + Z

Those are the validation of each element (X, Y, Z) to surely contribute to the proposed method. Or, maybe the comparison could also be like

• Baseline + Y
• Baseline + Z
• Baseline + Y + Z

Design the experiments first

You can design the experiment first, then in the implementation, you can imagine how the module X, Y or Z should be coded, or how should the data be annotated.