Any forward bias with a diode, even a very small applied voltage, will lead to some diode current.
Taking a look at your data table, it appears to me that your measurements with smaller applied voltages were off a bit -- the voltage across the resistor and across the diode do not add up well. They mismatch the applied voltage, when summed. This may be due to the voltmeter's loading on the circuit at those lower voltages.
To form an approximate model, I assumed a saturation current of \$90\:\text{pA}\$ for the diode (looking at a model for the 1N4007) and found an estimated emission coefficient of 1.297 for the larger applied voltage results in your table. (I preferred those.)
Here are the results:
def fna(v,r=1e3,isat=90e-12,ne=1.297,vt=25.865e-3): return v-r*ne*vt/r*LambertW(r*isat/ne/vt*exp(v/ne/vt))fna(1)0.519648879246456 # ~ 520 mV across the diode (~ 480 uA)fna(0.8)0.503467209759930 # ~ 503 mV across the diode (~ 300 uA)fna(0.5)0.446204017739379 # ~ 446 mV across the diode (~ 54 uA)fna(0.4)0.389944201719220 # ~ 390 mV across the diode (~ 10 uA)Which suggests that the model is consistent with the actual diode you have.
The Shockley diode equation is what I used as a base to get the above formula. It says that the diode current is exponentially related to the diode voltage.
Put another way for your diode case, every time you increase the diode voltage by \$\approx 77\:\text{mV}\$ you will get about \$10\times\$ the current. If you look at the formula results printed above, note that the voltage difference taken between the results for fna(1) and the results for fna(0.5) is \$74\:\text{mV}\$ and that the change in estimated diode current is very close to a factor of 10 between them.
A diode isn't an abrupt device that suddenly changes its behavior. It's a relatively smooth, though highly non-linear, behavior.
