invalid time - clock adjusted forward

[pamela fluente]

Hi friend, take the following simple code
It raises the exception:
The supplied DateTime represents an invalid time. For example,
when the clock is adjusted forward, any time in the period that is
skipped is invalid.
Parameter name: dateTime

Could i have some enlightening on this? (should be related with
daylight)
And how should i modify my loop to proceed normally in the generation
of new dates avoiding the error ?
[BTW i am generating time series of random prices, this is time, the
"x" axis]

--------------------------
Public TimeZoneInfo_LOCAL As TimeZoneInfo = TimeZoneInfo.Local
Public TimeZoneInfo_EDT As TimeZoneInfo =
TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time")
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles Button1.Click
Dim d As New Date(2011, 3, 1)
Do
Dim NewDate = d.AddMinutes(1)
Dim NewDate_loc As Date =
TimeZoneInfo.ConvertTime(NewDate, TimeZoneInfo_EDT,
TimeZoneInfo_LOCAL)
d = NewDate
Loop
End Sub
--------------------------

I am looking for a way to have the loop proceed normally generating
the dates
NewDate_loc dates (a gap of 1 hour is fine, no problem).

-Pam

 

[Armin Zingler]

Am 19.03.2011 20:31, schrieb pamela fluente:

Hi friend, take the following simple code
It raises the exception:
The supplied DateTime represents an invalid time. For example,
when the clock is adjusted forward, any time in the period that is
skipped is invalid.
Parameter name: dateTime

Could i have some enlightening on this? (should be related with
daylight)
And how should i modify my loop to proceed normally in the generation
of new dates avoiding the error ?
[BTW i am generating time series of random prices, this is time, the
"x" axis]

--------------------------
Public TimeZoneInfo_LOCAL As TimeZoneInfo = TimeZoneInfo.Local
Public TimeZoneInfo_EDT As TimeZoneInfo =
TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time")
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles Button1.Click
Dim d As New Date(2011, 3, 1)
Do
Dim NewDate = d.AddMinutes(1)
Dim NewDate_loc As Date =
TimeZoneInfo.ConvertTime(NewDate, TimeZoneInfo_EDT,
TimeZoneInfo_LOCAL)
d = NewDate
Loop
End Sub
--------------------------

I am looking for a way to have the loop proceed normally generating
the dates
NewDate_loc dates (a gap of 1 hour is fine, no problem).

As you already wrote, due to daylight saving time, not every point in time
is a valid local time. With EST, daylight saving time seems to start at
3/13/2011 2:00:00 AM. If the clock is set 1 hour ahead, the sequence is:

3/13/2011, 01:58 AM
3/13/2011, 01:59 AM
3/13/2011, 03:00 AM
3/13/2011, 03:01 AM

So, if you're trying to convert any time between
3/13/2011, 02:00 AM (inclusive) and 3/13/2011, 03:00 AM (exclusive),
the error occurs. Any point in time in this timespan simply does
not exist in that time zone.


On the other hand, whenever DST ends, there's an ambiguous timespan.
The clock is set back one hour on 11/6/2011, 02:00AM. Consequently the
sequence is

11/6/2011, 01:00 AM
11/6/2011, 01:01 AM
....
11/6/2011, 01:58 AM
11/6/2011, 01:59 AM
11/6/2011, 01:00 AM
11/6/2011, 01:01 AM
11/6/2011, 01:02 AM

You have ambiguous local times within 1 AM and 2 AM. Have a look at
TimeZoneInfo.GetAmbiguousTimeOffsets and the documentation of
TimeZoneInfo.ConvertTime how this is handled. IIRC I read something
about it there.

The resolution is to only work with UTC internally. You can
still display the local time. Or, use DateTimeOffset instead
of DateTime; then it's also always convertible.

 

[Armin Zingler]

Am 19.03.2011 21:14, schrieb Armin Zingler:

The resolution is to only work with UTC internally. You can
still display the local time. Or, use DateTimeOffset instead
of DateTime; then it's also always convertible.

In other words, if you've collected data at 11/6/2011, 01:30 (local time),
how do you know if it was before or after adjusting the clock?
This problem is because you use the DateTime type which is not appropriate
for these purposes. It only says "local time", but it has no offset.
That's why DateTimeOffset has been introduced.

 

[Armin Zingler]

Am 19.03.2011 21:24, schrieb Armin Zingler:

Am 19.03.2011 21:14, schrieb Armin Zingler:

In other words, if you've collected data at 11/6/2011, 01:30 (local time),
how do you know if it was before or after adjusting the clock?
This problem is because you use the DateTime type which is not appropriate
for these purposes. It only says "local time", but it has no offset.
That's why DateTimeOffset has been introduced.

Sorry, me again....
You can still use DateTime, but with Kind=UTC. Then you can associate
your date with these DateTimes.

 

[pamela fluente]

Am 19.03.2011 21:24, schrieb Armin Zingler:



Sorry, me again....
You can still use DateTime, but with Kind=UTC. Then you can associate
your date with these DateTimes.

Thanks a lot Armin. Very helpful.

I am also researching. Looks live using the IsInvalidTime() of the
timezone
i can easily skip on the hour.

Seems to work fine so far. Let me know if you see potential dangers.

-Pam

 

[Armin Zingler]

Am 19.03.2011 21:31, schrieb pamela fluente:

Thanks a lot Armin. Very helpful.

I am also researching. Looks live using the IsInvalidTime() of the
timezone
i can easily skip on the hour.

Seems to work fine so far. Let me know if you see potential dangers.

How do you intend to handle the ambiguous times?

 

[pamela fluente]

Am 19.03.2011 21:31, schrieb pamela fluente:




How do you intend to handle the ambiguous times?

Thanks a lot Armin. Very insightful.
I am not sure what you mean by ambiguous.

Let me clarify what i am doing, so will be clearer wheter i still
will have problems.

Take and InitialDate. an EDT (est time) date.
Starting form that InitialDate i generate a sequence of (date, price)

The sequence is pretty tight, being the date spaced a few
milliseconds.

The sequence (al est times) run up to another date as to cover N days.

InitialDate ---------------- InitialDate +N days

I use this series for various simulation.

Now while i am plotting this series (x = time y=price) i also
bother, tick by tick, showing what would be the time in LOCAL
timezone.

So i noticed that for some special selection of InitialDate and N i
found this error.
Now, to correct, i am thinking just to skip (in the loop) the invalid
times.


-Pam

 

[Armin Zingler]

Am 19.03.2011 22:09, schrieb pamela fluente:

Thanks a lot Armin. Very insightful.
I am not sure what you mean by ambiguous.

I mean that for example 11/6/2011, 01:30 AM (local time) occurs twice.

Let me clarify what i am doing, so will be clearer wheter i still
will have problems.

Take and InitialDate. an EDT (est time) date.
Starting form that InitialDate i generate a sequence of (date, price)

The sequence is pretty tight, being the date spaced a few
milliseconds.

The sequence (al est times) run up to another date as to cover N days.

InitialDate ---------------- InitialDate +N days

I use this series for various simulation.

Now while i am plotting this series (x = time y=price) i also
bother, tick by tick, showing what would be the time in LOCAL
timezone.

So i noticed that for some special selection of InitialDate and N i
found this error.
Now, to correct, i am thinking just to skip (in the loop) the invalid
times.

Where does InitialDate come from? Do you verify that it's a valid
and unambiguous time?
I would first convert the initial DateTime to UTC. From there
you can compute the UTC time at every x-position, and from every
UTC time, you can always and unambiguously compute the local time.

 

[pamela fluente]

Am 19.03.2011 22:09, schrieb pamela fluente:



I mean that for example 11/6/2011, 01:30 AM (local time) occurs twice.











Where does InitialDate come from? Do you verify that it's a valid
and unambiguous time?
I would first convert the initial DateTime to UTC. From there
you can compute the UTC time at every x-position, and from every
UTC time, you can always and unambiguously compute the local time.

Thanks Armin. I see.

If the local time appers 2 times it would not hurt. What hurts and is
a big problem is if the EDT time duplicates.
This is because for processing i am actually using the EDT time. While
LOCAL time is meant for illustration purposes only (just display).

Do you think i will still run into troubles?

-Pam

 

[pamela fluente]

i am trying running this, to verify:
it seems to get through ...


Private Sub Button1_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles Button1.Click


Dim d As New Date(2011, 3, 1)
Dim PreviousEDTDate As Date


Do

Dim NewDate = d.AddDays(1)

If Not TimeZoneInfo_EDT.IsInvalidTime(NewDate) Then

Dim NewDate_EDT As Date =
TimeZoneInfo.ConvertTime(NewDate, TimeZoneInfo_EDT,
TimeZoneInfo_LOCAL)

If NewDate_EDT < PreviousEDTDate Then
MsgBox("Time EDT backed up " & d & " -> " &
NewDate)
End If

If NewDate < d Then
MsgBox("Time backed up " & d & " -> " & NewDate)
End If

d = NewDate
PreviousEDTDate = NewDate

End If

If d > DateTime.MaxValue.AddMonths(-1) Then Exit Do

Loop

MsgBox("Done")

End Sub

 

[Armin Zingler]

Am 19.03.2011 23:00, schrieb pamela fluente:

i am trying running this, to verify:
it seems to get through ...


Private Sub Button1_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles Button1.Click


Dim d As New Date(2011, 3, 1)
Dim PreviousEDTDate As Date


Do

Dim NewDate = d.AddDays(1)

If Not TimeZoneInfo_EDT.IsInvalidTime(NewDate) Then

Dim NewDate_EDT As Date =
TimeZoneInfo.ConvertTime(NewDate, TimeZoneInfo_EDT,
TimeZoneInfo_LOCAL)

If NewDate_EDT < PreviousEDTDate Then
MsgBox("Time EDT backed up " & d & " -> " &
NewDate)
End If

If NewDate < d Then
MsgBox("Time backed up " & d & " -> " & NewDate)
End If

d = NewDate
PreviousEDTDate = NewDate

End If

If d > DateTime.MaxValue.AddMonths(-1) Then Exit Do

Loop

MsgBox("Done")

End Sub

In this case you're adding days. Time is always midnight.
No ambiguous or invalid DateTime values possible. Therefore
I don't know how much it helps.

You wrote what's your intention, but it's still not clear for me.
If you want a linear scale on the x-axis you have to use UTC.
You can't use a time zone with DST for the x-axis. Well, you can,
but than you have a missing hour in spring and a double hour in fall.

How'd do it if you must have an EDT time as the initial time:
(you still have to validate the EDT time before running this code)

Dim d As New Date(2011, 3, 26, 20, 30, 0) 'EDT

Dim UTCBase = TimeZoneInfo.ConvertTimeToUtc(d, TimeZoneInfo_EDT)

For i = 0 To 60
Dim dUTC = UTCBase.AddMinutes(i)
Dim dLocal = TimeZoneInfo.ConvertTimeFromUtc(dUTC, TimeZoneInfo_LOCAL)

Debug.WriteLine(dLocal.ToString)
Next

This way you don't have to care about invalid or ambiguous times.

 

[Armin Zingler]

I believe I know now what the problem is:
Your data (the price) is stored together with the EDT date/time.
Now you are trying to use EDT as the scale for the x-axis
because EDT is what you have. Is this right? If it is:

- You can _not_ have an _invalid_ EDT DateTime.
- You can _still_ have an _ambiguous_ EDT DateTime. Unless you
say this is only theory, it's plain wrong to use a time zone
that has DST.

Let's assume case 2 is really only theory, so that we can
reliably convert all times to UTC. Then you have two choices:


Version a)

in spring:

XXXXXXXXXXXXXX XXXXXXXXX
|------|------|-------|-------|

Local time: 00:00 01:00 02:00 03:00 04:00

(X = where data is possible)


in fall:
XXXXXXXXXXXXXX++++++++XXXXXXXXXXX
|------|------|-------|-------|

Local time: 00:00 01:00 02:00 03:00 04:00

(+ = statistically twice as much data)




Version b)


in spring:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|------|------|-------|-------|

Local time: 00:00 01:00 03:00 04:00 05:00

--> One hour missing.


in fall:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|------|------|-------|-------|

Local time: 00:00 01:00 02:00 02:00 03:00

--> One hour twice.


Please choose version a) or b).

 

[pamela fluente]

Thanks Armin for the beautiful and insightful elaboration. Let me
think better about it

Yes you are right i should have used addminutes(), for a good check.

The prices i generate as EDT, like if i where for instance the New
York Exchange.

Now while i do so, in spring i would create, just for an hour,
impossible instants.
But i guess i could just skip those, and proceed with the time/price
generation.

Now in Autumn let's see what happen. I am still creating EDT by
milliseconds increment.
Wouldn't this go ahead fine ? The addmilliseconds() function should
automatically take care of
date adjustment. (??) Or not, i guess that the hour would be wrapped
up causing replication in a precise hour.

Probably the best thing is just to use your suggested loop with the
UCT.

Dim d As New Date(2011, 3, 26, 20, 30, 0) 'EDT
Dim UTCBase = TimeZoneInfo.ConvertTimeToUtc(d,
TimeZoneInfo_EDT)
For i = 0 To 60
Dim dUTC = UTCBase.AddMinutes(i)
Dim dLocal = TimeZoneInfo.ConvertTimeFromUtc(dUTC,
TimeZoneInfo_LOCAL)
Debug.WriteLine(dLocal.ToString)
Next


For the initial test perhaps i might check it and add or subtract an
hour.

Or just assume that also the initial time is UTC (this is simulated
data anyway).


-Pam

 

[pamela fluente]

Actually, thinking better, i do need it to be EDT because the same
engine processes also the real data which is edt.
So i need to process edt.

As to the simulated data, i can still use the IsInvalidTime() to skip
the hole.

How could i further correct the loop to avoid duplicating that single
autumn hour ??

perhaps if time goes backward i could skip it, so that the hour is
simulated only once ?

-Pam

 

[Armin Zingler]

Am 20.03.2011 03:11, schrieb pamela fluente:

Actually, thinking better, i do need it to be EDT because the same
engine processes also the real data which is edt.
So i need to process edt.

As to the simulated data, i can still use the IsInvalidTime() to skip
the hole.

How could i further correct the loop to avoid duplicating that single
autumn hour ??

perhaps if time goes backward i could skip it, so that the hour is
simulated only once ?

-Pam

Pam, this is really hard to understand. You are not answering my
questions.
/What/ needs to be EDT? Do you understand that EDT times without
a time zone information can be ambiguous? Where do the EDT times
come from? I still don't know if this "double hour" in autumn is
only theory. Again: If you have an initial DateTime in EDT, you can
do it like shown in my loop and won't get a problem ever: Convert
the initial DateTime to UTC, and from there you can compute any
time in any time zone.

 

[Armin Zingler]

Pam,

long story short:
If you use EDT time stored in a DateTime,
your whole engine is broken!

The solutions are
- store UTC DateTime objects
- store EDT DateTimeOffset objects

 

[pamela fluente]

Pam, this is really hard to understand. You are not answering my
questions.
/What/ needs to be EDT? Do you understand that EDT times without
a time zone information can be ambiguous? Where do the EDT times
come from? I still don't know if this "double hour" in autumn is
only theory. Again: If you have an initial DateTime in EDT, you can
do it like shown in my loop and won't get a problem ever: Convert
the initial DateTime to UTC, and from there you can compute any
time in any time zone.

Hi Armin. Thanks a lot: you really helped clarify the question.

It's simple the same engine must process (datetime/price) pairs
that can be either real or created by me.

REAL
If it's real no problem should arise because in spring the time skip
an hour
ahead. The time/price series will continue without problems.
In Autumn the price might go backward. But i can simply stop new
tickdata until time
restarts going forward (the engine will stop it anyway, for security
reasons).
Further this happen on Sunday at 2:00 when markets are even closed (no
tickdata arriving). So there should be no problems.

SIMULATED
This is the case i was considering in my issue. This is time i
generate myself the dates
from a start date by adding milliseconds.
Looks like i can avoid the spring issue, just skipping the invalid
dates.
The autumn issue does not seem to exist because the time i generate
advances only
and cannot go backward (only REAL edt dates can be go backward,
because someone makes them to).

So from what i understood so far, simply ignoring the invalid
generated dates, should
do it. I am not seen any more issues thus far.

-Pam

 

[Armin Zingler]

Am 20.03.2011 15:36, schrieb pamela fluente:

Hi Armin. Thanks a lot: you really helped clarify the question.

It's simple the same engine must process (datetime/price) pairs
that can be either real or created by me.

REAL
If it's real no problem should arise because in spring the time skip
an hour
ahead. The time/price series will continue without problems.
In Autumn the price might go backward. But i can simply stop new
tickdata until time
restarts going forward (the engine will stop it anyway, for security
reasons).
Further this happen on Sunday at 2:00 when markets are even closed (no
tickdata arriving). So there should be no problems.

SIMULATED
This is the case i was considering in my issue. This is time i
generate myself the dates
from a start date by adding milliseconds.
Looks like i can avoid the spring issue, just skipping the invalid
dates.
The autumn issue does not seem to exist because the time i generate
advances only
and cannot go backward (only REAL edt dates can be go backward,
because someone makes them to).

So from what i understood so far, simply ignoring the invalid
generated dates, should
do it. I am not seen any more issues thus far.

I see, but then one hour is missing in autumn.

 

[pamela fluente]

Am 20.03.2011 15:36, schrieb pamela fluente:















I see, but then one hour is missing in autumn.

Yes, you are right and i see what you mean.
But in this case it would be weird to emulate the "reality" rolling
back an hour.

This data is just to simulate trading (strategy test purpose), so it
seems fine that I leave time just flow forward
(no need to go back an hour to emulate the "actual reality" and the
DST change) .

(in any case are event supposed to happen at "market closed")

-Pam

 

[Armin Zingler]

Am 20.03.2011 16:44, schrieb pamela fluente:

Yes, you are right and i see what you mean.
But in this case it would be weird to emulate the "reality" rolling
back an hour.

This data is just to simulate trading (strategy test purpose), so it
seems fine that I leave time just flow forward
(no need to go back an hour to emulate the "actual reality" and the
DST change) .

(in any case are event supposed to happen at "market closed")

Ok.