Re: Cash statistic fitting bug?

On Wed, Jul 31, 2013 at 11:50 +0200, Thomas Dauser wrote:
> On 07/30/2013 07:39 PM, John Houck wrote:
> > This limitation of mpfit is noted in the mpfit help page which
> > says:
> >
> >  DESCRIPTION
> >     The mpfit Levenberg-Marquardt algorithm is the default
> >     minimization algorithm used in fitting models to data.  This
> >     algorithm was derived under the assumption that the function to
> >     be minimized is the Chi-square fit-statistic.
>
> But as mpfit is the standard fit algorithm, I consider the current setup
> somehow dangerous: If you simply switch from Chi-square to Cash that
> means that you'll get wrong results if you don't adjust the
> fit-algorithm, too. Maybe some kind of warning could be added if the
> Cash statistic is evaluated with mpfit (or at some other place) to avoid
> further confusion and problems? Or even refuse to fit Cash with mpfit?
>

You're technically correct and I don't disagree, but I think
it's a matter of degree.

First, printing a warning is ineffective because users ignore
most warnings anyway, especially if the warning appears every
time a function is used.

Refusing to fit cash with mpfit is a more effective barrier,
and in fact early versions of isis behaved this way. Isis
similarly tries to prevent users from using conf with a
statistic that isn't known (to isis) to be chi-square
distributed in the appropriate way.

What happens in such cases is that users complain when isis
refuses to do something that they want to do.  Sometimes this
is a good thing because the user was ignorant and isis
prevented them from making a mistake.  Other times, the user
knows perfectly well that what they're doing isn't exactly
correct, but they want to do it anyway and are willing to take
responsibility.

The current policy is to document the issue but otherwise
assume the user knows what they're doing and let them do it.

I can make this policy stricter by generating a warning or an
error and then allow the user to over-ride it. But if I allow
an override, then many people will put the override in their
$HOME/.isisrc and never think of it again. Or they'll use a
$HOME/.isisrc they got from someone else -- with the override
in place -- and never see the warning at all.

Something like this latter situation has happened already with
Minimum_Stat_Err. A certain popular .isisrc has been widely
circulated, changing the default to Minimum_Stat_Err=1.e-30.
For some people that's a convenient default, but for others
that has caused frequent confusion because it leads to
surprising results.  

The point is that user-adjustable defaults sometimes don't have
the effect that I intend.

Anyway -- perhaps I'll implement a warning that can be
overridden. In that case, the user might be warned at least
once at run-time, in addition to the provided documentation.

Thanks,
-John
----
You received this message because you are
subscribed to the isis-users list.
To unsubscribe, send a message to
isis-users-request_at_email.domain.hiddenwith the first line of the message as:
unsubscribe