[Freeciv-Dev] Re: Split patch (was Re: [RFC PATCH] init_techs)

["Ross W. Wetmore" <rwetmore@xxxxxxxxxxxx>]

At 11:20 PM 01/09/30 +0200, Reinier Post wrote:
>On Fri, Sep 28, 2001 at 10:15:40PM -0400, Ross W. Wetmore wrote:
>> At 09:05 AM 01/09/28 +0200, Reinier Post wrote:
>> >On Fri, Sep 28, 2001 at 12:00:07AM -0400, Ross W. Wetmore wrote:
>> >> split should treat the buffer it was handed as working memory, return
>> >> pointers into the parsed string elements, and let the caller deal with
>> >> ALL memory issues.
>> >> 
>> >> It is really the only sensible general purpose solution for something 
>> >> like this.
>> >
>> >Only if you do not have to expand the input while tokenizing it.
>> >Then you can change whitespace to NUL a la strtok() either on the
>> >original or on a copy.  I think Freeciv will meets this case
>> >regardless of the extended command syntax Justin decides on.
>> >
>> >-- 
>> >Reinier
>> 
>> I think the restriction is that you always have to separate tokens by
>> at least some char sized element in the initial buffer.
>
>Yes, this is a restrictition if you want to split the input into tokens
>before doing any interpretation on them.  The problem is single-char tokens
>that do not have required surrounding whitespace, such as ";".
[...]
>> Note this is an example where you don't want to strdup() the initial 
>> split() elements, and may not need to do this for the lookup values 
>> depending on how the overall algorithm was designed. So extra 
>> strdup()/free() calls just waste performance cycles here.
>
>Yes.  I think it may be better to allow each stage to touch the original
>buffers (as strtok() does) as long as it undoes them before returning.

If you want to do this AND deal with a general language system that has
elements like ";" which you need to return, as opposed to treat as 
token separators, then split should return "counted strings" and not 
the usual "null-terminated strings". This is a slightly better way that
allows you to guarantee the const nature of the buffer at all times.

The return value then includes an array of offsets and lengths (or start 
and end buffer position), rather than simply an array of pointers.

This still leaves all memory management issues at the same (caller)
level, rather than imposing cumbersome and error prone restrictions.
Callers can use strncpy() or strndup() to extract items, or add their
own nulls if they are working in a system that permits such things.

But I think it has been said many times that such flexibility is not
required for Freeciv. It would be a wise move to really look at the
pros and cons before introducing such parse elements into any future 
command syntax design.

>-- 
>Reinier

Cheers,
RossW
=====