[Freeciv-Dev] Re: Graphic resource editor, and an architecture proposal
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
Eric S. Raymond wrote:
As promised, I have written a resource editor for flags, shields, and graphics.
A copy of the 0.1 version is enclosed with this note. It took me about a day.
In the process of writing this editor, I have become unhappy with some
aspects of the FreeCiv data architecture. Overall it is a good design
-- good enough that I will be using it as a case study in my next
book, "The Art Of Unix Programming". Here is what I have to say about it
in the draft:
<sect2><title>Case study: FreeCiv data files</title>
<para>In Chapter &multiprocesschapter; we exhibited the FreeCiv
strategy game as an example of client-server partitioning. This game has
another notable architectural feature; much of the game's fixed data,
rather than being wired into the server code, is expressed in a
property registry read in by the game server at startup time.</para>
And much of the data is both wired into the server/client code and put
into the ruleset registry. The idea, I suppose, is that the code
support will follow later. An example is of city improvements.
Other data like terrain types is only hardwired into the code, AFAIK.
<para>The game's registry files are written in a textual data-file
format that assembles text strings (with associated text and numeric
properties) into various internal lists of important data (such as
nations and unit types) in the game server. The minilanguage has an
include directive, so game data can be broken up into semantic units
(different files) that are each separately editable. This design
choice has been carried through to such an extent that it's possible
to define new nations and new unit types simply by creating new
declarations in the data files, without touching the server code at
all.</para>
<para>The FreeCiv's server's startup parsing has an interesting
feature which creates something of a conflict between two of Unix's
design rules, and is therefore worth closer examination.
The server ignores property names it doesn't know how to use.</para>
I've seen code that indicates the client will, on demand (IIRC, when
DEBUG is defined) spew warning messages for tileset entries it doesn't
know about. I don't know if the server does the same thing for
rulesets, but I imagine it does (it would be very easy to add IIRC).
This sort of on-demand checking is good for developers since in
debugging their data files, while non-intrusive for end-users.
But only if you know it's there. And I suspect most people don't...
<para>This makes it possible to declare properties that the server
doesn't yet use without breaking the startup parsing. It means that
development of the game data (policy) and the server engine
(mechanism) can be cleanly separated. On the other hand, it also
means startup parsing won't catch simple misspellings of attribute
names. This quiet failure seems to violate the Rule of Repair.</para>
<para>To resolve this conflict, notice that it's the server's job to
<emphasis>use</emphasis> the registry data, but the task of carefully
error-checking that data could be handed off to another program to be run
by human editors each time the registry is modified. The ideal Unix
solution would be a separate auditing program that analyzes the source
of the server code to determine the set of properties it uses, parses
the FreeCiv registry to determine the set of properties it provides,
and prepares a difference report.</para>
I don't think the ideal unix solution would do any source code analysis.
This seems to fly in the face of all concepts of simplicity.
IMO (though I am not an expert), the unix design would be where the
ruleset format was a well-specified standard, and a completely
independent program would be able to verify this.
In a world of changing ruleset formats, the freeciv solution would
probably be to add another (small) file server/verify.c, and have this
file (linked with the rest of the server code) compile into a "verify"
program to do the work.
<para>The aggregate of all FreeCiv data files is functionally similar
to a Windows registry, and even uses a syntax resembling the textual
portions of registries. But the creep and corruption problems we
noted with the Windows registry don't crop up here because no program
(either within or outside the FreeCiv suite)
<emphasis>writes</emphasis> to these files. It's a read-only
registry edited only by the game's maintainers.</para>
<para>The performance impact of data file parsing is minimized because
this operation is performed only once, at server startup time.</para>
</sect2>
As an aside: it sounds to me like you're confusing the server and the
client a little bit. The server reads the ruleset data, and sends some
of it to the client as needed. The client reads the tileset. But much
of the tileset depends on the ruleset data - for instance the ruleset
specifies the name of the sprite to use for each unit. This is sent to
the client when the game starts, and consequently the named sprites
cannot be initialized before then.
The system used is in some ways very elegant. We load all sprites and
put them into a hash based upon their name. Then when we receive the
ruleset data all we have to do is look up the proper sprite for each
element. Most of the ruleset elements have two tags specified - a
primary and an alternate - so we can fall back to a default if
necessary. To some extent, this allows things to work even though
modpacks and tilesets are distributed separately (for instance, a "Lord
of the Rings" modpack may come with a supporting tileset with special
graphics, but the alternate sprite names provide fallback graphics if
"standard" tilesets are used).
But the system does have drawbacks - for instance it is difficult or
impossible to free any "unused" sprites after we're done loading sprites
- first of all since we don't really know _where_ they may be used
(sprites may be used multiple times, or none at all, depending on what
the ruleset specifies), and secondly since there's always the
possibility that we may later connect to a server with a different ruleset.
What I don't say in that case study is that the FreeCiv data
architecture has a number of problems. One of these is superficial but
annoying; the fact that graphics like flags and shields and tiles are
lumped together into big resource arrays makes it difficult to add
indidual flags and shields to the game.
Technically the use of a multi-sprite array is completely optional. You
can always add a new graphics file for each sprite you use. But you
will still have to set up a specfile for that graphics file and this
specfile will have to specify the dimensions of the array (even though
it has just one sprite in it).
My editor solves this problem. But there are two deeper problem of which
the annoyance with flags and shields is a symptom. They are:
(1) The lumping of data into representations that are convenient for
presentation or performance, but which add complexity and obscure who
actually owns the data.
(2) The fact that in many cases it is not clear which entity in the system
actually owns a given piece of data.
These problems have practical implications. They make the configuration
data excessively interconnected and fragile, such that editing a piece
of data in one place breaks connections to it somewhere else.
I'll get specific...
1. Names are associated with graphics via tile coordinates in graphics
spec files. This means that you cannot edit a graphic resource (like
a flags.png or shields.png file) and be sure that the new graphics will
be properly associated with their FreeCiv resource name.
The tile coordinates in specfiles are a sign of broken design. They
exist only in order to implement a link by name, and they make that link
fragile and unnecessarily difficult to maintain.
Imagine for a moment that groups of graphics were stored as directories,
one image per file, with the name of the file being its resource name. Thus:
graphics/
shields/
usa
russian
boer
:
etc. Then all that tile-coordinate nonsense would go away. The
specfile would simply point at the directory and you'd do a name
lookup the way God and Ken and Dennis intended. No more tile
coordinates, no more possibility of inconsistencies. It would also
get really easy to add new graphics.
There are two arguments against this organization. One is silly, one more
serious. The silly one is that tile arrays are what people are used to
editing. That one should be solved with tools like my resource editor;
I could trivially teach it how to convert between tile-array and
directory formats.
The more serious one is that all those separate file opens might constitute
a performance hit on the server startup. This is possible, but it's an
effect that should be *measured* rather than *assumed*. Modern filesystems
are fast. Let's try leaning on that instead of making up a bunch of ad-hoc
mechanisms that break easily!
If performance does become an issue, we could fall back on using a
multi-image graphical format like MNG instead of tile-array PNGs. That
would cut the file opens to one per resource file, but we could
annotate each frame of the format with its resource name and look
them up that way.
The disadvantage of MNG would be that adding new graphics would get
more difficult. You'd have to use a graphics editor to do it. I
like simple, Simple is better. So, my recommended course is:
FIRST PROPOSED CHANGE: Replace the tile-array files with directories, one
for each array, with the tiles as named files beneath them. Nuke all the
tile-coordinate crap in the specfiles.
The -x option of imagehack does this -- it bursts tile arrays into
directories of named images.
What we lose on file opens (not a lot after the first one. when
the directory inode gets cached) we'll partly gain back on not having
to compute bounding boxes and clip the tiles.
One thing to keep in mind here is that we desire to keep
backward-compatability with old tilesets.
Allowing sprites to be more easily placed into a single file is a
simplee matter of introducing good defaults if the [grid] part of the
specfile is not specified. The specfile itself will have to be there -
but that's probably a good thing since it does specify things like the
author of the sprite.
But I don't know how much extra space will be consumed when the sprite
grids are broken up. Historically I believe this is why grids were used
in the first place, but I suspect with modern graphics formats it is not
an issue.
2. The bigger problem is that the ownership of data is in many cases
unclear. Nations sort of own flags, but not really because flags vary
in size by tileset. Some tilesets share flags (by size). Every
tileset uses the same shields file. There are funky crosslinks
everywhere. You can't change any one thing without breaking
everything.
Indeed, and this relates to the ruleset<->tileset interconnectedness I
mentioned above.
This is not a new problem, as any person with experience in database
architecture can tell you. What's really wrong here is that we've
been trying to jam relational data (like the fact that a 26x20 flag
named f.russian is different from a 30x30 flag with the same name)
into a hierarchical framework defined by the inclusions in the data
files.
The idea is that each tileset specifies its f.russian flag. The fact
that all tilesets use the same set a flag is just because of the
laziness of tileset authors (whether you consider this a good or a bad
thing is another issue).
But it does introduce a major problem: either each tileset includes its
own set of flags, in which case a new nation's flag must be ported to
all tilesets, or they all share a set of flags, in which case all flags
must be the same size. It does not allow any sort of query like "give
me a sprite for a flag for the Russian nation, preferably 30x30,
preferably Engels-style". This query would allow the advantages of both
systems (with some degredation in graphics quality when the "best"
sprite wasn't available).
I think the right answer is to start thinking of each graphic as being
indexed by four things: role, type, (optional) size, and (optional)
style. The role is a functional key. it tells us the role of the
tile in FreeCiv's universe (like "I am a terrain tile"). The type is
more role information. The size is is its pixel height and width.
The style is a hint about what visual appearance you want.
What we really want is a lookup that permits a client to hand it a tuple
of (role, type, size, style) and get back an image. It really doesn't matter
how, physically, the data is organized. In fact, no code in the system
other than the lookup function should care. Making this association should
*not* be the job of the data files, because doing that is how we got in the
trouble we have now! The data-file stuff works for attributes that are
naturally distributed in a tree; crosslinks and things that need to be
reached by multiple indices subvert and confuse that hierarchy.
Remember, some of this information comes from the ruleset and some is
specified by the client.
Generally, the information we get from the ruleset is required, while
what the client wants is optional. In your system, the ruleset tells us
role and type, while the client (through the tileset) specifies size and
style. Getting the wrong size may be a fatal problem in some cases
(terrain), a bad problem in others (units), and no problem in others
(flags). Getting the wrong style can be ignored.
One thing you don't consider is that in many cases iso and non-iso
sprites are not interchangable. So in some cases (but not all!) "view"
should be a part of the tuple.
So here's what I think we ought to do as the next step after breaking
the tile arrays into directories -- cop to the fact that graphics are
one big resource shared by everybody, and move those tile directories
into a common space.
SECOND PROPOSED CHANGE: Consolidate all the tileset-specific graphics
into one directory called 'graphics'. All image lookup calls dive
into this directory. Hierarchy beneath it goes <role>/<type>/<size>/<style>.
Client lookup gives a role, a type, a preferred size, and a style.
For purposes of our examples, let's suppose that the client wants
icons to fit in a base 30x30 size and has specified "engels" as a
style. Consider the following cases:
1. Looking up a shield by name: ("shield", "usa". "30x30", "engels").
Look in subdirectory graphics/shield for something named "usa". See
that it's an image; this stops the search. Return it.
2. Looking up a flag by nation and preferred size: ("flag", "usa",
"30x30", "engels"). Look under graphics/flag/usa. See that there
are two directories, 26x20 and 30x21. Aha! That's a size. By
least-squares metric the closest match is 30x21. Look for "usa"
under there. See a file. Return it.
3. Looking up a city tile by type, size, and style. ("city",
"industrial_wall", "30x30", "engels") Look under graphics/city
for industrial_wall. Find that. It's a directory. Look beneath that.
See some size directories available. Pick one based on a least-squares
match. Notice that there is nothing named engels there. Fall back
on the default "trident". See that it's an image. Return it.
Even if we don't use exactly this data organization, we can use this
lookup semantics. The point is:
(a) to stop trying to pretend that the tilesets actually own shared
resources like the flags and shields,
Well, the fact that they're in the misc/ directory should indicate no
_specific_ tileset owns that sprite. But it is a fact that most sprites
(not just flags and shields) are ruleset-dependent (although I don't
know who "pretends" otherwise).
(b) to nuke all the coordinates and tile sizes and other useless cruft
in the specfiles,
Actually, in some cases this allows things to work that otherwise
wouldn't. For instance you can take the graphics files out of civ2,
introduce the right set of specfiles, and have them work out-of-the-box
with freeciv. (There is a script to automate this process.)
(c) to abstract the lookup so we can do clever things like
automatically fetching a best-fit image when the size hint doesn't
match what we have available.
Yes, this is a big benefit.
-----
There is another advantage/limitation of the data format that you
haven't considered: translation. The ruleset files are all in English.
Specific items can be marked for translation by enclosing the string
in _() (a marker easily recognized by xgettext) and making sure to
post-translate those entries when they are loaded by the server. Other
entries may go untranslated; these are usually in the latin1 encoding
(which is a problem since not all locales, particularly any one using
UTF or that isn't from eastern europe, are fully compatible with this
encoding).
Although there are problems, most of the ruleset data is appropriately
translated. Freeciv supports a significant number of languages (we just
added a Hebrew translator), and for each language all the translator has
to do is provide a text file matching up the "native" strings to their
translated equivalents. Gettext does the rest.
Yet I know of no good way in which a modpack can be translated. The
translations are distributed in one big binary database-like file that
covers the entire "freeciv" domain. I don't know of any way to add a
modular addition to this domain. Perhaps this is a shortcoming of
gettext; and perhaps it has even been addressed and I just don't know
about it. But the way I understand it if a modpack author marks their
strings for translation, freeciv will load the ruleset data and look up
the entries in the "freeciv" domain - where they won't be found because
the .gmo files are not changed by the modpack. Even if there are
translators willing to work on the modpack, unless it is included in the
standard distribution the translations won't be in the "freeciv" domain
and will therefore not work.
jason
- [Freeciv-Dev] Re: Graphic resource editor, and an architecture proposal,
Jason Dorje Short <=
|
|
