From MAILER-DAEMON Fri Jan 09 15:02:34 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1Af2q1-0001gp-VF
	for mharc-cfengine-develop@gnu.org; Fri, 09 Jan 2004 15:02:33 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1Af2pz-0001gK-Vs
	for cfengine-develop@nongnu.org; Fri, 09 Jan 2004 15:02:31 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1Af2pS-0001Zo-UB
	for cfengine-develop@nongnu.org; Fri, 09 Jan 2004 15:02:30 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1Af2pS-0001Zf-CW
	for cfengine-develop@nongnu.org; Fri, 09 Jan 2004 15:01:58 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.10/8.12.10) with ESMTP id i09J0JJ7013302;
	Fri, 9 Jan 2004 20:00:24 +0100 (MET)
Message-Id: <200401091900.i09J0JJ7013302@nexus.iu.hio.no>
Date: Fri, 9 Jan 2004 20:00:19 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] ExecResult results interpreted as a list
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0312021648260.16999@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Fri, 09 Jan 2004 20:02:32 -0000


Luke, 
I agree with this patch and am applying it, subject to a note in
the manual.
M


On  2 Dec, Luke A. Kanies wrote:
> Hi Mark,
> 
> This is the third time I've encountered this, so I'm going to bring it up
> this time.
> 
> As I mentioned in my previous email, I'm calling an external script within
> cfservd.conf and trying to use the results of that script to decide whom
> I'm going to allow to connect to cfservd.
> 
> For a little background, the reason I'm doing this is to reduce the number
> of manual steps involved in building a server.  I already am putting all
> of my servers in LDAP, so if I can have cfservd retrieve all IP addresses
> from LDAP, that would be very easy.  Otherwise I'm going to have to
> dynamically generate a cfengine file with all of the IP addresses and then
> import that file.  I can do this, but I've already got a bunch of these
> and they're really, really annoying.
> 
> So, I want to retrieve a list from LDAP.  Well, call an external script
> using ExecResult.  Okay, that's easy.  Problem is, only certain variables
> support returning lists, vs. strings.  For instance, if you call
> ExecResult within AddClasses, then the result is split based on the '.'
> character.  If you call it from within AddInstallables, the result is
> split based on '%'.  A very interesting point about both AddInstallables
> and AddClasses is that you can use ExecResult to set completely invalid
> classes, including classes containing spaces.  Go ahead, try it.  It's
> fun.
> 
> There are probably a couple other variables that support lists, but these
> are sufficient.  We have completely inconsistent behaviour here.  Given
> three variables, two of them split based on separate characters and the
> third doesn't split at all.
> 
> Just getting this far and understanding where everything happens is
> relatively complicated, and I'm apparently one of the few people who
> encounter this problem.  My guess is actually that many people encounter
> it but most just give up.  I guess I'm stubborn.
> 
> So, my question is:  Is there any way we can bring some consistency here?
> I'm pretty convinced that this inconsistency is a result of how the parser
> is built, and making function call behaviour truly consistent would
> involve a lot of rewriting, but am I wrong?  Does someone see a good,
> simple solution?  Yes, I know that I could just add a wrapper function for
> every variable, like AddClasses has, but that's a hack and doesn't solve
> the deeper problem.  The section of cfengine (specifically in install.c)
> dealing with variables is already in dire need of abstraction.
> 
> Frankly, I'd like to solve this problem by making the "variable" settings
> in cfengine into what the really are: function calls.
> 
> Take the following snippet:
> 
> control:
>   actionsequence = ( files )
>   actionsequence = ( shellcommands )
>   variable = ( files )
>   variable = ( shellcommands )
> 
> The settings for variable result in a warning, but the setting for
> actionsequence result in actions being added to a stack.  Nearly all of
> cfengine's builtin variables would be better off as functions, from what I
> can tell.  I'm seeing if I could somewhat easily modify the parser et al
> to allow this, but as I mentioned, I don't think it will be terribly
> straightforward.
> 
> My main point, though, is:
> 
> Do you have any idea how we can relatively easily support using ExecResult
> to consistently gather more than one piece of data?  I need to allow every
> machine in LDAP access to cfengine, and I don't want to keep that list
> manually, and I don't want to have to generate a file that gets imported
> by cfservd.  That's a hack, and I hate it.
> 
> Luke
> 



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Sat Jan 10 01:49:23 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AfCvz-0006aP-Bh
	for mharc-cfengine-develop@gnu.org; Sat, 10 Jan 2004 01:49:23 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AfCvw-0006Z9-My
	for cfengine-develop@nongnu.org; Sat, 10 Jan 2004 01:49:20 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AedPE-0003Lt-9n
	for cfengine-develop@nongnu.org; Thu, 08 Jan 2004 11:53:43 -0500
Received: from [66.93.249.120] (helo=pixie.madstop.com)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AedPD-0003L6-A8
	for cfengine-develop@nongnu.org; Thu, 08 Jan 2004 11:53:11 -0500
Received: by pixie.madstop.com (Postfix, from userid 100)
	id 75A084C030; Thu,  8 Jan 2004 09:54:39 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
	by pixie.madstop.com (Postfix) with ESMTP id 7303F5602B
	for <cfengine-develop@nongnu.org>; Thu,  8 Jan 2004 09:54:39 -0600 (CST)
Date: Thu, 8 Jan 2004 09:54:39 -0600 (CST)
From: "Luke A. Kanies" <luke@madstop.com>
X-X-Sender: luke@pixie
To: Cfengine Developers <cfengine-develop@nongnu.org>
Message-ID: <Pine.GSO.4.51.0312091431520.11147@pixie>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Subject: [Cfengine-develop] my own branch of the tree
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Sat, 10 Jan 2004 06:49:22 -0000

Hi all,

(How many of us are there on this list?)

It looks like Mark is incredibly busy as usual right now, and there are
some patches (one especially) I think are necessary and useful
but which he is not going to be able to look at until at any time soon.
I also would really like to provide this functionality to a client of
mine, as it would allow me to cease some of my code generation.  And last
but not least, I plan to begin providing commercial support for cfengine
this year (hopefully in Q1), and I will need my own branch to help
customers out until my patches can be accepted into the main branch.

As a result, I've decided to start maintaining my own branch of cfengine.
I can't say at this point how much it will end up differening from the
main branch; if it's just me, I expect not many differences, but if others
are interested in participating, then it could be quite a bit.

I'm pretty happy with cfengine, but there are some aspects of it that make
it frustrating to use, and I think some of those aspects could be fixed
relatively easily.  For whatever reasons, Mark does not seem to be
frustrated by these same limitations and thus is not very interested in
fixing them. I'm willing to spend at least some time trying to make those
fixes, and I would very much appreciate help both in writing the fixes and
in testing them.  I will immediately provide read access to my CVS
repository, which means that at the least there will be a place that
provides anonymous CVS access to Mark's copy of cfengine (I'll sync it
from him nightly or something), but it also should make it very easy to
update from and contribute to my (or his) branch.

I will also provide some mechanism for submitting and sharing patches, so
that multiple people can vet a patch before I accept it to the branch but
also so that people are encouraged to send patches.  Patches are
definitely good, even if the patches aren't good. :)

Basically, I just want to encourage more participation in cfengine
development, both in getting patches and in testing them.  Mark is too
busy to do much development, patch vetting, or testing, so it's time the
community stepped in to help out, and this seems to be the easiest and
best way for us to do that right now.

I would definitely welcome any feedback, including chastisement, blame, or
flames.  If you have any better ideas on how to spur development, I'd love
to hear them.  I don't even expect to spend a ton of time on this, as I
have my own development work I want to do, but there are features I need,
bugs I need fixed (some that Mark does not consider bugs), and lots of
refactoring to be done, all of which I am willing to work on.

I'm largely considering this similar to Alan Cox (not that I deserve
comparison to him) maintaining his own branch of linux; it's not
competition, but he has his own ideas of what should be in there, and he's
patient in getting those ideas to the main tree.

Feel free to mail me directly with questions or concerns.

Luke Kanies

-- 
  "When a man sits with a pretty girl for an hour, it seems like a
minute.  But let him sit on a hot stove for a minute, and it's longer
than any hour.  That's relativity."   --Albert Einstein



From MAILER-DAEMON Sat Jan 10 03:44:15 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AfEj9-0004qW-Hi
	for mharc-cfengine-develop@gnu.org; Sat, 10 Jan 2004 03:44:15 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AfEj6-0004or-2z
	for cfengine-develop@nongnu.org; Sat, 10 Jan 2004 03:44:12 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AfEiV-0004d1-No
	for cfengine-develop@nongnu.org; Sat, 10 Jan 2004 03:44:07 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AfEhD-0004OK-3P
	for cfengine-develop@nongnu.org; Sat, 10 Jan 2004 03:42:15 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.10/8.12.10) with ESMTP id i0A7eWJ7003707;
	Sat, 10 Jan 2004 08:40:36 +0100 (MET)
Message-Id: <200401100740.i0A7eWJ7003707@nexus.iu.hio.no>
Date: Sat, 10 Jan 2004 08:40:32 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0312091431520.11147@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Sat, 10 Jan 2004 08:44:14 -0000


Luke, your timing is strange as I just sent you a message to say that
I have included your patch. I welcome help in these matters but I
am still very scared that folks will succumb to the "easy way" of
scripting "solutions" to problems rather than finding the reproducible,
general, convergent way. Cfengine has to be state of the art -- oops I mean
science. 

After LISA I was more worried than ever that people are not only not
so worried about this, but that some people are even going backwards
and writing brand new tools that look like ftp.

You have earlier criticised me for being against abstraction. I do not
believe that this is correct. I am against ambiguity. Cfengine has its
problems and I would love help in working them out, at the same time
as moving forward with the anomaly stuff, but I am scared of what I said
above -- that in its unbridled haste, the wheels of progress will
maim the key principles that make cfengine safe today. 
e.g. the whole discussion about locking and dual passes; peoples'
insistence on writing modules for every little thing instead of
developing the convergent mechanisms... 

The development team was put together from people whom I trust
to do good work, but it has been very quiet. The Savannah web pages
have been a complete waste of time, as I suspected. I don't even
understand how to maintain them anymore. They have become a burden
rather than a help. At times I will be too busy to even think about
cfengine -- but I have a few weeks now. Let's talk.

Mark

On  8 Jan, Luke A. Kanies wrote:
> Hi all,
> 
> (How many of us are there on this list?)
> 
> It looks like Mark is incredibly busy as usual right now, and there are
> some patches (one especially) I think are necessary and useful
> but which he is not going to be able to look at until at any time soon.
> I also would really like to provide this functionality to a client of
> mine, as it would allow me to cease some of my code generation.  And last
> but not least, I plan to begin providing commercial support for cfengine
> this year (hopefully in Q1), and I will need my own branch to help
> customers out until my patches can be accepted into the main branch.
> 
> As a result, I've decided to start maintaining my own branch of cfengine.
> I can't say at this point how much it will end up differening from the
> main branch; if it's just me, I expect not many differences, but if others
> are interested in participating, then it could be quite a bit.
> 
> I'm pretty happy with cfengine, but there are some aspects of it that make
> it frustrating to use, and I think some of those aspects could be fixed
> relatively easily.  For whatever reasons, Mark does not seem to be
> frustrated by these same limitations and thus is not very interested in
> fixing them. I'm willing to spend at least some time trying to make those
> fixes, and I would very much appreciate help both in writing the fixes and
> in testing them.  I will immediately provide read access to my CVS
> repository, which means that at the least there will be a place that
> provides anonymous CVS access to Mark's copy of cfengine (I'll sync it
> from him nightly or something), but it also should make it very easy to
> update from and contribute to my (or his) branch.
> 
> I will also provide some mechanism for submitting and sharing patches, so
> that multiple people can vet a patch before I accept it to the branch but
> also so that people are encouraged to send patches.  Patches are
> definitely good, even if the patches aren't good. :)
> 
> Basically, I just want to encourage more participation in cfengine
> development, both in getting patches and in testing them.  Mark is too
> busy to do much development, patch vetting, or testing, so it's time the
> community stepped in to help out, and this seems to be the easiest and
> best way for us to do that right now.
> 
> I would definitely welcome any feedback, including chastisement, blame, or
> flames.  If you have any better ideas on how to spur development, I'd love
> to hear them.  I don't even expect to spend a ton of time on this, as I
> have my own development work I want to do, but there are features I need,
> bugs I need fixed (some that Mark does not consider bugs), and lots of
> refactoring to be done, all of which I am willing to work on.
> 
> I'm largely considering this similar to Alan Cox (not that I deserve
> comparison to him) maintaining his own branch of linux; it's not
> competition, but he has his own ideas of what should be in there, and he's
> patient in getting those ideas to the main tree.
> 
> Feel free to mail me directly with questions or concerns.
> 
> Luke Kanies
> 



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Wed Jan 14 12:09:05 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AgoH1-0008No-EF
	for mharc-cfengine-develop@gnu.org; Wed, 14 Jan 2004 11:53:43 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AgndO-0002hz-4L
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 11:12:46 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AgnV4-00019y-Vc
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 11:04:42 -0500
Received: from [66.93.249.120] (helo=pixie.madstop.com)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AgnQp-0000fD-56
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 10:59:47 -0500
Received: by pixie.madstop.com (Postfix, from userid 100)
	id 40FAE4C031; Wed, 14 Jan 2004 09:59:43 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
	by pixie.madstop.com (Postfix) with ESMTP id 3D6605602B
	for <cfengine-develop@nongnu.org>; Wed, 14 Jan 2004 09:59:43 -0600 (CST)
Date: Wed, 14 Jan 2004 09:59:43 -0600 (CST)
From: "Luke A. Kanies" <luke@madstop.com>
X-X-Sender: luke@pixie
To: Cfengine Developers <cfengine-develop@nongnu.org>
Subject: Re: [Cfengine-develop] my own branch of the tree
Message-ID: <Pine.GSO.4.51.0401140959270.4350@pixie>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Wed, 14 Jan 2004 16:53:27 -0000

Hi Mark,

I apparently managed to delete your message while purging spam.  Skaar let
me know about it and helpfully sent me a copy.

On Tue, 13 Jan 2004, skaar wrote:

> ----- Forwarded message from Mark.Burgess@iu.hio.no -----
>
> Date: Sat, 10 Jan 2004 08:40:32 +0100 (MET)
> From: Mark.Burgess@iu.hio.no
> Subject: Re: [Cfengine-develop] my own branch of the tree
> To: luke@madstop.com
> Cc: cfengine-develop@nongnu.org
>
>
> Luke, your timing is strange as I just sent you a message to say that
> I have included your patch. I welcome help in these matters but I
> am still very scared that folks will succumb to the "easy way" of
> scripting "solutions" to problems rather than finding the reproducible,
> general, convergent way. Cfengine has to be state of the art -- oops I mean
> science.

I agree that the timing is strange.  I haven't looked into who sent what
first, but I have had my email sitting in my postponed messages box for
weeks, trying to decide the right course of action.

As to concerns about people finding the easy way...  On the one hand, that
is certainly my goal:  It is difficult and often impossible to do
moderately complex tasks in cfengine right now, and I want it to be
easier.  However, I think your real concern is cfengine morphing into
"easy" rather than "right".  I don't remember the author, but isn't the
quote "Things should be as simple as they can be, but no simpler"?
Cfengine is not as simple, nor as powerful, as it can be, and I would like
to fix that.  I do agree that it should not compromise its founding
principles to become easier to use, but I think it has a long way to go
before that becomes a problem.

As to the reason why I am constantly writing scripts that cfengine calls,
it is because cfengine cannot do what I require.  90% of that would be
resolved by some sort of built-in list variable and an iteration operator.

> After LISA I was more worried than ever that people are not only not
> so worried about this, but that some people are even going backwards
> and writing brand new tools that look like ftp.

Some people have different needs.  I manage a network of four machines at
home, and I certainly don't manage that with the same techniques I use at
work.

> You have earlier criticised me for being against abstraction. I do not
> believe that this is correct. I am against ambiguity. Cfengine has its
> problems and I would love help in working them out, at the same time
> as moving forward with the anomaly stuff, but I am scared of what I said
> above -- that in its unbridled haste, the wheels of progress will
> maim the key principles that make cfengine safe today.
> e.g. the whole discussion about locking and dual passes; peoples'
> insistence on writing modules for every little thing instead of
> developing the convergent mechanisms...

I don't know what your reasoning is for how cfengine turned out, I can
only see the product, and cfengine itself is somewhat pathologically
opposed to abstraction.  Methods are an attempt to build some abstraction,
I believe, but without iteration they are insufficient.

In nearly all cases, the modules I write are to provide some facsimile of
iteration, and sometimes to provide an attempt at higher-level data
structures.  For instance, I recently wrote a module to install packages,
as I described on the cfengine list.  To do this, I had to collect a list
of packages to install, along with the specific revision to install, and
then I had to execute the code necessary to do that work.

Well, cfengine does not have a list type, so I had to collect the list in
a string.  Cfengine does not support a hash type, either, so I had to name
the packages like <package>--<revision>, e.g., cfengine--2.1.0p1.  And
because cfengine does not have any kind of iteration, I had to pass that
list to an external module to get turned into the effectively iterative
process.  And lastly, because cfengine does not have anything resembling
'eval', I had to set up a confusing system whereby the file gets generated
on the first cfengine run, and the next run it gets imported because it
exists.  This really, really confuses most people I work with, but there's
no other good way to do this within cfengine right now.

As to why you don't ever run into these problems, I don't know.  I am
astounded that cfengine meets all of your system management needs, and if
they do, you are the only person with I've spoken for whom that is true.
Everyone else is stepping outside of cfengine for very significant
operations, either generating cfengine code (something Alva recommended as
the "best" solution a long time ago) or just avoiding cfengine entirely
for swaths of their system management.

I think cfengine is a great tool with a great pedigree, but taking it to
the next level is going to require some significant changes.  Now, I
believe it is possible to enact those changes while maintaining the
original goals of cfengine.  If you don't trust other people to interpret
that goal, then that's your prerogative, but we who are interested in
developing cfengine obviously do not have quite the same goals as you do.

I _personally_ find cfengine barely usable without iteration and lists.
Whether others agree, I don't know.  I _personally_ find cfengine's parser
to be the source of many, many problems, and much confusion.  Again, I
have no idea if others find that to be true.  But my goal is to spend the
smallest amount of time on cfengine fixing the biggest problems.  I don't
think I'll be able to easily add a list type, and I don't think I'll be
able to easily fix the parser (BTW, you should look at the parser for
bash; it's an excellent example).  However, I do think I can solve some of
the glaring problems, and bring some modicum of consistency to cfengine,
and that is certainly horribly lacking.

> The development team was put together from people whom I trust
> to do good work, but it has been very quiet. The Savannah web pages
> have been a complete waste of time, as I suspected. I don't even
> understand how to maintain them anymore. They have become a burden
> rather than a help. At times I will be too busy to even think about
> cfengine -- but I have a few weeks now. Let's talk.

I am not surprised that the development team have not accomplished
anything; they don't have CVS (or any other version tool) access, which
means that they have to implement their own synchronization mechanisms.
You have also expressed more fear of mistakes than interest in
advancement.  I think the fundamental disconnect is that you find cfengine
essentially satisfactory, whereas others (at least me) think there are
areas where it needs specific improvement.

If you are the gatekeeper to changes in cfengine, and you don't think it
needs to be changed, then the development group is naturally not going to
get much done.  At some point you have to learn which developers you trust
to keep your vision, and then let those developers into your inner circle.
You are apparently too busy to continue advancing cfengine very much, and
that's fine; you've been essentially the sole developer for 10 years.  But
you need to be able to at least pass the workload off on someone else.

I have gotten the distinct impression that you don't trust my work,
apparently because I have written in perl.  I actually think I have a
pretty good idea of the cfengine vision, and I think I've done well at
keeping that vision even in the modules I've written.  I just have the
need for abstraction, I want a quality parser, and maybe a few other
things.

Also, I plan on offering commercial support this year, which means I need
to be able to fix bugs that my customers submit to me without waiting for
a lengthy approval from you.  For this reason alone I will have to
maintain my own branch, but it would be great if that branch were
essentially in sync with the main branch.

Luke

-- 
Once...in the wilds of Afghanistan, I lost my corkscrew, and we were
forced to live on nothing but food and water for days.  -- W. C. Fields



From MAILER-DAEMON Wed Jan 14 14:56:36 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1Agr7z-00076f-VG
	for mharc-cfengine-develop@gnu.org; Wed, 14 Jan 2004 14:56:35 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1Agr7x-00076U-So
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 14:56:33 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1Agr7P-0006zz-NO
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 14:56:32 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1Agr7O-0006zR-Ux
	for cfengine-develop@nongnu.org; Wed, 14 Jan 2004 14:55:59 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.10/8.12.10) with ESMTP id i0EJtm2K000071;
	Wed, 14 Jan 2004 20:55:53 +0100 (MET)
Message-Id: <200401141955.i0EJtm2K000071@nexus.iu.hio.no>
Date: Wed, 14 Jan 2004 20:55:48 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0401140959270.4350@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Wed, 14 Jan 2004 19:56:34 -0000


Luke,

I agree with much of what you say, but you seem to mix rational sense
with spurious complaints. I would also like iteration. I just don't know
how to do it. Do you? I do know that none of this has got anything to do with
the parser. As I explained at LISA, you cannot write the kind of parser you
want until the language is completely known. As long as it continues to develop,
it makes no sense to go on about this. It also has nothing to do with
CVS. I get patches all the time, I just don't get development.
Also "everyone I speak to" cuts no weight with me. I talk to people too.

You have done good things with cfengine and show great insight im many things. If
I don't fully trust you it is because you seem too emotionally attached
to your own vision. I would love to harness your talents. Can you enter
into a discussion and accept a viewpoint that you do not agree with?
Every message from you is intense to the point of flaming. I embrace the rational
and shy away from heated blows.

I am very interested to hear about what you intend to do about iteration.
If you suggest something that I find convincing, I will happily work out
a solution that works for both of us. If you think that I will just open the
code for you to do whatever takes your fancy then, no that would be irresponsible.

I enjoyed speaking to you in private. So let's try to elevate the discussion
from politics to solutions.

Mark

On 14 Jan, Luke A. Kanies wrote:
> Hi Mark,
> 
> I apparently managed to delete your message while purging spam.  Skaar let
> me know about it and helpfully sent me a copy.
> 
> On Tue, 13 Jan 2004, skaar wrote:
> 
>> ----- Forwarded message from Mark.Burgess@iu.hio.no -----
>>
>> Date: Sat, 10 Jan 2004 08:40:32 +0100 (MET)
>> From: Mark.Burgess@iu.hio.no
>> Subject: Re: [Cfengine-develop] my own branch of the tree
>> To: luke@madstop.com
>> Cc: cfengine-develop@nongnu.org
>>
>>
>> Luke, your timing is strange as I just sent you a message to say that
>> I have included your patch. I welcome help in these matters but I
>> am still very scared that folks will succumb to the "easy way" of
>> scripting "solutions" to problems rather than finding the reproducible,
>> general, convergent way. Cfengine has to be state of the art -- oops I mean
>> science.
> 
> I agree that the timing is strange.  I haven't looked into who sent what
> first, but I have had my email sitting in my postponed messages box for
> weeks, trying to decide the right course of action.
> 
> As to concerns about people finding the easy way...  On the one hand, that
> is certainly my goal:  It is difficult and often impossible to do
> moderately complex tasks in cfengine right now, and I want it to be
> easier.  However, I think your real concern is cfengine morphing into
> "easy" rather than "right".  I don't remember the author, but isn't the
> quote "Things should be as simple as they can be, but no simpler"?
> Cfengine is not as simple, nor as powerful, as it can be, and I would like
> to fix that.  I do agree that it should not compromise its founding
> principles to become easier to use, but I think it has a long way to go
> before that becomes a problem.
> 
> As to the reason why I am constantly writing scripts that cfengine calls,
> it is because cfengine cannot do what I require.  90% of that would be
> resolved by some sort of built-in list variable and an iteration operator.
> 
>> After LISA I was more worried than ever that people are not only not
>> so worried about this, but that some people are even going backwards
>> and writing brand new tools that look like ftp.
> 
> Some people have different needs.  I manage a network of four machines at
> home, and I certainly don't manage that with the same techniques I use at
> work.
> 
>> You have earlier criticised me for being against abstraction. I do not
>> believe that this is correct. I am against ambiguity. Cfengine has its
>> problems and I would love help in working them out, at the same time
>> as moving forward with the anomaly stuff, but I am scared of what I said
>> above -- that in its unbridled haste, the wheels of progress will
>> maim the key principles that make cfengine safe today.
>> e.g. the whole discussion about locking and dual passes; peoples'
>> insistence on writing modules for every little thing instead of
>> developing the convergent mechanisms...
> 
> I don't know what your reasoning is for how cfengine turned out, I can
> only see the product, and cfengine itself is somewhat pathologically
> opposed to abstraction.  Methods are an attempt to build some abstraction,
> I believe, but without iteration they are insufficient.
> 
> In nearly all cases, the modules I write are to provide some facsimile of
> iteration, and sometimes to provide an attempt at higher-level data
> structures.  For instance, I recently wrote a module to install packages,
> as I described on the cfengine list.  To do this, I had to collect a list
> of packages to install, along with the specific revision to install, and
> then I had to execute the code necessary to do that work.
> 
> Well, cfengine does not have a list type, so I had to collect the list in
> a string.  Cfengine does not support a hash type, either, so I had to name
> the packages like <package>--<revision>, e.g., cfengine--2.1.0p1.  And
> because cfengine does not have any kind of iteration, I had to pass that
> list to an external module to get turned into the effectively iterative
> process.  And lastly, because cfengine does not have anything resembling
> 'eval', I had to set up a confusing system whereby the file gets generated
> on the first cfengine run, and the next run it gets imported because it
> exists.  This really, really confuses most people I work with, but there's
> no other good way to do this within cfengine right now.
> 
> As to why you don't ever run into these problems, I don't know.  I am
> astounded that cfengine meets all of your system management needs, and if
> they do, you are the only person with I've spoken for whom that is true.
> Everyone else is stepping outside of cfengine for very significant
> operations, either generating cfengine code (something Alva recommended as
> the "best" solution a long time ago) or just avoiding cfengine entirely
> for swaths of their system management.
> 
> I think cfengine is a great tool with a great pedigree, but taking it to
> the next level is going to require some significant changes.  Now, I
> believe it is possible to enact those changes while maintaining the
> original goals of cfengine.  If you don't trust other people to interpret
> that goal, then that's your prerogative, but we who are interested in
> developing cfengine obviously do not have quite the same goals as you do.
> 
> I _personally_ find cfengine barely usable without iteration and lists.
> Whether others agree, I don't know.  I _personally_ find cfengine's parser
> to be the source of many, many problems, and much confusion.  Again, I
> have no idea if others find that to be true.  But my goal is to spend the
> smallest amount of time on cfengine fixing the biggest problems.  I don't
> think I'll be able to easily add a list type, and I don't think I'll be
> able to easily fix the parser (BTW, you should look at the parser for
> bash; it's an excellent example).  However, I do think I can solve some of
> the glaring problems, and bring some modicum of consistency to cfengine,
> and that is certainly horribly lacking.
> 
>> The development team was put together from people whom I trust
>> to do good work, but it has been very quiet. The Savannah web pages
>> have been a complete waste of time, as I suspected. I don't even
>> understand how to maintain them anymore. They have become a burden
>> rather than a help. At times I will be too busy to even think about
>> cfengine -- but I have a few weeks now. Let's talk.
> 
> I am not surprised that the development team have not accomplished
> anything; they don't have CVS (or any other version tool) access, which
> means that they have to implement their own synchronization mechanisms.
> You have also expressed more fear of mistakes than interest in
> advancement.  I think the fundamental disconnect is that you find cfengine
> essentially satisfactory, whereas others (at least me) think there are
> areas where it needs specific improvement.
> 
> If you are the gatekeeper to changes in cfengine, and you don't think it
> needs to be changed, then the development group is naturally not going to
> get much done.  At some point you have to learn which developers you trust
> to keep your vision, and then let those developers into your inner circle.
> You are apparently too busy to continue advancing cfengine very much, and
> that's fine; you've been essentially the sole developer for 10 years.  But
> you need to be able to at least pass the workload off on someone else.
> 
> I have gotten the distinct impression that you don't trust my work,
> apparently because I have written in perl.  I actually think I have a
> pretty good idea of the cfengine vision, and I think I've done well at
> keeping that vision even in the modules I've written.  I just have the
> need for abstraction, I want a quality parser, and maybe a few other
> things.
> 
> Also, I plan on offering commercial support this year, which means I need
> to be able to fix bugs that my customers submit to me without waiting for
> a lengthy approval from you.  For this reason alone I will have to
> maintain my own branch, but it would be great if that branch were
> essentially in sync with the main branch.
> 
> Luke
> 



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Sun Jan 18 04:57:24 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1Ai9gK-0000oI-9r
	for mharc-cfengine-develop@gnu.org; Sun, 18 Jan 2004 04:57:24 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1Ai9gI-0000nS-5f
	for cfengine-develop@nongnu.org; Sun, 18 Jan 2004 04:57:22 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1Ai9fm-0000kU-Dw
	for cfengine-develop@nongnu.org; Sun, 18 Jan 2004 04:57:21 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1Ai9fl-0000kP-Vv
	for cfengine-develop@nongnu.org; Sun, 18 Jan 2004 04:56:50 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.10/8.12.10) with ESMTP id i0I9ue2K020739;
	Sun, 18 Jan 2004 10:56:45 +0100 (MET)
Message-Id: <200401180956.i0I9ue2K020739@nexus.iu.hio.no>
Date: Sun, 18 Jan 2004 10:56:40 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0401140959270.4350@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Sun, 18 Jan 2004 09:57:22 -0000



On 14 Jan, Luke A. Kanies wrote:
> Hi Mark,
> 
> I apparently managed to delete your message while purging spam.  Skaar let
> me know about it and helpfully sent me a copy.
> 

Luke,

can you not explain to our small list what you see as your vision for loops
in cfengine and what you mean by "abstraction" etc with sufficient detail
so that we can determine the best solution and move on constructively?
We should do this soon, while I have time to listen to the discussion.

I would also like input from the rest of the developers list, especially
on how to implement the solution without damaging  backward compatibility
or convergence.

Mark

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Mon Jan 19 06:53:03 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AiXw9-0007UV-Q8
	for mharc-cfengine-develop@gnu.org; Mon, 19 Jan 2004 06:51:21 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AiXvl-0007QU-5S
	for cfengine-develop@nongnu.org; Mon, 19 Jan 2004 06:50:57 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AiXrx-0006mC-JM
	for cfengine-develop@nongnu.org; Mon, 19 Jan 2004 06:47:32 -0500
Received: from [145.100.16.144] (helo=mta.sara.nl)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AiXrR-0006kq-AI
	for cfengine-develop@nongnu.org; Mon, 19 Jan 2004 06:46:29 -0500
Received: from sara.nl (bas.ka.sara.nl [145.100.6.35])
	by mta.sara.nl (8.11.3/8.11.3) with ESMTP id i0JBkKJ12262;
	Mon, 19 Jan 2004 12:46:20 +0100 (MET)
Message-ID: <400BC387.9050904@sara.nl>
Date: Mon, 19 Jan 2004 12:46:15 +0100
From: Bas van der Vlies <basv@sara.nl>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
	rv:1.6b) Gecko/20031205 Thunderbird/0.4
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
References: <200401180956.i0I9ue2K020739@nexus.iu.hio.no>
In-Reply-To: <200401180956.i0I9ue2K020739@nexus.iu.hio.no>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Jan 2004 11:51:14 -0000

Mark.Burgess@iu.hio.no wrote:

> 
> On 14 Jan, Luke A. Kanies wrote:
> 
>>Hi Mark,
>>
>>I apparently managed to delete your message while purging spam.  Skaar let
>>me know about it and helpfully sent me a copy.
>>
> 
> 
> Luke,
> 
> can you not explain to our small list what you see as your vision for loops
> in cfengine and what you mean by "abstraction" etc with sufficient detail
> so that we can determine the best solution and move on constructively?
> We should do this soon, while I have time to listen to the discussion.
> 
> I would also like input from the rest of the developers list, especially
> on how to implement the solution without damaging  backward compatibility
> or convergence.

I am also interested in the explanation and how to implement it!
-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************



From MAILER-DAEMON Tue Jan 20 11:42:09 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1Aiyx6-0007p3-Tk
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 11:42:08 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1Aiyx3-0007nI-0F
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 11:42:05 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AiywV-0007dS-0h
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 11:42:02 -0500
Received: from [66.93.249.120] (helo=pixie.madstop.com)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AiywU-0007cu-FQ
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 11:41:30 -0500
Received: by pixie.madstop.com (Postfix, from userid 100)
	id C2CC84C031; Tue, 20 Jan 2004 10:41:23 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
	by pixie.madstop.com (Postfix) with ESMTP
	id BEC77560C5; Tue, 20 Jan 2004 10:41:23 -0600 (CST)
Date: Tue, 20 Jan 2004 10:41:23 -0600 (CST)
From: "Luke A. Kanies" <luke@madstop.com>
X-X-Sender: luke@pixie
To: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
In-Reply-To: <200401141955.i0EJtm2K000071@nexus.iu.hio.no>
Message-ID: <Pine.GSO.4.51.0401151024090.9282@pixie>
References: <200401141955.i0EJtm2K000071@nexus.iu.hio.no>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 16:42:06 -0000

On Wed, 14 Jan 2004 Mark.Burgess@iu.hio.no wrote:

>
> Luke,
>
> I agree with much of what you say, but you seem to mix rational sense
> with spurious complaints. I would also like iteration. I just don't know
> how to do it. Do you? I do know that none of this has got anything to do
> with the parser. As I explained at LISA, you cannot write the kind of
> parser you want until the language is completely known. As long as it
> continues to develop, it makes no sense to go on about this. It also has
> nothing to do with CVS. I get patches all the time, I just don't get
> development. Also "everyone I speak to" cuts no weight with me. I talk
> to people too.

No, I don't have a great solution for iteration right now, but I think
that partly is the fault of the parser.  If I were to be able to just
willy-nilly modify the syntax, I would probably do something that looked a
heckuva lot like ruby, rather than perl:

control:
	list = ( 1 2 3 4 )

copy:
	${list}.each { |item|
		/path/to/files/${item} dest=/new/path/${item}
	}

links:
	${list}.each { |item|
		/usr/local/bin -> /usr/local/software/${item}/bin/*
	}

I'm not saying that this is _the_ solution, but once you get used to it,
it continues to make sense.

Another option is to implicitly support iteration so that the syntax does
not change at all:

control:
	list = ( 1 2 3 4 )

copy:
	/path/to/files/${list} dest=/new/path/${list}

links:
	/new/place/${list} -> /old/place/${list}

I know links already kind of supports this, but it's using a standard
string variable, not an actual list variable.  One of the real problems
with this type of iteration is that you can't then do nested iterations,
and you can't really do anything resembling hashes (associative arrays)
either, and I think they're also important.

However, I definitely think the parser is related to making cfengine
better.  There are a number of aspects of the parser that limit
abstraction in cfengine.  For instance, I have multiple cfengine servers,
and I want to collect their IP addresses into a variable and use that
variable for access control:

control:
	servers = ( 170.150.135.232 170.150.135.235 )
	AllowConnectionsFrom = ( ${servers} )
	TrustKeysFrom = ( ${servers} )

But I can't do that because of the way the parser is written.  I could
point out the details, but essentially, it looks for a value and is
willing to split that value based on space, _or_ it looks for a variable
and interprets the value of the variable as one value.  What's worse, this
value is then not sanity checked.

Until recently it was possible to easily bypass sanity checks on class
names by using ExecResult with AddClasses and AddInstallable.  This is
because the parser does variable interpolation at the same level as it
does value acceptance; it instead needs to do all of the interpolation
prior to value acceptance, which would automatically make sanity checking
happen to variable values.  Bash is a great example of how to do this
right; in fact, there's a diagram in the Bash book by O'Reilly which walks
through the way that bash parses a command.  It does every item (variable
interpolation, tokenizing, line splitting, alias lookup, etc.) in a
specific order, and everything is tightly defined.  As a result, you get
very little confusion about why something happens the way it does.

> You have done good things with cfengine and show great insight im many
> things. If I don't fully trust you it is because you seem too
> emotionally attached to your own vision. I would love to harness your
> talents. Can you enter into a discussion and accept a viewpoint that you
> do not agree with? Every message from you is intense to the point of
> flaming. I embrace the rational and shy away from heated blows.

Hmmm.  Yes, I believe that I can accept a viewpoint that I don't agree
with; at least, I have many times in the past.  I am not terribly happy at
accepting the idea that what I want to do is somehow "wrong" or "not the
cfengine way", however.

I am emotionally attached to my vision, I agree with that, but the reason
I'm emotionally attached is because every time I try to do anything with
cfengine I have to go through a painful process of discovery, figuring out
what it does differently from what I expect, reading the source to figure
out why it does it that way and if I can work around it, and etc.

I'm not exaggerating when I say that 4/5 times that I make a significant
expansion to my cfengine configuration (meaning that I begin managing a
different aspect of my systems) I have to read through a lot of source
code.  Discovering the above problem with the server list was definitely
not fun, nor was discovering what characters AddClasses and AddInstallable
split on (yep, they're different; nope, they're not documented; yep, I
should have submitted a patch to the docs).

The amount of pain that I go through while trying to use cfengine for
abstraction is what makes me emotionally invested in it.  I really do not
understand how anyone else is able to avoid running into these problems,
but I have spoken to others who do run into them.

I get frustrated, Mark, because it often seems like you believe that if
it's not a problem for you it is not a problem for anyone else, and that
is definitely not the case.  You and I obviously do things very
differently, and our relative frustrations with cfengine are a reflection
of that.

> I am very interested to hear about what you intend to do about iteration.
> If you suggest something that I find convincing, I will happily work out
> a solution that works for both of us. If you think that I will just open the
> code for you to do whatever takes your fancy then, no that would be irresponsible.

As I said above, I don't necessarily have a solution for iteration.  I
think that if it were up to me, I would use the ruby-style example above.
But if it were up to me, I would probably rewrite most of the parser,
which I think will be necessary to support such a thing.

We've already discussed having write access to the repository; I don't
necessarily need that, but it takes a good bit of effort to build an
environment of ongoing development around a project, and there is no such
environment around cfengine.  As far as I can tell, it's currently a one
man project and you are quite busy, which means it doesn't advance very
quickly.

I am of the opinion that cfengine's code base needs significant
refactoring, which makes sense given its age and history.  You are
probably never going to have the time to do that refactoring, which means
you need to find a way to encourage others to help you with it.  I don't
know exactly how one encourages this, but I do have some ideas, and at
least providing read access to CVS is one of them.

I expect you would get many more patches if you had read access to CVS,
and I also expect that you get many more people testing the latest version
of the code, which is pretty important for vetting patches.

> I enjoyed speaking to you in private. So let's try to elevate the discussion
> from politics to solutions.

I definitely enjoyed speaking with you at LISA, and I am hoping to work
something out so we can both be happy.  I do think there's an element of
politics to resolve, however, as usual.

At this point you are the final arbiter of what "the cfengine way" is.  I
apparently often produce recommendations that you don't believe fit into
that category but that I do.  The question of how to resolve this
conflicts is a question of politics, unless it can somehow be reduced to a
purely technical discussion.

If you are able to provide technical justifications for what you don't
like about my recommendations then I agree that politics can be left out,
but if the justifications are more "because I don't think they fit", then
it becomes a political problem.  I'm fine not dwelling on this issue, but
I think it's going to rear its ugly head a number of times, because it has
already come up with most of my recommendations.

On a technical level, though, I would like to take cfengine's parsing
towards the way that bash does its parsing.  No, not the syntax, not the
function, just the parsing.  Bash not only has a very good parsing
mechanism, its actual parsing code is some of the cleanest I've seen, and
I think cfengine would do very well to learn from it.

-- 
"Anyone who considers arithmatical methods of producing random digits
is, of course, in a state of sin."    -- John Von Neumann



From MAILER-DAEMON Tue Jan 20 12:13:36 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AizRY-00081Y-36
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 12:13:36 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AizRQ-00080A-6V
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:13:28 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AizQh-0007iv-TT
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:13:15 -0500
Received: from [66.93.249.120] (helo=pixie.madstop.com)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AizQh-0007in-Eq
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:12:43 -0500
Received: by pixie.madstop.com (Postfix, from userid 100)
	id 1B92E4C031; Tue, 20 Jan 2004 11:12:42 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
	by pixie.madstop.com (Postfix) with ESMTP
	id 17D84560C5; Tue, 20 Jan 2004 11:12:42 -0600 (CST)
Date: Tue, 20 Jan 2004 11:12:42 -0600 (CST)
From: "Luke A. Kanies" <luke@madstop.com>
X-X-Sender: luke@pixie
To: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
In-Reply-To: <400BC387.9050904@sara.nl>
Message-ID: <Pine.GSO.4.51.0401201043010.14572@pixie>
References: <200401180956.i0I9ue2K020739@nexus.iu.hio.no>
	<400BC387.9050904@sara.nl>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Cc: Cfengine Developers <cfengine-develop@nongnu.org>
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 17:13:30 -0000

On Mon, 19 Jan 2004, Bas van der Vlies wrote:

> Mark.Burgess@iu.hio.no wrote:
>
> Luke,
>
> can you not explain to our small list what you see as your vision for loops
> in cfengine and what you mean by "abstraction" etc with sufficient detail
> so that we can determine the best solution and move on constructively?
> We should do this soon, while I have time to listen to the discussion.
>
> I would also like input from the rest of the developers list, especially
> on how to implement the solution without damaging  backward compatibility
> or convergence.

Here are some examples of abstraction that is difficult or impossible in
cfengine.

Inetd:

I build my inetd file from scratch using cfengine.  As I'm sure you all
know, an inetd config file is composed of individual configurations for
many daemons.  Each daemon configuration is on its own line, and each line
has a fixed number of fields.  In perl, you would describe it a lot like
this (I'm using the field names from memory, so they're not correct):

'telnet' => {
	binary => '/usr/sbin/in.telnetd',
	arguments => '',
	user => 'root',
	protocol => 'tcp',
	wait => 'nowait',
	...
}

Okay, so I want to use cfengine to selectively enable services on hosts,
something like the following:

control:
	server1::
		<enable telnet>
		<enable ftp>
	server2::
		<enable ftp>
		<enable rstatd>

That <enable telnet> needs to result in the configuration for telnet being
put into inetd.conf.  In other words, that short statement somehow needs
to refer to the entire configuration of telnet, along with which file it
goes into and what class to set so that inetd gets HUPed if something
changes.  This is abstraction:  I'm abstracting "enable telnet" into a
specific configuration, where it goes, and how to restart inetd if the
config changes.

If I wanted to do this with cfengine right now without stepping outside of
cfengine at all, it would be, um, messy.  It would probably look something
like this:

control:
	telnet_binary = ( "/usr/sbin/in.telnetd" )
	telnet_user = ( "root" )
	telnet_protocol = ( "tcp" )
	...

editfiles:
	server1::
		{ ${inetd_conf}
			SetLine "telnet ${telnet_protocol} ..."
			AppendIfNoLineMatching telnet
			SetLine "ftp ${ftp_protocol} ..."
			AppendIfNoLineMatching ftp
			Define restart_inetd
		}
	server2::
		{ ${inetd_conf}
                        SetLine "ftp ${ftp_protocol} ...
                        AppendIfNoLineMatching ftp
			SetLine "rstatd ${rstatd_protocol} ..."
                        AppendIfNoLineMatching rstatd
			Define restart_inetd
                }

Notice that I had to repeat a bunch of information each time:  I had to
repeat the format for the inetd config file, I had to repeat how to get
the data into the file, I had to repeat the definition of whether the data
was actually in there or not (i.e., whether _any_ ftp configuration is
acceptable, or only that exact configuration).  In addition, once I've
defined for server1 how to add ftp, I have to repeat that exact
configuration all over again.

Sure, I could have switched it around so that it was "server1|server2"::,
but that would have reduced abstraction in a different direction.

Notice also that I have about six variables for every single inetd
daemon, and setting them all is incredibly messy.

Now, compare that to how I am currently doing this at one of my client
sites:

I have an external data file that is essentially one big hash file,
containing every single inetd config possible across all of my platforms.
I have named each of these configs something symbolic to me but
essentially meaningless to the software (e.g., "telnet_sun", "telnet_hp").
I have a simple perl script which accepts a list of these names on the
command line, compiles the actual configurations from the data file, and
compares the result to the file on disk.  If the file on disk is out of
data, the script overwrites it and prints out "+restart_inetd", thus
notifying cfengine that inetd needs to be restarted.

Within cfengine, this is what I do:

control:
	server1::
		inetd_svcs = ( "${inetd_svcs} telnet_sun ftp_sun" )
	server2::
		inetd_svcs = ( "${inetd_svcs} ftp_sun rstatd_sun" )

This reduces the amount of repetition merely to the symbolic name I am
using for servers.  It's trivial to read and understand, and it works
very, very well.  I consider this to be completely within "the cfengine
way" because the process of writing the inetd.conf file is idempotent; the
perl script compares the in-memory file to the on-disk file, and only
writes it and sets the class if the files are different.

This is generally what I mean by "abstraction", and if you can find a
means of providing this level of functionality within cfengine without a)
stepping outside of cfengine with an external script, or b) generating
cfengine code, then you're much more capable than I.

Packages:

An even messier problem is the way that I'm installing packages using
cfengine.

I have created a packaging system entirely using cfengine and some
external code generation scripts.  I really, really like it because I can
create a cfengine configuration script for each package and know that
every package set up this way will always be set up correctly forever.

However, to do this, I am having to generate all of the code that actually
does any work.

I have a list of packages and revisions, e.g., perl 5.8.2, cfengine
2.1.0p1, cfengine 2.0.7, openssl 0.9.7b, etc.  I want to be able to
install multiple revisions of the same package, but I want one specific
version to be listed as the "current"; its files will be directly linked
to /usr/local (e.g., the current cfengine will be linked as
/usr/local/sbin/cfagent, but other versions will be
/usr/local/sbin/cfagent-<version>).

Cfengine needs to copy the files for the package down, link the files into
place (ignoring different directories to copy for each package; I seldom
want to copy info, but sometimes I have to copy share), and then verify
the package is running correctly.  I also need the ability to configure
different versions of a package differently.

I have absolutely no idea how you would do this without generating code.
Really.  No idea.

What I am doing is something that works but with which I'm not terribly
happy.

I do the following:

control:
	server1::
		pkgs = ( "${pkgs} cfengine--2.1.0p1 openssl--0.9.7c" )
	server2::
		pkgs = ( "${pkgs} wuftp-1.0" )

Then I have a module which accepts that list of packages and generates
cfengine code that does the copying and linking.  It then includes a
cfengine file at $CFINPUTS/packages/<package>.cf and/or
$CFINPUTS/packages/<package>-<revision>.cf to execute after the package is
copied.  The copies only happen once a day, but the links and
configuration happen every time cfengine runs.

I know that I could do the actual work within a cfengine method, but how
would I iterate across the list of packages?  In other words, how do I
abstract the list of packages I want to install from how they are actually
being installed?

This is what i mean by "abstraction".

I would be glad to give more information.  I am hoping to give a talk on
this packaging system at OSCON in Portland, OR this year, and I may submit
it as a LISA talk, also.

Luke

-- 
Westheimer's Discovery:
        A couple of months in the laboratory can frequently save a
        couple of hours in the library.



From MAILER-DAEMON Tue Jan 20 12:13:49 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AizRk-00084M-RX
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 12:13:48 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AizRc-00082b-4u
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:13:40 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AizR2-0007sI-UY
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:13:36 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AizB5-0003dD-5x
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 11:56:35 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.11/8.12.11) with ESMTP id i0KGuIwV003279;
	Tue, 20 Jan 2004 17:56:22 +0100 (MET)
Message-Id: <200401201656.i0KGuIwV003279@nexus.iu.hio.no>
Date: Tue, 20 Jan 2004 17:56:18 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0401151024090.9282@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org, Mark.Burgess@iu.hio.no
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 17:13:46 -0000


> I would probably do something that looked a
> heckuva lot like ruby, rather than perl:
> 
> control:
> 	list = ( 1 2 3 4 )
> 
> copy:
> 	${list}.each { |item|
> 		/path/to/files/${item} dest=/new/path/${item}
> 	}
> 
> links:
> 	${list}.each { |item|
> 		/usr/local/bin -> /usr/local/software/${item}/bin/*
> 	}
> 
> I'm not saying that this is _the_ solution, but once you get used to it,
> it continues to make sense.



What would nested iteration look like in this scheme?


> Another option is to implicitly support iteration so that the syntax does
> not change at all:
> 
> control:
> 	list = ( 1 2 3 4 )
> 
> copy:
> 	/path/to/files/${list} dest=/new/path/${list}
> 
> links:
> 	/new/place/${list} -> /old/place/${list}
> 
> I know links already kind of supports this, but it's using a standard
> string variable, not an actual list variable.  One of the real problems
> with this type of iteration is that you can't then do nested iterations,
> and you can't really do anything resembling hashes (associative arrays)
> either, and I think they're also important.

Cfengine does support nested iteration (see so-called 2d lists).
It is just not allowed on all variables.

It strikes me that one of the biggest problems in iteration is
what happens when there is nesting and how it should be evaluated.

Today, iteration lacks conceptual clarity in cfengine, but it does
work.

M

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Tue Jan 20 12:21:10 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AizYr-0001H7-OK
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 12:21:09 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AizYp-0001GS-0z
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:21:07 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AizYI-00016Q-9R
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:21:05 -0500
Received: from [66.93.249.120] (helo=pixie.madstop.com)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AizYH-00015v-OL
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:20:33 -0500
Received: by pixie.madstop.com (Postfix, from userid 100)
	id B38B44C042; Tue, 20 Jan 2004 11:20:30 -0600 (CST)
Received: from localhost (localhost [127.0.0.1])
	by pixie.madstop.com (Postfix) with ESMTP
	id A9968560C5; Tue, 20 Jan 2004 11:20:30 -0600 (CST)
Date: Tue, 20 Jan 2004 11:20:30 -0600 (CST)
From: "Luke A. Kanies" <luke@madstop.com>
X-X-Sender: luke@pixie
To: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
In-Reply-To: <200401201656.i0KGuIwV003279@nexus.iu.hio.no>
Message-ID: <Pine.GSO.4.51.0401201113050.14572@pixie>
References: <200401201656.i0KGuIwV003279@nexus.iu.hio.no>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 17:21:08 -0000

On Tue, 20 Jan 2004 Mark.Burgess@iu.hio.no wrote:

>
> > I would probably do something that looked a
> > heckuva lot like ruby, rather than perl:
> >
> > control:
> > 	list = ( 1 2 3 4 )
> >
> > copy:
> > 	${list}.each { |item|
> > 		/path/to/files/${item} dest=/new/path/${item}
> > 	}
> >
> > links:
> > 	${list}.each { |item|
> > 		/usr/local/bin -> /usr/local/software/${item}/bin/*
> > 	}
> >
> > I'm not saying that this is _the_ solution, but once you get used to it,
> > it continues to make sense.
>
>
>
> What would nested iteration look like in this scheme?

control:
	list1 = ( 1 2 3 4 )
	list2 = ( a b c d )

copy:
	${list1}.each { |number|
		${list2}.each { |letter|
			/${number} dest=/${letter}
		}
	}

This is pretty much straight out of ruby.  I think this exact syntax would
not work, because variables aren't objects so the concept of calling a
method on a variable (".each") isn't really valid.  I like the basic idea,
though.


> Cfengine does support nested iteration (see so-called 2d lists).
> It is just not allowed on all variables.

I could not find any mention of 2d lists in the cfengine reference, and
none of the mentions of lists seemed to be speaking about them.  Where is
this described?

> It strikes me that one of the biggest problems in iteration is
> what happens when there is nesting and how it should be evaluated.

I think the above system is pretty clear and clean, but it requires a
number of things cfengine doesn't have, with variable scope being near the
top of the list.

> Today, iteration lacks conceptual clarity in cfengine, but it does
> work.

Yeah, I just find that it never works in a way that's useful to me.
AFAIK, it only works in shellcommands and links, with some strange
implicit iteration in other areas (like the 'home' directive).

I might be able to figure out what's happening, but I'm essentially
positive that I could not comfortably hand such iteration off to someone
else.  Either way, though, the available iteration has never satisfied my
needs for abstraction, largely because it needs to be configuration-wide,
not just specific actions.

Luke

-- 
    "What is the sound of Perl?  Is it not the sound of a wall that
     people have stopped banging their heads against?"
                --Larry Wall in <1992Aug26.184221.29627@netlabs.com>



From MAILER-DAEMON Tue Jan 20 12:25:51 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AizdP-0002QZ-1m
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 12:25:51 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AizdM-0002Pw-NH
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:25:48 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1Aizck-0002IX-Uk
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:25:45 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1Aizck-0002IH-2n
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:25:10 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.11/8.12.11) with ESMTP id i0KHOwYK004021;
	Tue, 20 Jan 2004 18:25:02 +0100 (MET)
Message-Id: <200401201725.i0KHOwYK004021@nexus.iu.hio.no>
Date: Tue, 20 Jan 2004 18:24:58 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0401201043010.14572@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org, Mark.Burgess@iu.hio.no
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 17:25:49 -0000


Ok - it strikes me that you could solve several of the problems
quite eloquently with the new methods -- I have an example of
unpacking/compiling/installing a tarfile for instance. 

I do not recall off the top of my head how iteration works in methods.
But that is something that can be looked at. My strategy (as always) has been
to start simply and create a robust substrate on which to build.

Many people ram down my throat ideological misconceptions about cfengine.
My main ideology is that each operation should be safe and convergent,
observing the best security procedures that I know of. That is the
way I think we should build it in the future.

(I want to resist "bloat/trends" becaue they happens anyway and makes a mess
that later is just a nuisance.)

Mark

On 20 Jan, Luke A. Kanies wrote:
> On Mon, 19 Jan 2004, Bas van der Vlies wrote:
> 
>> Mark.Burgess@iu.hio.no wrote:
>>
>> Luke,
>>
>> can you not explain to our small list what you see as your vision for loops
>> in cfengine and what you mean by "abstraction" etc with sufficient detail
>> so that we can determine the best solution and move on constructively?
>> We should do this soon, while I have time to listen to the discussion.
>>
>> I would also like input from the rest of the developers list, especially
>> on how to implement the solution without damaging  backward compatibility
>> or convergence.
> 
> Here are some examples of abstraction that is difficult or impossible in
> cfengine.
> 
> Inetd:
> 
> I build my inetd file from scratch using cfengine.  As I'm sure you all
> know, an inetd config file is composed of individual configurations for
> many daemons.  Each daemon configuration is on its own line, and each line
> has a fixed number of fields.  In perl, you would describe it a lot like
> this (I'm using the field names from memory, so they're not correct):
> 
> 'telnet' => {
> 	binary => '/usr/sbin/in.telnetd',
> 	arguments => '',
> 	user => 'root',
> 	protocol => 'tcp',
> 	wait => 'nowait',
> 	...
> }
> 
> Okay, so I want to use cfengine to selectively enable services on hosts,
> something like the following:
> 
> control:
> 	server1::
> 		<enable telnet>
> 		<enable ftp>
> 	server2::
> 		<enable ftp>
> 		<enable rstatd>
> 
> That <enable telnet> needs to result in the configuration for telnet being
> put into inetd.conf.  In other words, that short statement somehow needs
> to refer to the entire configuration of telnet, along with which file it
> goes into and what class to set so that inetd gets HUPed if something
> changes.  This is abstraction:  I'm abstracting "enable telnet" into a
> specific configuration, where it goes, and how to restart inetd if the
> config changes.
> 
> If I wanted to do this with cfengine right now without stepping outside of
> cfengine at all, it would be, um, messy.  It would probably look something
> like this:
> 
> control:
> 	telnet_binary = ( "/usr/sbin/in.telnetd" )
> 	telnet_user = ( "root" )
> 	telnet_protocol = ( "tcp" )
> 	...
> 
> editfiles:
> 	server1::
> 		{ ${inetd_conf}
> 			SetLine "telnet ${telnet_protocol} ..."
> 			AppendIfNoLineMatching telnet
> 			SetLine "ftp ${ftp_protocol} ..."
> 			AppendIfNoLineMatching ftp
> 			Define restart_inetd
> 		}
> 	server2::
> 		{ ${inetd_conf}
>                         SetLine "ftp ${ftp_protocol} ...
>                         AppendIfNoLineMatching ftp
> 			SetLine "rstatd ${rstatd_protocol} ..."
>                         AppendIfNoLineMatching rstatd
> 			Define restart_inetd
>                 }
> 
> Notice that I had to repeat a bunch of information each time:  I had to
> repeat the format for the inetd config file, I had to repeat how to get
> the data into the file, I had to repeat the definition of whether the data
> was actually in there or not (i.e., whether _any_ ftp configuration is
> acceptable, or only that exact configuration).  In addition, once I've
> defined for server1 how to add ftp, I have to repeat that exact
> configuration all over again.
> 
> Sure, I could have switched it around so that it was "server1|server2"::,
> but that would have reduced abstraction in a different direction.
> 
> Notice also that I have about six variables for every single inetd
> daemon, and setting them all is incredibly messy.
> 
> Now, compare that to how I am currently doing this at one of my client
> sites:
> 
> I have an external data file that is essentially one big hash file,
> containing every single inetd config possible across all of my platforms.
> I have named each of these configs something symbolic to me but
> essentially meaningless to the software (e.g., "telnet_sun", "telnet_hp").
> I have a simple perl script which accepts a list of these names on the
> command line, compiles the actual configurations from the data file, and
> compares the result to the file on disk.  If the file on disk is out of
> data, the script overwrites it and prints out "+restart_inetd", thus
> notifying cfengine that inetd needs to be restarted.
> 
> Within cfengine, this is what I do:
> 
> control:
> 	server1::
> 		inetd_svcs = ( "${inetd_svcs} telnet_sun ftp_sun" )
> 	server2::
> 		inetd_svcs = ( "${inetd_svcs} ftp_sun rstatd_sun" )
> 
> This reduces the amount of repetition merely to the symbolic name I am
> using for servers.  It's trivial to read and understand, and it works
> very, very well.  I consider this to be completely within "the cfengine
> way" because the process of writing the inetd.conf file is idempotent; the
> perl script compares the in-memory file to the on-disk file, and only
> writes it and sets the class if the files are different.
> 
> This is generally what I mean by "abstraction", and if you can find a
> means of providing this level of functionality within cfengine without a)
> stepping outside of cfengine with an external script, or b) generating
> cfengine code, then you're much more capable than I.
> 
> Packages:
> 
> An even messier problem is the way that I'm installing packages using
> cfengine.
> 
> I have created a packaging system entirely using cfengine and some
> external code generation scripts.  I really, really like it because I can
> create a cfengine configuration script for each package and know that
> every package set up this way will always be set up correctly forever.
> 
> However, to do this, I am having to generate all of the code that actually
> does any work.
> 
> I have a list of packages and revisions, e.g., perl 5.8.2, cfengine
> 2.1.0p1, cfengine 2.0.7, openssl 0.9.7b, etc.  I want to be able to
> install multiple revisions of the same package, but I want one specific
> version to be listed as the "current"; its files will be directly linked
> to /usr/local (e.g., the current cfengine will be linked as
> /usr/local/sbin/cfagent, but other versions will be
> /usr/local/sbin/cfagent-<version>).
> 
> Cfengine needs to copy the files for the package down, link the files into
> place (ignoring different directories to copy for each package; I seldom
> want to copy info, but sometimes I have to copy share), and then verify
> the package is running correctly.  I also need the ability to configure
> different versions of a package differently.
> 
> I have absolutely no idea how you would do this without generating code.
> Really.  No idea.
> 
> What I am doing is something that works but with which I'm not terribly
> happy.
> 
> I do the following:
> 
> control:
> 	server1::
> 		pkgs = ( "${pkgs} cfengine--2.1.0p1 openssl--0.9.7c" )
> 	server2::
> 		pkgs = ( "${pkgs} wuftp-1.0" )
> 
> Then I have a module which accepts that list of packages and generates
> cfengine code that does the copying and linking.  It then includes a
> cfengine file at $CFINPUTS/packages/<package>.cf and/or
> $CFINPUTS/packages/<package>-<revision>.cf to execute after the package is
> copied.  The copies only happen once a day, but the links and
> configuration happen every time cfengine runs.
> 
> I know that I could do the actual work within a cfengine method, but how
> would I iterate across the list of packages?  In other words, how do I
> abstract the list of packages I want to install from how they are actually
> being installed?
> 
> This is what i mean by "abstraction".
> 
> I would be glad to give more information.  I am hoping to give a talk on
> this packaging system at OSCON in Portland, OR this year, and I may submit
> it as a LISA talk, also.
> 
> Luke
> 



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Tue Jan 20 13:23:40 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1Aj0XL-000379-Km
	for mharc-cfengine-develop@gnu.org; Tue, 20 Jan 2004 13:23:39 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AizuK-00038C-Rv
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:43:20 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1Aizqd-0000rT-NZ
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:40:09 -0500
Received: from [199.232.41.8] (helo=mx20.gnu.org)
	by monty-python.gnu.org with esmtp (TLSv1:DES-CBC3-SHA:168)
	(Exim 4.24) id 1AizoT-0006ZQ-61
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:37:17 -0500
Received: from [128.39.89.10] (helo=nexus.iu.hio.no)
	by mx20.gnu.org with esmtp (Exim 4.24) id 1AizlP-0003i8-EB
	for cfengine-develop@nongnu.org; Tue, 20 Jan 2004 12:34:07 -0500
Received: from iu.hio.no (nexus.iu.hio.no [128.39.89.10])
	by nexus.iu.hio.no (8.12.11/8.12.11) with ESMTP id i0KHXo5p004484;
	Tue, 20 Jan 2004 18:33:54 +0100 (MET)
Message-Id: <200401201733.i0KHXo5p004484@nexus.iu.hio.no>
Date: Tue, 20 Jan 2004 18:33:50 +0100 (MET)
From: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
To: luke@madstop.com
In-Reply-To: <Pine.GSO.4.51.0401201113050.14572@pixie>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
X-Test-Ignore: Test
X-SpamAssassin-Score: 0.339 NO_REAL_NAME
X-Scanned-By: MIMEDefang 2.37
Cc: cfengine-develop@nongnu.org, Mark.Burgess@iu.hio.no
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Jan 2004 17:46:06 -0000


You are right the the multiple iteration seems only to be mentioned
in shellcommands, but it also works elsewhere.

That is a flaw in documentation.

M

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Work: +47 22453272            Email:  Mark.Burgess@iu.hio.no
Fax : +47 22453205            WWW  :  http://www.iu.hio.no/~mark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~




From MAILER-DAEMON Wed Jan 21 08:49:23 2004
Received: from list by monty-python.gnu.org with archive (Exim 4.24)
	id 1AjIjS-0000vR-J5
	for mharc-cfengine-develop@gnu.org; Wed, 21 Jan 2004 08:49:22 -0500
Received: from list by monty-python.gnu.org with tmda-scanned (Exim 4.24)
	id 1AjIjN-0000st-Oe
	for cfengine-develop@nongnu.org; Wed, 21 Jan 2004 08:49:17 -0500
Received: from mail by monty-python.gnu.org with spam-scanned (Exim 4.24)
	id 1AjIip-0000gN-6p
	for cfengine-develop@nongnu.org; Wed, 21 Jan 2004 08:49:14 -0500
Received: from [145.100.16.144] (helo=mta.sara.nl)
	by monty-python.gnu.org with esmtp (Exim 4.24) id 1AjIQb-0004rR-4z
	for cfengine-develop@nongnu.org; Wed, 21 Jan 2004 08:29:53 -0500
Received: from sara.nl (bas.ka.sara.nl [145.100.6.35])
	by mta.sara.nl (8.11.3/8.11.3) with ESMTP id i0LDTWJ01350;
	Wed, 21 Jan 2004 14:29:32 +0100 (MET)
Message-ID: <400E7EB4.1070404@sara.nl>
Date: Wed, 21 Jan 2004 14:29:24 +0100
From: Bas van der Vlies <basv@sara.nl>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
	rv:1.6b) Gecko/20031205 Thunderbird/0.4
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: Mark.Burgess@iu.hio.no
Subject: Re: [Cfengine-develop] my own branch of the tree
References: <200401201725.i0KHOwYK004021@nexus.iu.hio.no>
In-Reply-To: <200401201725.i0KHOwYK004021@nexus.iu.hio.no>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Cc: cfengine-develop@nongnu.org
X-BeenThere: cfengine-develop@nongnu.org
X-Mailman-Version: 2.1.2
Precedence: list
List-Id: Cfengine developers  <cfengine-develop.nongnu.org>
List-Unsubscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>, 
	<mailto:cfengine-develop-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://mail.gnu.org/pipermail/cfengine-develop>
List-Post: <mailto:cfengine-develop@nongnu.org>
List-Help: <mailto:cfengine-develop-request@nongnu.org?subject=help>
List-Subscribe: <http://mail.nongnu.org/mailman/listinfo/cfengine-develop>,
	<mailto:cfengine-develop-request@nongnu.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Jan 2004 13:49:20 -0000

Just my 2 cents.

When we just started the development team there was also much talking 
about the issues about the parser, lists and associative arrays (hashes 
in perl and dictionaries in python, every language has is own description).

I like the idea to have list and associative arrays. Now it depends on 
the actionsquence if a variable is interpeted as has list or a string.
So if the parser supports this kind of type it makes it much cleaner.

In the early days of the development team i had some suggestions for the
parser/lexer. It has changed and i did not touch it. It always good to 
look how other projects have done the  parser/lexer stuff. I still think 
a lot of the C-code in cfengine to check if we have supported the right 
kind of variable (eg ip-address, numbers, etc) can be done in 
lexer/parser which elminates a lot of code. But Mark is right that this 
is a lot of work, bu we have to start somewhere. If the parser/lexer is 
good then it can also be easily extended with new types and functions.

-- 
--
********************************************************************
*                                                                  *
*  Bas van der Vlies                     e-mail: basv@sara.nl      *
*  SARA - Academic Computing Services    phone:  +31 20 592 8012   *
*  Kruislaan 415                         fax:    +31 20 6683167    *
*  1098 SJ Amsterdam                                               *
*                                                                  *
********************************************************************



