[dojo-contributors] Dojo Utilities?

So I've always been +1 on breaking out pieces of our repository, and utils
seems to be a great place to start. Not only would it enable a separate
development and release cycle, but it would help bring down the size and
complexity of our distribution. There are many web developers who would
feel overwhelmed by other languages and technologies besides html/css, and
this particular app can ultimately be provided "in the cloud", in which case
many developers would not require the source at all, so I think this is a
great candidate.

-Adam

ok with goal of "breaking out pieces of our repository", but there are some
things that need to be ironed out ahead of time...
-Chris

1) Is this new GitHub project going on James' personal github account? If
so, this will be a problem...if James were to go away for whatever reason
(not that that will happen any time soon). I hear there is a Dojo
Foundation account that can be used... Where are the details for how we can
establish GitHub projects maintained under this account.

2) The Web Builder is new code that hasn't existed in dojo before, so it's
easy easier to break out as it's own thing now. It has client side parts
that can follow the package metadata spec from Ttrenka, but what are
considerations for package structure of the server pieces (Java-based) in
this case.

3) For existing project's already part of Dojo distro's, for example gfx
that we want to split out to github, I believe we need to address (1) above,
and mandate that the separating packages follow ttrenka's package metadata
format. We really need an exemplar project here, that illustrates how
developers should structure docs, tests, licenses, etc. so that Dojo.next
can be aggregated back together--not just from a JS module perspective.
Until an exemplar project is put together and an example of how the
extracted git projects can be re-aggregated to form Dojo.next, we should be
very careful...other wise Dojo WILL fragment, losing the value it's built up
to companies in the past.
a) Docs need to be structured (phiggins' work should help here, but
guidelines still need to be written) so that they can be pulled together
into a reference guide. I don't think James' project is the one to use for
this because of (2)).
b) We need to be able to run tests across sanctioned projects as a whole,
to ensure the separated modules continue work properly together.

4) If we don't have the ability to pull separated projects back together
(3), it will be untenable for companies (like IBM) to do the legal work
necessary to make Dojo's pedigree the quality it needs to be for enterprise
customers. This is not a small amount of work, but is something the OSS
community needs to be aware of in terms of downstream consumption in large
companies...it is important to continue to maintain our quality around
pedigree when things split out, and doing this on an aggregate distro is the
only feasible way right now.

-Chris



2011/3/10 Adam L. Peller <adam at peller.org>

Then other forks of the project would just continue to thrive. Welcome
to the world of distributed version control ;) One person can't kill a
distributed project in this world.

Joking aside, it certainly can be better organized and more robust to
actually use an organization account in git. Maybe you could setup an
ibm-dojo account for the group of devs over there?
This is possible to, but this seems counterproductive in terms of
granularity of control and ownership which is part of the reason for
breaking DojoX up. I think it would be better to have smaller units of
organization that can be more specifically controlled.

I think it beyond the scope of our package structure to dictate server
structure (and there are plethora of languages that would have to be
defined), and I think there are already best practices for organizing
Java projects.
Here is a template I put together. I am sure it will continue to evolve
as we work through issues and clarify things:
https://github.com/kriszyp/commonjs-package-template
I agree, it is important to keep the necessary consistency in projects
to make this continue to be achievable.

Thanks,
Kris

Except where code is under CLA, it would be preferable, I think, to have
access control which assures the origins conform to the contributor list.
Perhaps github could help us track or even automate this. For some
projects, we may wish to restrict access further, others would be free to
develop even without CLA if they choose. I think we were talking about
having a bit in the project metadata to indicate CLA status, which might be
further limited by a subset which indicates Dojo "ownership" for
maintenance.

-Adam

Yes, we should maintain a list of CLA-acceptable github accounts and
then projects from those accounts could designate whether or not they
are CLA conformant (for the Dojo foundation). I would assume a ibm-dojo
account could be on that list.
Kris

All good questions. ttrenka can correct me if I'm wrong, but I don't think
the package structure applies to webapps like this so much, so perhaps
James' project is an opportunity to explore the github infrastructure/access
control issues and ease our way into this, without worrying about other
issues of drag and dependencies just yet.

2011/3/10 Chris Mitchell <ccmitchellusa at gmail.com>

I would agree. This is much more of an application and much less
something that would be distributed as part of a toolkit or framework,
so it would be easier to release as standalone and let the Dojo
community promote it separately.

I do have thoughts about the other things mentioned on this thread,
but at the moment I'm trying to focus and will respond later, if
that's ok.

Regards--
Tom

2011/3/10 Adam L. Peller <adam at peller.org>:

@Tom - There are a number of applications (tools) that have been distributed
with Dojo toolkit in the past,

- Doc to OAA metadata tool
- API Doc tool
- CLDR build tool
- Shrinksafe

I really don't view James application (tool) any different than these.

Again, I'm agreeing that it makes sense to separate these tools out going
toward Dojo into Dojo Foundation owned, individually run projects. All of
the current code was contributed to Dojo Foundation under CLA which
maintains Dojo Foundations copyrights and standards on pedigree and quality
wherever possible.

@Kris - It's not that a distributed version control system allows forking
that is the problem here. The problem I'm trying to point out is that
putting CLA'd code contributed to the foundation in individual's (or
corporate's) github accounts goes against the whole notion of contributing
the code to the foundation in the first place under CLA. Sure, IBM is very
capable of creating its own corporate account to hold all of our code
contributes, and we could just as easily place those contributions under
under an IBM open source LICENSE, but we would prefer not to do that, and
instead contribute it to the Dojo Foundation, in a neutral 503c(6) org in a
place that the foundation ensures will keeps the code that's been
contributed available as people come and go from the community, so that it
can live on without dependency on our account.

I just want to make sure we have a plan for how companies can continue get a
full distro containing the founddation contributed pieces as they've been
able to do it in the past, with the quality they expect not just around
code, but tests, docs, etc. There are many companies that need to pull the
aggregate distro, and not just pieces.

Regarding the backend architecture of these applications, it does matter to
some degree, in terms of tool chain dependencies. If I need to get Ruby,
Python, Java, PHP, and whatever latest and greatest scripting language in
order to do basic internationalization, build api docs for my code, optimize
my code, and test my code, there's huge complexity involved and we'd be
failing our users. Of course, that's not the case... we've made technology
decisions when building these tools to reduce the tool chain dependencies to
our users. When we separate off projects, I would hope we could give some
guidance as to what's expected when providing tools for Dojo, so that the
tools can be easily used with others. This cross-tool project guidance is
outside the scope of package metadata spec.

I'm not trying to debate the merit of which dojox projects should live and
die. As you say, that will happen on it's own over time.

I like the goal of splitting these out assuming we can keep quality control
in place and not leave ourselves, or any customers to having to go find all
the pieces themselves, do the testing, etc. that we've been doing for them
in the past.

-Chris

This brings up a good question, and something posed privately by
Dylan...for those companies that distribute some version of DTK, do we
need to move things to a shared Dojo Foundation code-hosting solution?
We'd discussed (and even setup) Github accounts for each of the
individual projects, but maybe we really need to have a master Dojo
Foundation account, under which things live to satisfy some of the
legal requirements of our partners...I don't know Github well enough
to know if it is possible to limit commit access based on individual
projects or not but perhaps this is something to investigate sooner
than later.

With regards to the projects you mentioned above, Chris: all four of
those projects are considered to be core functionality to be used by
developers, and have minimum requirements (basically a Java instance
and a PHP one). James' web builder requires a bit more than that IIRC
(a Java server like Tomcat, plus others). We don't distribute the
actual API doc website tool (what is running at dtk.org/api/), and I
think James' work falls into that category. (We do distribute the
actual doc parser, but not the corresponding website--that lives in a
DTK SVN repo that is separate from DTK itself).

Regards--
Tom

How is this any better than keeping a list of Dojo foundation approved
accounts/projects? If I want Joe's code in my codebase, I can copy it in
just as easily as I can grant access to someone to add it. Ultimately it
comes down to Dojo committers making proper use of their authority and
delegation of it. If we maintain a list of what we can safely aggregate
into a distribution, that gives us as committers final control over what
can and can't be included in the distribution.
Kris

I think the question posed by Chris is less about keeping some sort of
big repository and more about making sure the hosting itself helps,
from a legal standpoint, to show clear IP and "sanctioned" packages.

From a practical perspective, at least with git, I would have expected that either we would do one of the following:

a) Maintain a DojoX repository which is essentially a top level structure, perhaps with some boiler plate files/readmes/docs/whatever. Then simply add a git subproject for each of those modules we are 'officially' endorsing (have been vetted by some quality/conformance metrics and are legally clean). These are then available via the single wrapper project, but they also don't 'freewheel' follow the parent repository. They essentially stick to whatever revision you checkin the submodule at until you update and push this back.

b) Alternatively, or in addition to, we could simply make a standard practice of making a fork at the dojofoundation for any projects we include in our current/past distributions. That doesn't mean we are actually forking their projects to apply changes, it can be a shadow of the master. It is simply a way to eliminate any risk of a repository suddenly disappearing while be included in our 'official' package set.

In the end I would expect that some modules may live in the Dojotoolkit Organizational Account and others in individual or or other organization accounts. The one plus of forcing someone to take ownership is that it can lead to an automatic distribution of work and makes stale modules/projects drop away on their own when the owner isn't able to maintain the expected criteria.

Dustin

i was thinking along the same lines as Dustin. by using submodules or
forks, we can put our stamp of approval on a project up until the
specific commit we fork or checkin as a submodule. this gives us the
chance to make sure that project owners are doing the right thing before
we update our fork/submodule.

i was also thinking that packages in our package repository may have
some kind of meta data that indicated their standing wrt a dojo
foundation stamp of approval. perhaps even a separate repository/url
that only contained approved packages.

ben...

To be clear, we already have a dojo foundation account in github:

https://github.com/dojo

That is not a Foundation account; it is one of the three accounts
following the Dojo Toolkit. The Foundation is a different entity (and
sorry to pursue semantics, but when you get hounded by people being
pushed by lawyers...).

An actual foundation account should be made, controlled by the
Foundation and not anyone associated with a particular project,
regardless of how prominent that project is.

-- Tom

2011/3/11 Bill Keese <bill at dojotoolkit.org>:

Hmm, it's unfortunately named then. Don't we already have some group
account set up? I'm sure people have told me about it before. (I wasn't
the one that set it up but I've heard of it.)

PS: the account is titled: dojo *(Dojo Foundation)*
It does have the dojo toolkit icon though, should be changed to the one on
dojofoundation.org.

Yeah I was confused too. It sounds kind of like Tom is describing the
repository https://github.com/dojo/dojo, rather than the parent account
itself, https://github.com/dojo

I have one other question relevant to this thread (though admittedly at
this point it may be less relevant to the OP) - for packages like
dojox.widget, which contain lots of mostly-standalone components rather
than a cohesive set, is there any thought to how this will be broken
out? i.e. will it still be one single monolithic package, or is there
any thought to allowing it to be further atomized? (It's quite possible
some consumers might be interested in a few, but not all of them.)

--Ken

That it is. But descriptions can be unfortunate/deceiving; this
particular account is part of the effort Eugene made (and Dustin
modded) as the github equivalents of our SVN versions of dojo, dijit
and dojox. I'm thinking there needs to be a pure Foundation account.
There are a number of different projects under the aegis of the Dojo
Foundation; DTK may be the most visible but it isn't the only one.

-- Tom

Not saying this is or should be the account, but we did change it to an organization account from a normal account so it can be managed like that. Give people privs to edit on one project vs the next and that kind of thing (which is we changed it to an org acct). This can include giving admin privs for it to the appropriate people at the foundation as well as including other projects and sub projects.

Dustin

Not sure who can do it, but would it be possible to talk about changes
to that account (like the overall name) so that it is the actual Dojo
Foundation, along with gravatar? Or is the something that should be
discussed by the foundation officers first?

-- Tom

I've made the foundation board aware of the conversation.

Right now, I think the best thing is for us to all figure out what we
want to do, and then we'll do it.

I think it would make sense to create a short list of the concerns and
issues we're trying to address, solve them, and then we'll be able to
come up with a clear set of decisions.

I see the following as what has been discussed so far:

* Simplicity and cohesion, filtering, marketing
* Flexibility/modularity
* Ownership/official versions
* Ease of contributing and growing community, while under CLA and IP safe
* Version compatibility (don't turn into jar hell or linux dependency
management hell)
* Testing releases
* Documentation (accuracy, completeness, aggregation by release)
* Risk management (relying on a single commercial service)
* Module owner gets bored/abandons, what becomes official next
* Structure of projects (foundation, within dtk)
* Management of the official Dojo Foundation github projects, managing
patches, etc.

I think most of these have been solved or have solutions on the way, but
feel free to add answers or clarify, or add points I've missed.

It would probably make sense to put this into an etherpad or google doc
or wiki page, give everyone that cares time to comment and hammer out
the details, and then we'll do what needs to get done.

Regards,
- -Dylan

on 3/12/11 8:55 AM (GMT-07:00) Tom Trenka said the following:

I generally agree, a few comments inline.
The overall name is "Dojo Foundation" currently, though I assume you are referring to the 'dojo' portion of that. It is not easy to change the 'dojo' portion of that. Would be easier to just create a new account. Gravatar and all that is trivial to change of course too.

Dustin

Also agree with the great list you put together, Dylan. @Dustin: OK,
that's fine; I can envision a bit of confusion between "dojo" the
foundation and "dojo" the project though, which is the only reason I
asked. It also looks like that if this remains on github, we'll
probably have to start paying for space (free limit is 0.30GB), so
perhaps that will also guide the decisions?

Regards--
Tom

It's hard to tell from your list, it seems like you forgot ACL, i.e. whether
every project's committer list is the same or not. That's another good
reason to split projects up between dojo toolkit and other foundation
projects, but even within dojo toolkit we might want different projects to
have different lists of committers. Of course that's probably the
opposite of what the lawyers want.

I'm assuming that everything under thttps://github.com/dojo must have the
same list of committers?

In preparation for the Web Builder going live, I'd like to push the
Dojo Web Builder's code to a new project owned by the Dojo
(Organisation|Foundation) account at https://github.com/dojo

I think this is the most appropriate place for it to live externally.

Does anyone have any major objections or concerns?

I was aiming to announce the hosted version this Friday but
open-sourcing the code can be delayed if needed.

Tonight's meeting may be a good opportunity to discuss anything.

If it helps, this is basically what I envision for 2.0+ in terms of
distribution:

1) We assemble, from separate packages, what we would consider to be
Dojo Core and Dijit today;
2) We include some sort of directory where other packages should live
once installed;
3) We include *selected* projects that formerly lived in DojoX, and
distribute in that separate packages directory;
4) We create an archive (zip/gz) for download by anyone to use.

In essence, what we would consider the "official" distribution to be
as a pre-assembled set of packages, along with guidance on how to
install secondary packages people might use (for instance, if Charting
is not part of the core distribution, we'd include it in the separate
packages folder as an example of a pre-installed package). At this
point, the actual distribution is more of a
marketing/best-practice-as-recommended by the Dojo community, and less
a single entity; the idea being that if someone wanted to create their
own custom package, they could go to the source repositories and
manually assemble them using the package management tools Kris is
working on (and I've volunteered to create a web-friendly version of).

Does that make sense, and do you think that would satisfy the
requirements of companies creating their own custom distributions?

Regards,
Tom

I would agree that the tool doesn't belong within the normal release
distributions. I don't believe the majority of users will need to run
the Web Builder locally and don't think they should suffer an
relatively large increase in the size of the normal source
distribution to accommodate this application. However, I think it
would be nice for this to live in the official dojo foundation account
on github although I'm not overly fussed as long as the it's owned by
the foundation.

Maybe we could offer a create a kitchen-sink release, which included
the normal Dojo release plus a selection of tertiary tools, like the
Web Builder and other new utilities projects? Or have the official
dojo account on github offer a meta-repository which included
sub-modules for dojo source and other tools that people could clone to
get everything?