Yeah Egon has some great ideas and reading his code replies to my previous
posts is where I noticed this particular opportunity.
I agree that there are a lot of approaches, to dealing with unstructured
data.
My intent was more narrow and was intended to show how easy it is to
reimplement something akin to hasOwnProperty from the JS world in Go.
This is not to say that either approach mentioned by Egon is in anyway
deficient, just the particular use case of needing an object or value at
some arbitrary depth in unstructured data does not appear to be addressed
any where that I have been able to find.
I showed an example using permissions since that was the thing that was
buggered up when I started, but consider a different use case for a moment.
When you store a document in Mongo you tend to store the whole document,
these things do not lend themselves to relational structuring very easily
and can get heavy if you're not careful.
Again I'm not addressing the stupidity of the idea of trying to store
*.world+dog , but merely, what do you do when you are presented with a need
to extract something particular from an unstructured document containing
dog+world and at completely arbitrary depth? You can say things like "just
make an struct that addresses it" or "you shouldn't do it that way in the
first place!". But that's not helpful when what you have has been handed
to you and your job is just to make it work.
Use case...
There are custom forms created using a custom tool that runs clientside,
built on top of angular.
Any form, can contain any field those fields can and do contain subforms to
any arbitrary depth.
This allows for the creation of business forms that meet regulatory
compliance by stringing them together from other forms that also meet
compliance. In otherwords the person creating the forms does not want to
have to re-invent the wheel. To their mind they are just adding new pages
to existing forms.
These were persisted wholesale in node and now the "legacy" node app is
being replaced, we need to ensure that anything that node may have returned
previously is also returned by go.
In this particular case, the User has a custom reporting system that needs
to extract data from field z of subform y of form x, which itself is
embedded in form n (and n could be embedded in l etc).
What you have as far as information about the thing you need to return is a
document id generated by mongo during the last upsert, and a command coming
from the browser that looks like
[query: {collection: 'forms', _id: 100, path:'n.x.y.z' },...]
The server neither knows, nor cares about the type of data stored there,
it's job is to isolate requested field(s), extract the data from those
fields and pass it back to the caller assuming they had permission to read
it :).
This was handled in node with nothing more than(pseudocode follows)
document = db.fetchOne(query._id)
if(document.hasOwnProperty(query.path)){
res.write(JSON.stringify(document[query.path]))
}else{
res.send(404)
}
I can't fathom what that would need to look like in Go, using the
recommended or idiomatic approach, but replacing that lovely little tidbit
was a task, way up high on my todo list.
The replacement code now looks about the same
(again, psuedo code here)
collection.Find(bson.M{'_id': query.ID}).One(&document)
section := json.HasOwnProperty(query.path,&document)
if section !=nil {
fmt.Fprintf(w, "%s",json.Marshall(&document))
}else{
http.Error(404)
}
Obviously the "correct" answer here is to dedup and normalize the data ,
however the goal of the project is to unify disparate information systems
and to do so while having the smallest impact possible to existing systems
all the while maintaining current levels of data integrity. Easiest path
to maintaining data integrity is to leave legacy data laying where it sits
while gradually phasing in replacment front ends and ensuring new data ends
up stored in a more coherent fashion. Eventually the old data ends up in
long term storage anyways, but the retention period on some of these
records is 10 years before they go to archives.
I'm in a position where I didn't make the data that I'm expected to deal
with, but I have to ensure that every user of that data has at least the
same quality of experience they do now.
On the other hand I have no desire to take 1,000 lines of node and convert
it into 100k lines of Go just to cover every edge case.
My instincts which are informed by over a decade of coming in and cleaning
up after 1st year college freshman, H1Bs who couldn't spell SQL without
"My" in front of it and *shudder* PHP devs, tell me that things like I've
described tend to be the norm after any "super pumped, agile, web 2.0,
html11, kitchen sink" project reaches more than about 6 months of
development.
I have a feeling that as node and other "web server" tech becomes
supplanted by Go, things like this are going to come to light pretty
frequently, and my goal with this posting was to leave a trace for anyone
in the future bumping up against similar things. Hope it's helpful to
someone.
On Thursday, January 22, 2015 at 3:47:13 AM UTC-8, Egon wrote:On Thursday, 22 January 2015 13:31:16 UTC+2, Klaus Post wrote:
Hi!
Good call by Egon - you should go for that.
Actually, my recommended approach for that problem was
http://play.golang.org/p/4SqbLxr_lg. But for dealing random arbitrary
data is simplejson is better. For something inbetween, there are multiple
approaches, i.e. use map[string]interface{} directly, or properly marshal
into a nice type. Using a proper type simplifies code else-where while
keeping the problematic part in a single place.
+ Egon
Also it is trivial to implement what you need, see:
http://play.golang.org/p/5oeZxliMynOf course you would need to type assert the results you get, but that is
the deal when dealing with loosely structured data.
/Klaus
On Thursday, 22 January 2015 07:57:03 UTC+1, maxpow...@gmail.com wrote:Note to the group: This message is intended for future people who like
me are coming to golang after spending a good deal of time out in nodejs
land.
I've been a programmer for a lot of years, C/C++, Java, Node.js you name
it I've probably put it to good use at some point.
Lately I've found that I like Go alot and I really want to keep liking
Go, but having used node.js as my go to webdev language for the last couple
of years has spoiled me in the way I interact with loosely structured data.
This is most obvious when I start dealing with JSON being passed back
and forth across the wire.
I really do hate Javascript & callback hell and all the other reasons I
decided this current project would not be built with node, but I love
dealing with JSON in node because it's native and feels natural.
I saw that Go advertises strong support for JSON, and I'll admit that
the language features a very good parser.
However if you need data from an arbitrary field, it's just not able to
do it and messing with the idiomatic methods of doing it just looks ugly in
my opinion.
I spent a few days struggling with this, because I guess most folks who
use Go are used to dealing with predictable structured data so most of the
JSON examples don't really go into any great depth of how to deal with data
that doesn't fit into structures that can be built at compile time.
In my case actual data interaction is minimal, but when I need a field
it may be buried 10 layers deep in a JSON object, and I don't even get to
control whether or not that field is present.
I just need the language to get out of my way and let me have the data
at field x.y.z.a.n.w.t.f or tell me something along the lines of "Sorry
that field couldn't be found."
This is the one place where Javascript really shines, I can simply say
something like
if(someObject.hasOwnProperty('some.field')){
doSomethingWith(someObject)
}
And it just works!
As far as I can tell, go has no equivalent.
So I was faced with a few choices.
#1 Ditch Go and run back node.js (not going to happen even if I have to
hard code every possible data structure, lives will literally depend on
this code)
#2 Hard code structs for the expected types merely for the purpose of
checking for field values, (that's a LOT of code and I am a very lazy
person)
#3 Use one of the Javascript interpreters for Go (too much overhead just
for this, and leaning on JS could lead to very bad things)
#4 Take a close look at my previous node projects and see what features
were actually being used the most and try to replicate them in Go.
I opted for number 4.
The most common things I do with server side javascript
#1 Marshall Strings into JSON
#2 Check JSON for presence of fields
#3 Take logical actions based on data contained in said fields
#4 Ship JSON out somewhere else
Actual data manipulation is very, very rare in my past projects.
Usually once data has been JSONized somewhere, the primary purpose of said
data is to tell some controller somewhere to take some action.
Thus I did not include any way to actually manipulate JSON Objects.
The result is here,
https://play.golang.org/p/cRgh7j2TW2I've given you two functions.
The first one is json.Parse which takes a string and gives you a
map[string]interface{} which is the closest thing to an actual JSON object
that you are going to find in Go.
The second function is json.HasOwnProperty which takes a field label and
the json object created by Parse and will return to you either the value at
said label or nil
It does differ from Javascript here in that it will return the actual
value of a label rather than a boolean indicating whether it was present or
not.
I did that because asking if a field is present and then asking it to
retrieve that field are two steps that really should be just one, imho.
You can use the code like this...
First get a JSON string and parse it into a JSON object
const B = `{
"allowedtypes" :{
"medical": {
"sources":{
"myself": ["*"],
"mylocation":["r"],
"other": ["a"]
}
}
}
}`
var objB := json.Parse(B)
Next query and assign the field to a variable so you can take action on
it...
permissions := json.HasOwnProperty("allowedtypes.medical.sources.myself"
, objB)
fmt.Printf("Permissions for allowedtypes.medical.sources.myself
%+v\n", permissions)
That's it!
Thanks for taking the time to read this and I hope someone who stumbles
on this later can put it to good use.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.
