Does anyone know where I can find a FAST JSON "serializer" ?
Does anyone know where I can find a FAST JSON "serializer" ?
I need to store my Object into persistent files. I've so far used XSuperObject for this purpose. XSuperObject allows me to save my "Delphi" object into a JSon string simple as typing MyObject.AsJson... the only drawback is that serialization is slow has hell as soon TList item count grows...
I'm bascially using such structure
TWareHouseEntry= class(TPersistent)
public
Order: string;
Reference: string;
Palette: string;
Details: TList;
end;
TCartonDetail = class(TPersistent)
public
QRCode:String;
Quantity: double;
User: string;
CreationDate:TDatetime;
end;
I'm saving structure this each time my user scan a carton.
Under Android, it takes me 6 000ms to serialize an object having 1200 items...
This is tooo much time. Tooks about 10ms in Java.... Any idea, tips ?
I need to store my Object into persistent files. I've so far used XSuperObject for this purpose. XSuperObject allows me to save my "Delphi" object into a JSon string simple as typing MyObject.AsJson... the only drawback is that serialization is slow has hell as soon TList
I'm bascially using such structure
TWareHouseEntry= class(TPersistent)
public
Order: string;
Reference: string;
Palette: string;
Details: TList
end;
TCartonDetail = class(TPersistent)
public
QRCode:String;
Quantity: double;
User: string;
CreationDate:TDatetime;
end;
I'm saving structure this each time my user scan a carton.
Under Android, it takes me 6 000ms to serialize an object having 1200 items...
This is tooo much time. Tooks about 10ms in Java.... Any idea, tips ?
Not the main problem, but if performance matters I'd imagine that the overhead with using a class would be worth removing. These items look more like values to me, i.e. records.
ReplyDeleteI guess that using TRecord will speed up a "little" bit but will not change this HUGE 6 000ms to serialize my object.
ReplyDelete/sub
ReplyDeleteYou could try github.com - ahausladen/JsonDataObjects
ReplyDeleteMaybe write your serializer in code rather than rely on RTTI?
ReplyDeleteThere is a JSON writer in DWScript's dwsJSON unit, it is meant to be invoked manually (by code, not RTTI), but should be able to handle your case without breaking a sweat.
Stéphane Wierzbicki No, I did not say that it would. It's a broader issue. Why have all those objects scattered about carrying their extra payload?
ReplyDeleteAnyway, anytime you rely on RTTI things are not going to be fast.
Have you asked the XSuperObject guys? Maybe there's a simple fix in their code that can solve the issue -something like a BeginUpdate or allocating a buffer instead of growing it one by one.
ReplyDeleteDavid Heffernan I'm using TList because it is really easy to handle items... I can for sure :
ReplyDelete- change my TCartonDetail class into a Record type.
- change my TList to Array of TCartonDetail
Problem is only time... I'm almost done with my project and this will required a lot of time to rewrite all available TList searching, deleting, adding and filtering methods(to name the most important).
I expect things to be less optimized when using RTTI but not on a such factor...
Martin Wienold I'm aware of this library. I still have to write my serializer.
Eric Grange do you have a simple example for this ? Is DWScript's dwsJSON cross platform ?
Uli Gerhardt Nope, I'll post an question on his github repo. I'll get for sure an answer but I don't think that speed issue will be largely improved
Maybe their is another way to persist my object ? Can I save it to a stream or something similar ?
ReplyDeleteStéphane Wierzbicki The list is fine as a TList, you can just change the items to a record and then you still have an array internally. But it's probably not worth doing so if it causes a headache.
ReplyDeleteYou don't need to serialize using RTTI. Just write code that emits each field manually.
Stéphane Wierzbicki there are several unit tests, but the usage is quite simple, you Begin/EndObject or Array, use WriteString to write a string, etc.
ReplyDeleteAbout cross-platform, it passes units tests in both XE 32bits and XE2 64bits, no idea beyond those.
David Heffernan Yep.
Using RTTI for serializing is a recurring Bad Idea IMHO, because as soon as you have persisted something with RTTI, you will have essentially tied your internal field or property names as well as data types forever... at the risk of breaking persistence or needing to add another layer of inefficiency to alias things.
Also as the use in DFM components readily illustrates, it can get very messy in terms of forward and backward compatibility because you cannot (easily) perform versioning of format detection.
If these are the entire structures (or mostly), being that they're quite small, I'd write a serialize/deserialize routine that doesn't rely on RTTI and is designed to manage memory optimally for large numbers of known objects in a way that preallocates space for a bunch of them at a time.
ReplyDeleteIf I were to guess at the problem however, the first thing I'd look at is whether the underlying code is either allocating memory one object at a time and being very inefficient at it, or is allocating them in fixed-sized chunks (like arrays) and doesn't know how many to allocate initially, so it's growing dynamically and having to reallocate-and-copy the existing bag of objects several times as it grows.
Also, I'm curious if there's much of a performance difference between streaming OUT vs. streaming back IN?
RTTI (and ORM) are as overestimated as danish beer. Stéphane Wierzbicki i agree with Eric. I did not use DWScripts units for this but rather RTC's (RealThinClient). I write a loop, pushing info into the structure meant for emitting and after that call ".ToJSON". Speed and efficiency are cool. What do i gain using RTTI/ORM? A little bit of safety against typos, maybe a little less stress on memory. That's all. What do i loose? Flexibility, maintainability and performance. Also, look at FastReports. If you inherit from TfrxReport then you cannot load a report anymore. Because there's a base of RTTI serialization that uses class names. 2006 you go - ok, let's stora and load reports, great! 2011 you go - wow, let's inherit and then i'll get rid of all the global sh*t around reports... Oooops!! Unfortunately RTC costs, but i suspect the unit in DWScript works similarly. I suspect that the extra allocation, copy and free of memory is almost nothing compared to RTTI serialization.
ReplyDeleteDany Marmur as Dane I can agree on the beer part :). I would try Andreas Hausladens as mentioned earlier, or the free and opensourced TclJsonSerializer clevercomponents.com - JSON object serializer for Delphi
ReplyDeleteContact me if you Need help with DJSON
ReplyDeleteSteffen Nyeland yes, I'm 45 minutes away but i do not have to travel to buy dk beer though i do avoid it... lol.
ReplyDeleteOmg g+ sucks. Why are we here? Why did we outsource nntp to Google?
ReplyDeleteDJSON is anche open source project, It is a multiengine Deep serializer, There are There engines, One of them is based on Andreas Hausladens JSON parser. It is very configurable. You Can eventuali create and register custom serualizers if needed (rare) or if you Need Extreme Speed.
ReplyDeleteSorry I'm typing fronte my phone
ReplyDeleteThe JSON parser that comes with mORMot framework from Synopse is by far the fastest I've used - in terms of serializing to/from json.
ReplyDeleteMaurizio Del Magno where can we find DJSON ?
ReplyDeleteJason Smart for sure but it is not fully cross platforms. JSonRegisterClass for example is only available is not available for nexgen compiler
Stéphane Wierzbicki github.com - mauriziodm/DJSON
ReplyDeleteJason Smart I recently saw some library on github, they claim to target to outperform mORMot without going lo-level. I do not know if they succeeded, but they set it as one of their talking points :-)
ReplyDeleteThe mORMot serializer, specialized for the ORM, is perhaps the fastest, but it does not work under Delphi for Android (it works with FPC for Android, though..).
ReplyDeleteThe Delphi RTL's JSON parser is one of the slowest, for sure... and difficult to work with (it creates class instances for each node).
See http://blog.synopse.info/post/2015/02/16/Benchmarking-JsonDataObjects-JSON-parser
For your exact purpose, with cross-platform in mind, Andrea's library seems the way to go: it is very optimized. See github.com - ahausladen/JsonDataObjects
Thanks Jacek, DJSON works well on Objects, Collections of objects (TStrings, TObjectList, TList, TDictionary, Arrays etc), Images. Is compatible with interfaces (ex. an object with a property of some interface type) or collection of interfaces (ex: TList).
ReplyDeleteI do not know if it's fast enough for you but you can try it and see.
ReplyDeleteIf any of you try it let me know what you think and if you need help please contact me.
David Heffernan if you cache rtti it could be fast. mORMot proves that, with a sax approach. And faster than using self contained dom values since you don't need to store and lookup for the field names. On our benchmarks, record/array serialization via rtti is a clear winner, also in terms of memory consumption.
ReplyDeleteEric Grange a good practice is to define DTO classes or records for the info transmitted by representation. Compile time check of field names and types is a killer feature...
ReplyDeleteThe default engine used in DJSON work on stream directly, no DOM. It consumes 4 times less memory than other 2 engines.
ReplyDelete/sub
ReplyDeleteA. Bouchez i wonder if RTTI with anonymous functions can provide for record (de)serialized implemented as Forth-like threaded code.
ReplyDeleteI recently saw an article about CPU emulators impl strategies, and depening on host CPU and Cpp compilers and settings different approaches were becoming winners and loosers.... Human, and compilers optimizations gave quite an interference....
A. Bouchez that's what i dislike about SuperObject. It is so lazy-easy to use in DOM way, but DTOs are absolutely non-extensible there, unless you provide a complete self-made marshaller (which would use DOM approach)...
ReplyDelete/sub
ReplyDelete