Looking for a flexible and efficient binary serialization format? Try our implementation of Google's Protocol Buffers!
Looking for a flexible and efficient binary serialization format? Try our implementation of Google's Protocol Buffers!
https://blog.grijjy.com/2017/04/25/binary-serialization-with-google-protocol-buffers/
https://blog.grijjy.com/2017/04/25/binary-serialization-with-google-protocol-buffers/
Excellent blog post... 😎
ReplyDeletereally cooooooool
ReplyDeleteThis is an excellent example of the use of attributes!
ReplyDeleteIs there any particular reason why this same approach couldn't be used to serialize the data contained within classes in addition to records?
David Schwartz A similar approach could be used to serialize objects. But I wanted to keep things simple (from a user's perspective). Once you add support for serializing objects you run into issues like how to construct new objects, who owns those objects, and how to handle class hierarchies. We addressed those issues in our blog post on JSON/BSON serialization, but it makes it less easy to use and increases the chances of bugs and memory leaks. Or maybe I was just too lazy to implement class support😉...
ReplyDeleteWow! Cool!!! Thank you very much Erik van Bilsen
ReplyDeleteIMHO attributes at object level are not the way to go with protocol buffers, unless you make a "code-first" approach and you can generate the .proto file.
ReplyDeletePersonally, I used to work with protobuf, but prefer now the CBOR format http://cbor.io which is easier than protobuf, and designed for IoT, without the need of designing a schema and have both ends synchronized. For schema-less values serialization between Delphi ends, I use our binary serialization for records and dynamic arrays https://goo.gl/1By7Gr
ReplyDeleteIn all cases, business objects should not benefit from being directly serialized, as David Schwartz ask for: it is an identified good practice to define Data Transfer Objects (DTOs) for transmission, not directly leak your business model. Records are perfect for defining DTO value objects.
This seems to me to be one the most frequent mistakes made by Delphi developers, leaking business model by serialising these objects. Arnaud talks a lot of sense on this topic.
ReplyDeleteWell done, your blog posts are of high quality and your team is providing high quality staff to the Delphi community constantly, kudos! PS, ProtocolBuf has a wide variety of use cases, but when deciding the format for persisting data to disk, consider how the *human-readability* of the format will affect you when debugging.
ReplyDeleteDavid Heffernan Main problem with any serialization is usually tight coupling between class and serialization framework. And that is a problem that exists everywhere.
ReplyDeleteIf you have to slap serialization attributes to a class you are doing it wrong. Same goes to adding serialization framework to your business objects uses (import) list or inheriting from common base class that know how to perform specific serialization. Regardless whether that class (record) or whatever represents business object or not.
From my experience with cross-platform serialization to multiple formats, the only win-win situation is using frameworks that support complete custom configuration from the outside without messing with you business objects and even without messing with DTOs. Actually, in many cases I can completely skip having any DTOs and save myself from writing a whole a lot of boilerplate code.
Hi Erik van Bilsen Have you considered adding this (and some other libraries) to GetIt?
ReplyDeleteHi Erik van Bilsen Have you considered adding your repositories to Delphinus?(PackageManager for Delphi XE-Tokyo)
ReplyDeletegithub.com - Memnarch/Delphinus
Dalija Prasnikar So when you say:
ReplyDelete"Actually, in many cases I can completely skip having any DTOs and save myself from writing a whole a lot of boilerplate code."
you mean that you somewhere else define the serialization of your business objects without actually modifying them and thus don't need the extra DTO overhead?
That would mean an approach like the original protocol buffer implementation, right? (define some template somewhere and generate code from that).
Erik van Bilsen I think your implementation misses one of the purposes of Protocol Buffers. Any existing .proto file that exists is basically useless when using this implementation because you have to reimplement a record type that matches the definition in the .proto file plus every time it might change you have to change your code. On the other side if you want to exchange data with any other application you have to write the .proto file anyway which has to match your record definition. In the end you are doing everything twice plus risking things not matching.
Stefan Glienke Not exactly...
ReplyDeleteI am usually dealing with quite simple business objects. They are almost always serializable as-is.
If I want to decouple business logic from actual serialization needs and frameworks, there are two paths - creating separate DTO objects that will be coupled with serialization framework or use serialization framework that can operate on my business objects from the outside.
With former there is extra code to be written (or generated) to create DTOs and map business objects with them. Using later (with good and flexible serialization frameworks) I can perform serialization with minimal configuration changes - depending on how much actual properties differ in names and/or types.
Also if needed I can just go from such "direct" serialization to more complex serialization implementation without touching single line of code regarding business objects.
What is usually seen in practice, when dealing with such simple needs, is that people tightly couple serialization with their business objects and when they need to change serialization part later on whole hell breaks lose.
David Millington Alexander Benikowski We haven't considered exposing our code through GetIt or other package managers yet. We aren't library developers, but just enthusiasts that share some of the code we use in our own business. But it is definitely something we will consider!
ReplyDeleteDalija Prasnikar Stefan Glienke I agree that you should separate your DTOs from your business logic. With protocol buffers and our implementation you can just do that: all DTOs are in a separate set of attributed records that can be produced/consumed by your business objects.
ReplyDeleteUsing records instead of objects for DTOs reduces the "urge" to mix logic and DTOs together.
Stefan Glienke Regarding the use of .proto files: if this is a concern for you, then there are other implementations out there that do compile a .proto file to Delphi source code. As said in the post, our implementation is an "Alternative implementation" that doesn't use this approach. If you control both the BAAS and the client apps, then this is not a concern, and I think that our solution is much easier to use in that case.
ReplyDeleteIf you do need to consume 3rd party .proto files, then you are right, and you would have to update your source code every time the 3rd party makes an update. But you probably need to make changes in that case anyway because of changes in data. You run into the same issue when a web service uses JSON for example, and they decide to change the specification.
Erik van Bilsen "there are other implementations out there that do compile a .proto file to Delphi source code" Are there? The few I found some while ago mostly looked like proof of concepts and generated classes afaik.
ReplyDelete"If you control both the BAAS and the client apps" Typical Delphi developer disease tbh (without offense). Why care about interoperability when both ends are written in Delphi. Guess why Delphi is lacking so many implementations of commonly accepted "standards" - because most devs don't care. True on protocol level they are supposed to be compatible but who can guarantee that if you cannot share a proto definition with another language and then exchange data to check that.
Looks like there are a lot of opinions on how to do serialization. There are so many tools and formats out there. In general (and this applies to most things we do), you want to pick the best tool for the job. Sometimes this means using a schema-less model. Other times a structured model is better. Sometimes human-readable is better, sometimes binary. Sometimes speed is a priority. Sometimes compactness is. These are decision you'll have to make. We just provided a tool that may or may not fit your decision.
ReplyDeleteStefan Glienke I didn't look into detail into the other solutions out there. For our business, we didn't want to use .proto files and generated source code, and those libraries did't fit that. So I cannot judge the quality of those other libraries.
ReplyDeleteDalija Prasnikar "They are almost always serializable as-is."
ReplyDeleteThis means read and write all public (or published) properties? Possibly with ignoring read only ones as they cannot be deserialized anyway and are probably calculated ones?
"or use serialization framework that can operate on my business objects from the outside."
This is the one I usually prefer with the exception that I usually allow metadata (attributes) for that. Yes, this means modifying the code of the classes which you might want to avoid. The other option would be to provide the metadata somehow different. What would be your approach?
Stefan Glienke Basically, yes - in Delphi that means published properties, in Java non transient ones, Swift does not have such language built in option. Also it is important that framework can be configured to ignore specific properties and that it can map names and types, as well as use converters.
ReplyDeleteI avoid attributes like a plague because I often serialize same classes to different formats and sometimes using different frameworks. Attributes are absolutely useless for me.