Home > DataSnap, Delphi, mORMot, REST > DataSnap analysis based on Speed & Stability tests

DataSnap analysis based on Speed & Stability tests

Image

It’s not difficult to read and listen about the wonders of Embarcadero DataSnap technology around the world. I attended the Delphi Conference 2011 and 2012 in Sao Paulo, Brazil, after the release of versions of Delphi XE2 and XE3, and the words “performance” and “stability” are not spared when the subject is DataSnap. Apparently a perfect solution for companies that have invested their lives in Delphi and now need to reinvent themselves and offer web and mobile services. But does DataSnap really work on critical conditions?

I found some references from other people talking about it on StackOverflow:

The company I work for stands as one of the five largest in Brazil when it comes to software for supermarkets. The main product is a considerably large ERP developed in Delphi (for over ten years). Recently we initiated a study to evaluate technologies that would allow us to migrate from client / server to n-tier application model. The main need was to be able to use the same business logic implemented on a centralized location on different applications.

The most obvious option is the DataSnap, which works very similarly to what is found in legacy applications. It can drag components, using dbExpress, for example.
We tested the DataSnap technology in order to know what level of performance and stability it provides and to verify if it really meets our requirements. Our main requirement was the server’s ability to manage many simultaneous connections, since the application is big and used by many users.

Objective

Our objective was to test the DataSnap REST API and to answer some questions:

  • How does it behave in an environment with many concurrent connections?
  • What is the performance in a critical condition?
  • Is it stable in critical condition?

Methodology

The tests are based on a lightweight REST method without any processing or memory allocation, returning only the text “Hello World”. The DataSnap server was created using the Delphi XE3 wizard and implemented the HelloWorld method. The testing was performed on all types of lifecycles (Invokation, Server and Session) and also with VCL and Console application. However, as the results were identical, the presented data is only based on the results obtained using the console version with “Server” lifecycle only.

We decided to create a few servers based on other technologies, not necessarily similar or with the same purpose, just to have a basis for comparison. Servers have been created using the following frameworks:

  • mORMot (Delphi)
  • ASP.NET WCF
  • Jersey/Grizzly (Java)
  • Node.JS (JavaScript)

As you can see, those are totally different frameworks, some with different purposes, in different languages​​. I want to make clear that we do not establish a comparison between the performance of frameworks, because we do not dominate these technologies. We will use them simply to get a sense of how they behave compared to DataSnap.
The hardware environment used in the tests:

We used the Jmeter tool for testing. The Jmeter is a great tool developed in Java specialized for server performance testing. Before anyone questions the use of this software, I obtained similar results with a software developed by myself in Delphi. The only problem we had was with faster servers, if we kept the focus on the results screen the tests were affected because the screen refresh rate delays the sending of HTTP packets to the server, but we took care for this not to occur during our tests.

Tests without concurrency

The first tests that were executed without concurrency in order to visualize the behavior of servers in a less critical situation. We did two tests, one with 100.000 and another with 1 million requests. This allowed us to evaluate whether the server behavior changes with increasing requests.

Performance testing (Requests per second) obtained the same result in both tests (100 thousand and 1 million requests):

We observe in this test that the other servers are practically 13 times faster than Datasnap, the difference is huge. It is also possible to note a bottleneck (since all other servers have obtained the same result), possibly in the amount of requests that the client machine can send using a single thread.

The tests of memory consumption had differences in the two tests (100 thousand and 1 million requests), so let’s look at both:


NOTE: The memory consumption of Java server needs to be evaluated carefully because it is the measurement of the java virtual machine.

In this graph we see two different behaviors. MORMot, WCF and Node.js servers use the same amount of memory to perform on both tests. The increase of requests made both the Java and DataSnap server raise their memory consumption significantly to get us worried enough.

Why is the server consuming so much memory only answering HelloWorld requests? Before seeing the results I would bet that none of the servers would use as much memory just to answer this simple request.

In the case of DataSnap, we note that even if the tests continue, without any pause to the server, executing 10 million requests for example, the memory consumption follows the trend of rise (to infinity and beyond). We consume up to 1GB of memory, only an answering HelloWord, how to explain this?

Another interesting detail is that to stop the tests, gradually and very slowly (taking hours in some cases), the memory consumption of the DataSnap server backs up to the level it was when it opens. Leaves the question, if the server works 24/7?

Tests with concurrency

These tests aim to stress the server and verify the maximum efficiency in the management of his requests and HTTP packets. We did two tests, one with 50 and another with 100 threads. Consequently 50 and 100 simultaneous requests, respectively. Each thread should send 100.000 requests to the server, totaling 5 and 10 million requests in the respective tests.
To my surprise and disappointment, it was not possible to perform this test with the DataSnap server. I tried several times and the server simply crash.

Evaluate the results of the other servers only.


Node.js got the worst result, but do not be fooled, because he got this result using only a single core. It is a limitation of the architecture of that framework, but I could, for example, have four instances initialized it and made ​​a load balancing, which would bring me a performance higher than that obtained in this test. Unfortunately I did not had time to implement this solution.

The ASP.NET WCF and Java servers had very good results, with a certain advantage for WCF. Both had the same results with 50 and 100 threads, which indicates that this is their limit in this environment and hardware. During testing we noticed these servers use almost all hardware resources, while the client application worked at 60%, which indicates the limit on the server.

The mORMot server surprised me, firstly, the excellent results in the test with 100 threads, and then by the way he got this result. In contrast to what occurred in the Java and WCF server, we noted a use of only 50% of the resources of the server machine, while the client machine was at 100% of processor usage. This indicates that there was a bottleneck in the client application, and if there were a better client machine or more machines (in the case of a real application) the server could be even faster.

Considering the performance and especially efficiency, mORMot gave better results using these implementations, but in general, all of them had very good results, except the DataSnap, who did not complete the test.


The results of memory consumption remained similar to previous tests to mORMot, WCF and Node.Js servers. The Java server followed the trend already observed in other tests with a memory consumption proportional to the number of requests, at least concerning.

Conclusion

We conclude that the DataSnap has a serious stability problem when placed in an environment with concurrent connections. The server stops working even when we put only two concurrent connections under stress test, which is very worrying.

Watching the performance, we observed that the DataSnap is much slower than the other solutions tested.

Overall, I can say that all the other frameworks have the performance and stability that is necessary in a large application. In particular, the mORMot surprised me with a performance truly outstanding. The guys at Synopse are doing a fantastic job.

Why is the DataSnap so slow? And why it crashes?

Difficult to specify the cause of these problems without knowing all the DataSnap structure and without having participated in its development, but I have a theory.
Apparently, the structure of the HTTP communication framework makes use of the Indy components, which creates a thread for each HTTP request. This behavior causes the server to create and destroy dozens of threads per second, which causes a huge overhead. This overhead is certainly the cause of poor performance of the framework. Hard to say which is the cause of the problem that crashes the server but I believe it has a connection.
With the help of ProcessExplorer we view this overhead. This graph shows the threads that have been created recently (green) and those that are being destroyed (red).
Status of the first test:

Status of the second test with 100 threads, captured before the server crash:


In this image we can see that the server was running 47 threads and the list of threads being created and destroyed is huge.
Another important detail is that practically all consumption of CPU in DataSnap server is overhead in creating and destroying threads.

Final Thoughts

I’ll end with a quote from the text of Marco Cantù (Currently, Delphi Product Manager) in the book Delphi 2010 Handbook.

I think that if you want to build a very large REST application architecture you should roll out your own technology or use one of these prototypical architec-tures. For a small to medium size effort, on the other hand, you can probably benefit from the native DataSnap support.

All the source code of the servers, as well as the binaries and the Jmeter test plans are available for download at https://github.com/RobertoSchneiders/DataSnap_REST_Test.
I personally spoke with some staff of Embarcadero in São Paulo, on 10/23/2012 (Delphi Conference) about this problem. We informed Embarcadero of Brazil about the partial results of our tests on 10/30/2012 and sent the final results of the tests presented here on 11/14/2012 by e-mail. So far, nobody of Embarcadero offered support or any solution to the problem we are experiencing.

To all of the people involved in this testing period, I leave my thanks. I would like to thank Eurides Baptistella (Systems Analyst, Sysmo Sistemas) that created the Java server and helped in the preparation and execution of tests. To Adriano Baptistella (Systems Analyst, Sysmo Sistemas) for creating ASP.NET server. To Dean Michael Berris (Software Engineer, Google) and Everton Antunes de Oliveira (Development Research Analyst, Pixeon Medical Systems) per help with creating a c++ server (cpp-netlib), who unfortunately could not finish in time for use in testing. To Cesar Arnhold (Third Engineer Officer, Maersk Supply Service) and Mateus Artur Schneiders (Systems Analyst, QuickSoft) for reviewing this text.

If you liked this feel free to leave your comment.

This post continues on DataSnap analysis based on Speed & Stability tests – Part 2

About these ads
  1. November 23, 2012 at 10:27 am | #1

    You didn’t tested Delphi on Rails ! (http://code.google.com/p/delphionrails)

    • November 23, 2012 at 11:30 am | #2

      I’m sorry. There are many frameworks out there and as I said my goal was not to make a comparison between the REST frameworks, in which case I would have to compare dozens of frameworks. I just tried to have some basis for comparison.

      Anyway, I’ll take a look at this framework.

      • zedalaye
        November 28, 2012 at 10:46 am | #3

        I pushed a branch with a DoR server on Github.

  2. Francisco Ruiz
    November 23, 2012 at 10:39 am | #4

    Hi,

    As I have worked in a similar application and tested the problems from datasnap. Can you send me your source for the sample tests in Delphi? Or publish it?

    There are some different approaches to web serving in Delphi apart from those exponed. Mormot is one of the best solutions, but for me is excesively complex to undestand for a newby. The product is incredibly sucessfull and has a design way ahead of Datasnap aproach.

    Aside, I’m using Delphi for web/jsonapi with sucess for several years. Not without tears on the begin but at the end solutions could be stable and with a fantastic performance.

    • November 23, 2012 at 11:28 am | #5

      The code I used was a simple loop (for) calling a HTTP request with THttpRio. But I can send the sources, of course.

      • Bruce McGee
        November 26, 2012 at 9:41 am | #6

        Did you mean TIdHTTP?

        It might be worth including this test in the GIT repository.

      • November 26, 2012 at 10:15 am | #7

        Sorry, I meant TIdHTTP!

        I do not put it in the repository because I do not have a client in delphi prepared for use with these servers. I have a prototype with quite messy code.

        To do the test using a client delphi is only loop and call the method “idHTTP1.Get(‘http://127.0.0.1:777/datasnap/rest/TServerMethods1/HelloWorld’);”

  3. LDS
    November 23, 2012 at 11:23 am | #8

    It’s very interesting the implementation doesn’t think about recycling threads from a pool, instead of creating and destroying them. IIRC Indy can do it. Anyway Indy is another issue in Delphi, Embarcadero should take control of the project and pay development for it, or switch to another library. It’s very difficult to work with a basic library like TCP/IP support which is developed outside the project and often breaks compatibility forbidding updates of the maybe buggy one delivered with the product an upon which other libraries are developed and you can’t recompile them.
    Anyway I would profile Datasnap with AQTime or the like, and check were it does really spend so much time and allocates memory. Something maybe someone at Embarcadero should also perform…

    • November 23, 2012 at 5:37 pm | #9

      DataSnap is built on top of Indy low level TCP/IP and HTTP components. DataSnap also supports pooling of connections just ask dbExpress supports pooling – DataSnap is built on top of the dbExpress lower level framework.

      http://docwiki.embarcadero.com/Libraries/XE3/en/Datasnap.DataBkr.RegisterPooled

      Also – on the DSTCPServerTransport – you can work with the MaxThreads and PoolSize properties.

      http://docwiki.embarcadero.com/Libraries/XE3/en/Datasnap.DSTCPServerTransport.TDSTCPServerTransport.PoolSize

      http://docwiki.embarcadero.com/Libraries/XE3/en/Datasnap.DSTCPServerTransport.TDSTCPServerTransport.MaxThreads

      • LDS
        November 23, 2012 at 7:49 pm | #10

        IMHO one of the Datasnap main design flaws is exactly to be built on top of dbExpress and not the other way round. Remoting is not a specialization of DB calls, it’s exactly the opposite.
        Anyway AFAIK DSTCPServerTransport is the socket base transport, while all the tests here are HTTP (REST) based.

      • November 23, 2012 at 8:26 pm | #11

        David, Embarcadero should be less political, assume the problems and seek to solve. Clearly DataSnap has structural problems that keep you from being the best.

        Why in official events Embarcadero (Delphi Conference SP 2001/2012) lectures always have to use the “Wizard” for creating servers, never focused on scalability, stability, high performance? The problems reported are not here now, because you do not make a post about high performance Datasnap and show that he is the best? Show that the DataSnap can respond to 100,000 requests using 50 threads in a reasonable time without crash!!

        We must stop the DataSnap makeup … David, make like Marco, assume that we have problems and we focus on a solution …

        “It is true that DataSnap REST lacks the optimization (and at times the features) of other libraries out of the box, [...] and we at Embarcadero should make much easier to tune, configure, and optimize [...].”

  4. November 23, 2012 at 2:09 pm | #12

    Some of the issues you noticed and reported here are due to default settings or extra features that are part of DataSnap. For example, each request creates a new session, which expires by default after 20 minutes. And while I often use the IdHTTPserver myself, you might get better threading management deploying your application as an IIS library.

    It is true that DataSnap REST lacks the optimization (and at times the features) of other libraries out of the box, but it is a solid foundation that can be tuned (and we at Embarcadero should make much easier to tune, configure, and optimize). In the coming days I’ll prepare a more detailed response on my blog, but I fully accept the criticism that things should work better straight out of the box and configuration parameters should be more direct.

    • November 23, 2012 at 2:54 pm | #13

      Thank you for caring Marco. In fact, it is the first time that an official from embarcadero is really trying to help me. It is also the first time we put the problem in a blog ;)

      I hope we can find a solution to these problems, it would be good for everyone.

      As I said to you by email. The most worrying problem for us is when the application crashes. The high memory consumption may have a way around. The problem of performance is the least critical of all, but also is important.

      I do not know if the application behaves better using IIS. I did some tests and the results were not good. But we can evaluate more carefully.

    • Kent Morwath
      November 25, 2012 at 11:30 am | #14

      It’s a bit strange for a tool aiming to be a xplat solution requiring IIS to get decent performance ;) Sure, it you rely on someone else who knows how to write server code maybe it’s better… but what about using Apache (or something alike)? If Embarcadero can’t code a good application server, it’s better to rely on already existing tools.
      Moreover if you’re bound to Windows why not implement some set of component to take advantage ot Windows HTTP APIs directly, when you don’t need a xplat approach?
      Delphi should decide it it wants to be a tool for small business only, or if it wants to aim to the real *enterprise* market seriously. Otherwise, please rename Delphi Enterprise to Delphi Small Business – just to make people sure about what they’re buying.

      • Arioch
        November 26, 2012 at 4:42 am | #15

        Apache or IIS – those all are heavy-weight solutions. For the kind of server described here (very light request working over, lots of requests) the server should be like forward-proxy, those like Nginx, Lighttpd, ErlyWeb and such (there was also some very lite server from some grand like Amazon or E-bay bu don’t remember). This and ancient yet reliable FastCGI :-)

      • November 26, 2012 at 3:09 pm | #16

        Kent – DataSnap server technology is only running on Windows using Delphi or C++ for now. Linux is on the roadmap and would be a good second platform for DataSnap servers. We do support multi-platform DataSnap client technologies.

        I miss you Kent. Why don’t you send me an email sometime?(davidi@embarcadero.com). We can have a conversation about this and other things sometime.

  5. November 23, 2012 at 2:17 pm | #17

    Give it a thread pool. Before starting the server in the .dpr code:
    LServer.Scheduler := TIdSchedulerOfThreadPool.Create(LServer);

    This is XE2, I don’t have XE3 yet, but it’s probably the same.

    • November 23, 2012 at 2:45 pm | #18

      Thanks Ondrej Kelle!

      I’ll research more about it, here it is something I did not know. Probably the embarcadero documentation does not mention this.

      I did a quick test here and the server behavior has not changed much. He did not crash, but stopped responding to requests. This happened to me in other tests as well.

      In debug mode an exception is shown several times with the following text “Maximum number of concurrent connections exceeded. Please try again later!”

  6. November 23, 2012 at 2:30 pm | #22

    For an ISAPI DLL, put IsapiThreadPool in your uses clause.

  7. Francisco Ruiz
    November 23, 2012 at 2:44 pm | #23

    @tonderj, depending the delphi version + the IIS version this will corrupt the isapi at runtime. Do it whith care specially in modern versions of IIS.

    • November 23, 2012 at 2:47 pm | #24

      Francisco, that’s interesting, do you have any details on that? Is there a QC report? Thanks!

      • Francisco Ruiz
        November 28, 2012 at 9:24 am | #25

        Sorry tondrej I forgot this post.

        I’m not sure if IsapiThreadPool is needed in IIS 7.0 and higher because the server itself does a fantastic pooling of isapi apps and I don’t see any advantage in performance as previous ones. As you know Isapi was deprecated for some years in the Delphi roadmap. For this reason I was unable to put any QC as it was deprecated.

        I din’t try those tests with XE3.

        I was getting a big problem when some SEO robots maped my servers. For thas reason I started some stress test using a freeware tool from realthinclient.com, for me the target was support an explosive requests: 10.000 open connection x 100 requests using 64 threads all responded correctly.

        My actual configuration “if I remember it correctly”:

        In the isapi:

        1st. Drop DBXPooling in SQLConnections (I think this was repaired by Embarcadero)

        2nd. Drop isapithreadpool (as no advantage in performance)

        3rd. change maxconnections exactly here:
        Application.Initialize;
        Application.MaxConnections := 2000;
        Application.WebModuleClass := WebModuleClass;
        Application.Run;

        In that time MaxConnections internally was 8 and wasn’t docummented!

        In the IIS application config:
        1st.Pipeline mode to integrated
        2nd. queue size 5000 <<very important to tune this
        3rd. .net framework "none"

        I hope it helps. Thanks.

  8. November 23, 2012 at 3:53 pm | #26

    Outstanding review.. simply outstanding!

  9. November 23, 2012 at 4:06 pm | #27

    Wow! Very nice post!!!

    DataSnap is using Indy. Indy’s model of handling connections is: one thread per connection. This is bad or at least resource consuming and reduces chances for scaleability. IIRC windows x86 can handle no more than 2000 threads i.e. 2000 simultaneous connections (when not using Indy’s pooling) especially if the clients request a time consuming operation from the DS Server.
    Win x64 can create more threads but this hits performance no mather how powerfull your server is.
    Anyway – this is bad design when dealing with hundreds of client connections (TCP / DataSnap etc.)
    My personal opinion: a scaleable solution must use something like IO Completion Ports (IOCP). Yes, it’s hard to implement, even harder to change DataSnap to use it . But this way a casual server can handle 15 000 – 20 000 or more connections at a time!
    Also DataSnap is considered to be Enterprise solution. Embarcadero should understand that the “Enterprise” term is connected to Big, Large, Many, High volume, FailOver etc. The business is not only looking for features, it’s looking for scaleability and availability of their investment!

    EMBT should rearchitecture DataSnap if they want it to be real Enterprise solution. For now it may be a solution…but for small & middle sized companies rather than the Enterprise…

    To Marco Cantu: I’m surprised to see such kind of detailed test on your customer’s site. Instead on Embarcadero’s website I see only how featurefull DS is: DS is wonderful, DS can do this, DS can do that, DS is tarrific! Isn’t this frustrating to you? For Me – IT IS. EMBT should do tests, comparisons, studies etc. and publish them on it’s official website. This would be somekind of a proof for customers that DS is good and satisfy business needs. Performance is the *LAST* thing EMBT shouldn’t take care of, don’t you think?

    • November 24, 2012 at 6:39 am | #28
    • November 24, 2012 at 6:59 am | #29

      WCF and mORMot uses IOCP for serving JSON content over HTTP, via the native kernel-level http.sys server available since XP 2. This is one of the reasons of their stability and performance. mORMot is lighter and has a KISS and modular design, so seems to scale better than WCF, and use less resources.

      “Native” server applications using Delphi can rock, and beat “managed” versions, whatever marketing and dev power/money is injected (like WCF).
      But to have a fast solution, you need something in which speed was part of the design, from the ground up. This is the case of mORMot, but clearly not the case of DataSnap (built on DBExpress and Indy?).

      I work daily in a cross-platform project using WCF, and when compared to mORMot, WCF is much more complex and need much more code to work with. For instance, you have to define one endpoint per interface, and write a client class. All this is transparent with mORMot: you just use interface instances, and you work with it. Take a look at the sources provided with this blog article, or the one available in http://synopse.info/fossil/dir?name=SQLite3/Samples/14+-+Interface+based+services and make yourself an idea.

      • Arioch
        November 26, 2012 at 4:46 am | #30

        Did you by chance considered actor-based frameworks like Erlang or Scala ? That is probably close to IOCP by idea. And that approach should be easy to millon-of-hello-world type of stress, like in this essay.

  10. Tom
    November 23, 2012 at 6:24 pm | #31

    This is test of bad design Indy server.
    We need test of DataSnap framework so pleas test with IIS, also check this project
    http://sourceforge.net/projects/uvc/
    This library is replacement for Indy and also contain UVCDSTCPServerTransport.pas DataSnap transport. It’s time for Embarcadero to develop IOCP server and integrate in DataSnap.

  11. DHR NY
    November 24, 2012 at 1:40 am | #32

    Delphi Servers: Worst Performance
    http://delphihaters.blogspot.com/2012/11/delphi-servers-worst-performance.html

    Pingback from DHR-NY

    Can you also test IntraWeb and let us know results? We did a test some time ago and performance was so poor and quality so appalling it was joke.

    • November 24, 2012 at 9:42 am | #33

      Thanks for the comment.

      These tests take time, so do not know if I’ll have time to perform the tests with IntraWeb. Another point is that the Intraweb is not in our plans.

      Anyway, I think the performance will be poor.

  12. November 24, 2012 at 9:27 am | #34

    Can anybody commment on whether these issues only relate to the REST implementation, or are they likely to be similar in a “standard” Datasnap Server, eg. sending data over TCP/IP as a service?

  13. November 24, 2012 at 6:49 pm | #36

    I have done some some testing too. Indy does not have a queue but creates as many threads as there are requests (till there are no more threads left) This even happens when you use a thread pool. The other effect of this is that requests are not handled in the order that they came. (Tested this) This will result into random performance because some requests which came in first can be handled much later then request posted afterwards. When you use iis for your websnap application you will have a proper queue and pool. But be aware I know some nice hidden tricks in websnap to do direct database access which may open up your internal database if the attacker can guess some of your internal component names. I would be very care full before connecting a websnap server directly to the internet.

    • November 24, 2012 at 6:52 pm | #37

      Silly typo I mean datasnap not websnap. ( should not write this stuff at nearly 23:00)

  14. Alexandre
    November 25, 2012 at 8:51 pm | #38

    Roberto, have you try to talk with Andreano Lanusse? He is no longer at Embarcadero but he can give you some insights, he is the guy when the topic is Datasnap and his blog (http://www.andreanolanusse.com) has a lot of good stuffs about Datasnap

    I have from smal to medium applications using DataSnap through TCP and works fine for me, but I have the plans to move to a REST Server architecture and your post is great and give us some reference.

    • November 25, 2012 at 9:09 pm | #39

      I contacted Andreano by e-mail, but he did not answer me yet.

      • November 28, 2012 at 3:10 am | #40

        Hello everyone,

        There is a lot of great stuff on this benchmark and congratulations to everyone that expand time on that.

        At least the Delphi community see some interesting Delphi alternatives for multi-tier development, like mORMot. Would be nice to see Data Abstract and kbmMW as part of this tests.

        The performance issue on DataSnap REST Servers is not only related with Indy, but also with the Delphi JSON parser which is slower than other libraries like mORMot, superobject and jsonchimera. It will in parts affect TCP/IP connections, but TCP is another history.

        Also, changing the DataSnap lifecycle doesn’t do anything on DataSnap REST Server, as all the requests works as Invocation, reason why you should set the Session expiration time to a much lower number.

        On benchmarks like this running thousand of time, for sure JSON Parser contribute for the slow performance, on the other hand the other frameworks used has a much faster JSON parser and different architecture of course.

        Indy definitely needs optimization and several changes was made in XE2 to allow Indy updates without affecting DataSnap, however for this scenario I think it’s necessary updates on DataSnap and Indy to fix the problem.

        In the mean time would be good to see the same benchmark using TCP/IP and running the 3 different lifecycles. This article can help you to set that in runtime http://www.andreanolanusse.com/en/registering-datasnap-server-class-in-runtime-with-delphi/

        Like I said at the beginning there is a lot to talk about this benchmark, I will post on my blog some on my thoughts about this soon.

      • November 28, 2012 at 7:33 am | #41

        Hello Andreano. Thanks for the comment.

        I had planned to test the Data Abstract (RemObjects). But when I contacted support they said that does not support REST.
        I did some basic tests. I noticed that they also use indy, and apparently also has problems. In performance tests I did RemObjects SDK seemed to be slower than the DataSnap.

      • November 28, 2012 at 7:46 am | #42

        Andreano remembered something very important, the Delphi JSON parser is something different from the vast majority of Frameworks. As we integrate Delphi (DataSnap) with other languages ​​we have to use other JSON parsers as Superobject to ensure compatibility of conversions.

        I learned that the TMS Aurelius ORM can not use the native Delphi JSON parser because of compatibility and performance.

  15. Carlos Matias
    November 25, 2012 at 9:13 pm | #43

    To Marco Cantu, hello from Brazil

    Host in IIS doesn’t help at all, Indy and the poor DataSnap architecture and Indy still there, and this is the root of the problem. How you guys release this kind of solution and doesn’t provide any test around scalability and speed? Please fix that and give us a update for XE2.

    David I just demo wizards and doesn’t go deep on DataSnap, look his comment on this thread, he talk about TCP/IP and the whole issue here is REST, he doesn’t understand that REST Service is the standard now?

    The same issue Roberto had with the local partner here in Brazil I had several times, they say they are Embarcadero but they are not, they are a local company working as reseller and doesn’t add any value for me, they charge 60% more for a single Delphi license. Marco, why do I have to pay 60% more? In the past when I had problem I always contacted Andreano Lanusse and he always help me, but unfortunately he is gone.

    Embarcadero PLEASE look our situation in Brazil, you are losing lot’s of developers here because your reseller.

    • November 26, 2012 at 9:09 am | #44

      Hi Carlos,

      Indy is not needed or used at all in an ISAPI extension/Apache module.

      • Carlos Matias
        November 26, 2012 at 11:49 am | #45

        Ondrej, I know Indy is not need in IIS, however the problem still there and why should I host my server in a IIS to fix a problem on datasnap architecture?

  16. alexandre
    November 26, 2012 at 12:48 pm | #46

    There is a ridiculous limitation in the Datasnap. If you use datasnap with Shared connection to connect sqlconnection from client to sqlconnection in the server (DSAdmin.GetConnection) you can only transfer blob minor then 64k. You can increase BufferKbSize but does not work. The datasnap will return blob as null.

  17. November 26, 2012 at 2:39 pm | #47

    Carlos,

    I understood your comment “Indy still there” to imply that a DataSnap server still used Indy in an ISAPI extension – which is not true. (Sorry if I misunderstood it.)

    Why use IIS? A good reason might be because it’s been optimized for acting as a web server. Not to “fix a problem on datasnap architecture” – it can’t do that, of course.

    If you’re aware of any bugs please report them in Quality Central so they can be fixed. Thank you!

    • November 27, 2012 at 5:13 am | #48

      I would rather not use IIS as ISAPI extension.
      First of all, it will need a 64 bit library to work on any modern system (since IIS will be loaded as 64 bit), and it will add unneeded overhead.
      Delphi 64 bit compiler is less optimized than the 32 bit (apart from the floating point), and you certainly do not need more than 2/3 GB of memory, unless you need to change your server architecture. And you will face some compatibility issues when using 64 bit compiler (like third-party dependencies and more bugs).

      The best is to directly use http.sys kernel-mode server, just like IIS, and bypass the IIS layer.

      Both WCF and mORMot can use this http.sys kernel-mode server. This is one of the reasons of their very good performance.

  18. Felipe Piña
    November 26, 2012 at 7:03 pm | #49

    I think that some how you open a pandora’s box, Marco and David, I think that embarcadero has a serious decition to make about the quality and client satisfaction. It is a great tool, and could be better, but i think that th custumer complains are getting bigger, The waka of pascal also has a lot of complains, I think that you should stop develpoping more features and concentrate on fixing and making a stronger product. Sometimes i feel that embarcadero sells more and envelop than a real product. We want a real powerful development tool with which we can feel proud and of course be very productive and delivery good and fast apps.

  19. Mitko
    November 27, 2012 at 11:42 am | #50

    Hi,

    I work with DataSnap since Delphi 2010. There is a problem when you use a Server on Session lifecycle with statefull work.
    The best way to work with DataSnap is Invocation with stateless method. Then your test will not crash! Try it and write, what happens!

    • November 27, 2012 at 12:42 pm | #51

      Hello Mitko. Thanks for the comment.

      I tested all kinds of lifecycle, and all had the same behavior in this test. The server crashes in all cases, regardless of the lifecycle.

      • November 27, 2012 at 1:24 pm | #52

        Can you compile your test projects with Debug DCU’s turned on and with MadExcept / Eurekalog? This can show you the callstack

      • November 27, 2012 at 1:35 pm | #53

        Yes upow, I can compile using MadExcept. The problem is I can not do anything to correct, even though the callstack. Who can do something is the Embarcadero.

        As far as I know Marco Cantu’s working on it.

  20. Daniel Sonda
    November 27, 2012 at 2:29 pm | #54

    Tomorrow will happen a webinar about DataSnap and one of the topics is making scalable and robust services. So how will that be?
    http://edn.embarcadero.com/article/42686

  21. November 28, 2012 at 6:40 am | #56

    Seriously, a DataSnap console REST server? I wonder if these are actually used as deployed applications. For all the DataSnap Servers I’ve written and deployed (over a dozen in the last two years), only one was NOT an ISAPI DLL.
    When create a DataSnap REST Server application as an ISAPI DLL, deployed using IIS, you may find very different results. At least no Indy issues (which seem to be part of the problem)…

    • November 28, 2012 at 7:13 am | #57

      Hello Bob. Thanks for the comment.

      This question is for me or Embarcadero?

      I’m no expert, as you seem to be. Sorry if I should have used ISAPI in tests. In all conferences and texts I read about DataSnap, the advantage of not needing IIS is always remembered, so I thought I could use a console application.
      In my view, if I have the option to create a console application using DataSnap, it should work, at least!

      I also had problems using IIS, but I did not spend much time analyzing.

      Marco Cantu and his team are analyzing the problem. Apparently there is a competition problem that occurs even using an ISAPI DLL.

    • Francisco Ruiz
      November 28, 2012 at 9:29 am | #58

      Absolutly agree, Indy can be used for tests but it is a requeriment to deploy as Isapi. Yes or yes.

      • Arthur Hoornweg
        November 29, 2012 at 5:59 am | #59

        Well, that restriction does not seem to apply to competing frameworks like mORMot … Also, installing an ISAPI inside IIS is not a trivial task and has lots and lots of pitfalls. In my experience, writing a reliable “setup.exe” for an ISAPI DLL may be more work than writing the DLL itself. That is the one aspect of ISAPI development that everyone (including drBob) prefers not to mention – how do you deploy the damned thing in a user-friendly way.

  22. November 28, 2012 at 7:56 am | #60

    To see wether Indy or DataSnap is the bottleneck, I ran a single-threaded JMeter test of the Indy HTTP server on a quite slow system (Mobile Core2 Duo @ 2.1 GHz), with HTTP 1.1 keep-alive enabled. The average throughput was 320 requests/second, with zero (0) % HTTP errors. The CPU load was around 40%, Client and server where running on the same system. In the tests described above, DataSnap with Indy as HTTP transport reached 62 requests/second on a Core i5 750 with 4 x 2,67 GHz system.

    • November 28, 2012 at 8:13 am | #61

      There are many things that can influence the tests. I performed the tests on two machines without any other software.

      I’m preparing another post about software that influence the DataSnap. In my case I noticed absurd performance differences in some cases. When I’m using Google Chrome or have opened the EyeBeam the DataSnap server is much faster. I still do not know why, I’m researching.

      There are also some JMeter settings that can influence the result.

      In the test with a single thread CPU usage is around 12% in Core i5.

  23. Michael
    November 28, 2012 at 8:13 am | #62

    The test is ok. What you tried others simply will try too. If the click through doesn’t work why bother … People simply test this way.

  24. zedalaye
    November 28, 2012 at 11:49 am | #63

    zedalaye :
    I pushed a branch with a DoR server on Github.

    Did some basic benchmarking with JMeter (50 clients connections and at least 1M requests) :
    mORMot : ~5900 r/s (CPU below 10% / 4Mb memory)
    Node.js : ~5400 r/s (CPU around 25% / memory oscillating between 8 and 55Mb)
    DoR : ~3900 r/s (CPU around 20% / 5.2Mb memory)

    • November 28, 2012 at 12:04 pm | #64

      The performance issue on DataSnap REST Servers is not only related with Indy, but also with the Delphi JSON parser which is slower than other libraries like mORMot, superobject and jsonchimera. It will in parts affect TCP/IP connections, but TCP is another history.

      Delphi JSON parser is quite slow, indeed, and could definitely be a bottleneck.

      With mORmot we tried to make the faster JSON parser possible for our purpose.
      In fact, all JSON content is parsed in-place in the input text buffer: no memory is allocated, just data is un-escaped as UTF-8, and an array of pointers to the input buffer is created. This is incredibly fast (a similar algorithm is used in the fastest XML readers).
      See http://blog.synopse.info/post/2011/06/02/Fast-JSON-parsing
      This is diverse from SuperObject or the JSON parser included in DWS (which is faster than SuperObject): both creates objects tree in memory to map the content.
      Threfore, the parser included in mORMot is in fact *much* faster, especially when used in a multi-threaded server. The same for the JSON producer of mORMot, which avoid as much memory allocation as possible, and try to achieve best possible speed in multi-thread environment.

  25. November 28, 2012 at 2:24 pm | #65

    In the DataSnap server project (DataSnap_Console_Server), I see no KeepAlive setting for the LServer. Your JMeter test has KeepAlive enabled, but the DataSnap server defaults to KeepAlive False (found at http://stackoverflow.com/questions/11513222/datasnap-rest-server-enable-http-keepalive). I guess the server will always close the connection immediately, leaving (one million?) of sockets with state TIME_WAIT in the OS,

  26. November 28, 2012 at 2:28 pm | #69

    Just wanted to mention I blogged on this issue offering some more insight at http://blog.marcocantu.com/blog/datasnap_deployment_performance.html

    • November 28, 2012 at 5:48 pm | #70

      Hello Marco.

      I understand and respect your opinion. Surely the server I used in testing is not optimized, but we will work to optimize it, and then I’ll redo the tests (with the fixes already made ​​by Embarcadero).

      I know the frameworks that I’ve tested have different features, to different purposes, I put it in the text. I also put my goal was not to compare the frameworks, and yes, test the performance of DataSnap!

      I want to make clear (Maybe gotten the wrong impression) that I am not wanting a Wizard with more options and everything is configurable thru clicks. None of the other servers I have developed has Wizards and are still working perfectly. I think the documentation of the DataSnap is very weak. I am not charging everything to be easy to implement, just need to have documentation to tell me how the framework works, so I can implement.

      Thanks for all the support that I’m taking from you Marco. It’s a shame the support of the Embarcadero (at least in Brazil) does not follow the example and does not help us.

      • November 29, 2012 at 6:22 am | #71

        Thank you for the work you are doing. Putting pressure on the development team (with positive criticism and suggestions) is always welcome.

  27. Arthur Hoornweg
    November 29, 2012 at 5:27 am | #72

    Hi Roberto,

    I’m a bit surprised that the Remobjects (and/or DataAbstract) framework is missing in your lineup. Especially since it is a multi-platform framework, it would give you more implementation options than any of the other frameworks you tested here and maybe be more future-proof.

    • November 29, 2012 at 7:12 am | #73

      Hi Arthur.

      The RemObjects support told me that the framework does not support REST. That was one of the reasons why I have not used it in testing. Another reason is that the framework of RemObjects showed a behavior very similar to datasnap in some tests I did. They also use Indy. I did some testing with this framework and server performance was worse than the DataSnap. Of course, there must be optimizations to be made, so with DataSnap, but the framework is very large, I have not had time to study it.

      The main point is that does not support REST, we will not be able to use it anyway.

      • Arthur Hoornweg
        November 30, 2012 at 4:57 am | #74

        Fair enough. I use Remobjects with Synapse myself (never liked Indy).

  28. Tom
    November 29, 2012 at 7:17 am | #75

    My proposal to Embarcadero is to write new TCP/HTTP server components, Sync and Async IOCP for Windows and epool for Linux , Mac etc. and write new Server Framework.
    Don’t try to bug-fix DataSnap. Performance of DataSnap is too far from modern Application Servers.
    60-100 req/sec, even 10000 req/sec is slow compared to some async servers for simple heloworld tests.
    Some tests from other servers

    http://forum.gwan.ch/benchmark#loan100

    Server Avg RPS RAM CPU Time for test
    —— ——- ——— ——— ————————-
    Nginx 167,977 11.93 MB 2,910,713 01:53:43 = 6,823 seconds
    Lighty 218,974 20.12 MB 2,772,122 01:09:08 = 4,748 seconds
    Varnish 103,996 223.86 MB 4,242,802 03:00:17 = 10,817 seconds
    G-WAN 729,307 5.03 MB 911,338 00:28:40 = 1,720 seconds
    729307 >>>>>>>>>>>>>>>>>> 62 req/sec

    • zedalaye
      November 29, 2012 at 7:44 am | #76

      Some work has already been done :

      http://code.google.com/p/delphi-iocp-framework/

      • November 29, 2012 at 11:13 am | #77

        Under the Windows platform, you should better use http.sys API, which works in kernel mode, and uses IOCP.
        Since it is part of the OS kernel, and is very tuned, you will get best possible performance and scaling. Better than an hand coded solution.
        Writing a HTTP server is complex, and even if you achieve good performance, you have to take care of the stability and security. For instance, how would a pure Delphi IOCP framework respond to DoS or fuzzing attack? http.sys already includes such protections.

      • Arioch
        November 29, 2012 at 11:28 am | #78

        http://en.wikipedia.org/wiki/Monoculture_(computer_science)

        implementing your own server you might show your own bugs, but you probably would not be subject to Windows malware.

        Relying on HTTP.SYS you get MS workforce for you, but you also get all Windows-centered crackers for you.

      • abouchez
        December 3, 2012 at 3:59 am | #79

        @Arioch

        It does not take a genius hacker to test a DoS with a simple “hello wolrd” query.
        It will be the first test every “villain” will do on the server side, even before targeting with http.sys. Every hacking factory toolset has such a robot.
        Then, any DataSnap server before XE3 next hotfix will just break after a pretty basic attack.
        I’m was not talking about complex security leaks, but about a major concern of DataSnap usability in production.
        Take in account that most attacks are not targeted. Bots just browse the Internet in research of weak servers. And DataSnap is more than a honey pot here!

  29. AndreFM
    November 29, 2012 at 9:23 am | #80

    Andreano Lanusse :
    Hello everyone,
    At least the Delphi community see some interesting Delphi alternatives for multi-tier development, like mORMot. Would be nice to see Data Abstract and kbmMW as part of this tests.

    Don’t forget RealThinClient. Check their stability and performance tests…

  30. November 29, 2012 at 10:14 am | #81

    Roberto Schneiders :
    I did some basic tests. I noticed that they also use indy, and apparently also has problems. In performance tests I did RemObjects SDK seemed to be slower than the DataSnap.

    FTR, RemObjects SDK does support a wide variety of standard communication component libraries on Delphi, including Indy, but it does not require the use of Indy (nor is Indy our recommended/preferred choice for the channel).

  31. Tom
    November 29, 2012 at 7:09 pm | #82

    abouchez :
    Under the Windows platform, you should better use http.sys API, which works in kernel mode, and uses IOCP.

    abouchez what about TCP communication, http is not the only Internet protocol. Protobuf, Thrift, MsgPack, BSON etc, is binary protocol and http i big overhead in some systems.
    Did http.sys enable use of tcp/ip socket protocol?

    • November 30, 2012 at 10:11 am | #83

      Thanks Tom for the comments.
      But we must take care to not mix the layers in such cases!

      1. About transmission protocol, HTTP is a layer over TCP, and has not such a big overhead. By design, http.sys won’t enable use of TCP/IP protocol (unless you use Windows 8 and switch to websockets mode, but it is not very well supported over all networks).
      A TCP layer won’t be faster in practice.

      But mORMot propose other transmission protocols, as in-process calls, or named pipe or GDI messages (the later is very efficient in practice, but only works locally).
      You can have several protocols at once, in just one code line each, to publish an unique server. Or you can host several servers on the same process/service. For instance, you can do load-balancing in a logical n-Tier architecture, using GDI messages locally between application and domain layer, but a public HTTP front end.
      See http://blog.synopse.info/post/2012/05/25/Domain-Driven-design

      2. About application protocol, we use JSON and a REST/stateless routing scheme.
      We found out JSON to be very efficient, better than XML, and comparable to binary protocols in most cases. We can optionally use deflate/zip compression or our own very optimized SynLZ algorithm, to reduce bandwidth.
      In practice JSON is much more easy to work with from any client (including JavaScript/AJAX or even a human brain) than any binary layout. Packet inspection software will like JSON much more than a binary content.
      For instance, mORMot is able to use a “stretched” layout when returning a list of data, sending them as an array of items with the headers just sent once, instead of an array of objects. We have to embed small BLOBs as Base64, but the framework prefers a direct RESTful remote access via a dedicated URI and direct binary download.
      We use a very fast in-place parsing (like a SAX approach) instead of a DOM-like template.
      And when it deals with database access, we directly convert the DB client buffers into JSON content, with no temporary storage. Performance is outstanding – http://blog.synopse.info/post/2012/07/25/Synopse-mORMot-benchmark – and can be even better when cache is enabled (it was disabled for this testing, on purpose).

      From our tests on real world data, due to the TCP packet overhead, using JSON over HTTP is just a very good option, just in-between XML/SOAP and proprietary binary content.

  32. Arthur Hoornweg
    November 30, 2012 at 6:06 am | #84

    I’d like to remark that, since Datasnap is part of the “expensive” Delphi SKU’s and supposed to be a high-end feature, I would have expected it to be more thoroughly tested and benchmarked by Emb. One simply can’t afford any bottlenecks / memory leaks / reliability issues in such a thing.

    • November 30, 2012 at 7:21 am | #85

      I totally agree. Apparently performance and stability is not the focus. Let’s see if Marco Cantù can change that.

      • Rob
        November 30, 2012 at 9:42 pm | #86

        “Let’s see if Marco Cantù can change that”

        I think he already has :-)

        See :
        http://blog.marcocantu.com/blog/datasnap_webinar_bedelphi_material.html

        “Going forward, we are adding some serious multithreaded testing to the internal test suites, to avoid similar issues in the future. Reworking and improving DataSnap is on my table, and I’m open to suggestions.”

  33. December 3, 2012 at 2:48 pm | #87

    Some very cool JMeter plugins are here: http://code.google.com/p/jmeter-plugins/ – ther PerfMon plugin for example can monitor CPU, Memory, Swap, Disks I/O and Networks I/O on almost all platforms

  34. phil
    December 7, 2012 at 9:59 am | #89

    Very clear post, with a lot of informations. I think that Indy may, also, be one of the problem. For other purpose I tried some perf. tests and Indy was really less efficient than Overbyte ICS. May be it was a tuning problem, but we can’t spend to many time to tune…

  35. orko
    December 9, 2012 at 9:18 pm | #90

    Can Embarcadero fix this problem with new update?

    • December 10, 2012 at 9:51 am | #91

      They are trying. Until now, even with all optimizations and fix that Marco Cantu made ​​available, the problem persists.

      The Marco said his team is testing various fixes. The problem is that the update will be to version XE3. Anyone using the XE2, which is my case, will be forced to upgrade.

  36. December 15, 2012 at 7:23 am | #92

    Daniele Teti recently posted some new test results with XE3 upd1, though he could get it to not crash, the timings look horrible (500 ms to 2040 ms latency for a simple reverse string and only 100 users)

    • December 15, 2012 at 10:37 am | #93

      Very interesting.

      I do not redid the tests with XE3 update 1, but I will. I’m on vacation until January.

      An interesting detail is that in my tests using the benchmark software Apache Bench datasnap server does not crash. There are significant differences between the AB and JMeter. Not sure why the AB does not crash server.

      What I can say is that in preliminary testing that I conducted with the fixes that Cantu sent me, the server kept crashing.

  37. December 15, 2012 at 8:36 am | #94

    no comment, thats a shame. i like embarcadero and think they do a good work at all, so many great new features in the last time so its clear that there are many bugs. but we upgraded all our licences to xe exterprise because of datasnap, run into many of you problems and fixes them with “noodle coding”, so its working for us now. but that there wont be a patch for at least the last versions for such ciritcal bugs is really anoying. we have maintenance contract’s but we wont go to a new enviroment because of some bugfixes. we have to say, like many other people in the german usergroup too, that the maintenance contract is not worth the word, except that you can test the new delphi version. we have a complex product with many 3rd party components and its not possible to upgrade every year and try to fix new bugs after.

    what about the new mobile studio? we wont buy it. its version (0.)1. this means many bugs and no support after lets say 10 month. so it will be a unusable version for productvity and ROI efforts.

    ps – roberto, thats a really great work of you. people can say u missed some informations but all people, except the spezialists that where involved, miss this informations. keep this good work. additionally embarcadero seems to look on things, if they are written on such a proffessional way.

  38. December 23, 2012 at 11:45 pm | #96

    Thank you Daniel

  39. orkun
    January 6, 2013 at 5:22 pm | #97

    is there any news? any update for datasnap?

    • January 6, 2013 at 6:23 pm | #98

      Yes. Embarcadero has released an update for XE3. I’m making tests and I will publish a new post in a few days.

      • orkun
        January 6, 2013 at 6:34 pm | #99

        Thank you

      • orkun
        January 9, 2013 at 8:28 am | #100

        are you making test for only “hello world”? or are you use sessions and real world senerio?

      • January 9, 2013 at 8:45 am | #101

        For now only tests using Hello World. How is a rest server, there are no sessions, he works as invocation all the time.

        These tests are time consuming to make, mostly because DataSnap servers are too slow, I just do not have time enough to do more tests.

        Anyway, after these tests will be difficult to rely on DataSnap and use it in a production environment.

  40. Fabricio K.
    January 10, 2013 at 9:09 am | #102

    Roberto Schneiders :
    For now only tests using Hello World. How is a rest server, there are no sessions, he works as invocation all the time.
    These tests are time consuming to make, mostly because DataSnap servers are too slow, I just do not have time enough to do more tests.
    Anyway, after these tests will be difficult to rely on DataSnap and use it in a production environment.

    This statement holds true for your any type of DataSnap server, or you are referring only to DataSnap Rest?

    • January 10, 2013 at 9:39 am | #103

      My tests were only in REST servers, hard to say something about the other servers. However, I believe that the datasnap TCP/IP server suffer from similar problems, especially problems related to performance, who I’ll show in the next post in REST servers.

  41. January 16, 2013 at 2:24 pm | #104

    Please put a link here to the new tests, once you have time to do them. Great article and great attitude. These type of tests can really get the ball rolling.

  42. orkun
    January 16, 2013 at 7:52 pm | #105

    we waiting your test Roberto.

    • January 17, 2013 at 7:09 am | #106

      I finished the tests. The next post is almost ready. I still need to review it. As you know, English is not my native language, I have some people helping me revise the texts.

      Anyway, I hope to publish it until the weekend.

      Thanks for your interest.

  43. April 30, 2013 at 9:08 am | #107

    Olá Roberto. Sou editor da Clube Delphi. Quero lhe fazer um convite. Você não gostaria de compartilhar com a comunidade essa sua pesquisa/análise em um artigo para a revista? Inclusive, estendo o convite para outros artigos sobre o mORMot. O que acha? Me contata em private.

  1. November 23, 2012 at 10:51 am | #1
  2. November 26, 2012 at 9:11 am | #2
  3. November 28, 2012 at 5:09 am | #3
  4. November 28, 2012 at 12:16 pm | #4
  5. January 18, 2013 at 7:07 am | #5

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: