"The Plugin you are using seems to generate a bad URL" on private GitLab Server

0 votes
asked May 5, 2020 in Bug by SydonX3 (120 points)

image

The plugin you are using seems to generated a bad URL.
This URL does not look like DEFLATE data.
It looks like your plugin is using HUFFMAN encoding.

Since this morning I get the message above instead of the diagrams when looking at a markdown file. The weird thing is that I haven't updated the GitLab Server and no news is mentioned on the plantuml site. (yesterday everything worked).

I updated my GitLab Server but it hasn't resolved my issue.

2 Answers

0 votes
answered May 5, 2020 by plantuml (294,960 points)
We have been bound to make changes on http://www.plantuml.com/plantuml because of performance issues.

Which plugin is your GitLab server using ? There is a slight change to be done in the URL generated by this plugin.

An additional string ~1 must be added in the URL generated. It's really a minor change.

But it is not clear to us which plugin is using your GitLab server.

Do you know which team is developping this plugin ? We will be glad to help them.
commented May 5, 2020 by SydonX3 (120 points)

I'm using a GitLab Server at version 12.9.5. I don't know where exactly the plugin is developed. Probably somewhere in the GitLab Repository. But seeing that it works on the official server it either uses its own server or it has already updated as it runs on a preview version 13.0.X.

I solved the issue by running my own server with the latest docker file: https://hub.docker.com/r/plantuml/plantuml-server

I've created an issue in the GitLab repo:
https://gitlab.com/gitlab-org/gitlab/-/issues/216587

commented May 5, 2020 by SydonX3 (120 points)
edited May 5, 2020 by SydonX3
I should probably also note that the current Docker version does neither support the !procedure  syntax nor the deprecated !function syntax, but fortunately doesn't require the ~1. This is really weird because the Docker tag suggests the version is v1.2020.8 which is the version that I use locally in Atom that works well with the procedures.
commented May 14, 2020 by Horacio
The library used by Gitlab for generating PlantUML diagrams is:

   - https://github.com/hsanson/asciidoctor-plantuml

Thanks to some contributed code I already released an updated version that uses the new URL but I still need to test this when I have time during the weekend.

Some questions:

  - I implemented the HUFFMAN encoding by looking at PlantUML java source, would

    be amazing if the new DEFLATE encoding is documented somewhere so plugin devs

    can also implement it.  Specially if there are plans to deprecate HUFFMAN in the

    future.

  - Is there a mailing list or RSS where these breaking changes are announced in advance?

    I know people should install and keep their own instances of plantuml to avoid these                issues, still this change most likely broke Gitlab for a lot of them.
commented May 14, 2020 by plantuml (294,960 points)
First, I apologize about this URL change that was done somehow unexpectedly, even for us :-)

I think I should explain more about this :

The traffic on http://www.plantuml.com/plantuml/ is constantly increasing. This is great because it means that more and more people are using PlantUML. However, it means that we have to constantly improving our infrastructure, especially because PNG image dynamic generation is far more expensive (in term of CPU usage) than regular html page generation.

So recently we have added some more powerfull servers (using JVM 14) to our bunch of servers to be able to serve everyone.

However, we quickly notice some strange behaviour: occasionally, the new servers freeze (that is, their CPU goes to 100% for a long time). It was very annoying because it was jeopardizing the whole PlantUML service. Hopefully, end users did not notice this (too much) because the old legacy servers did not suffer from this disease. But those old servers could not run the increasing load.

It takes us some time to understand that the issue was coming from this line :

https://github.com/plantuml/plantuml/blob/master/src/net/sourceforge/plantuml/code/TranscoderSmart.java#L61

When receiving a request, we try to decode it using zlib decoder. If it fails, we try then huffman decoding (we see latter why such difference). It appears that on the new JVM, for some unknown reasons, some Huffman encoded requests (which are not valid zlib data) confuse the zlib decoder and throttle CPU to 100%.

BTW there is a mistery in the mistery because when we look your code at https://github.com/hsanson/asciidoctor-plantuml/commit/7833ae009bf8f90562322f3f499417172943eaa7 it seems that you are Zlib::Deflate which should generate zlib data. I have no idea why the generated data are "Huffman" to our Java code. This is really strange.

Anyway we were stuck here because we had absolutely no idea about who was calling our service with those newly problematic URL and there was no easy way to warn people. Still, this was crashing our service. Furthermore, the issue did not come from the use of Huffman but more specifically on the fact that we were trying to guess if a request is zlib or Huffman. It was this guess that causes the issue.

Adding a simple header to choose between zlib and Huffman was the easiest solution. Since the vast majority of requests was zlib, we decide to impose the header for Huffman. So there was no impact for zlib (which, again, is the majority of requests).

Doing so, we were fully aware that we were breaking all users using Huffman, so we decide to print an error message asking to people to add this strange ~1 header, even if we did not explain why (at that moment). I must say that we did that very quickly and running against time (and againt servers crashes :-)

As conclusion, you can safely use this ~1 header and the huffman encoding now. We have no plan to deprecate this in the future. Since we have some header now, it's easy to detect it. We are going to update https://plantuml.com/en/text-encoding page with huffman encoding (we are just out of time!).

I hope this is more clear : feel free to ask questions !
commented May 22, 2020 by Horacio
Thanks for the detailed explanation. It is understandable the breaking change was necessary to keep the service.

The plugin I wrote is used in several places with lots of users (e.g. Gitlab) and I really hope it is not this plugin the one that caused issues for PlantUML.  If the plugin ever causes any problems to PlantUML please report it in github and I will take prompt actions to fix it.
commented May 23, 2020 by plantuml (294,960 points)

The plugin I wrote is used in several places with lots of users (e.g. Gitlab) and I really hope it is not this plugin the one that caused issues for PlantUML.  If the plugin ever causes any problems to PlantUML please report it in github and I will take prompt actions to fix it.

Don't worry, the root cause of all this is really on our side, not on anyone else.

Could you give us the URL of your plugin in github so that we can check if you need some actions ? Thanks !

0 votes
answered Oct 15, 2020 by Anders B

When searching the net for solutions, there isn't much to be found...To actually resolve the issue mentioned i had to do the following:

Edit /etc/gitlab/gitlab.rb and add:

gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' }

For the changes to take effect, run gitlab-ctl reconfigure: 

sudo gitlab-ctl reconfigure

Note: i got most of it from the official guide: https://docs.gitlab.com/ee/administration/integration/plantuml.html#gitlab but it failed to mention the "reconfigure" part. 

...